2024-12-17T23:58:03.1109914Z Current runner version: '2.321.0' 2024-12-17T23:58:03.1117642Z Runner name: 'i-0a62689d000b23974' 2024-12-17T23:58:03.1118594Z Runner group name: 'Default' 2024-12-17T23:58:03.1119708Z Machine name: 'ip-10-0-44-187' 2024-12-17T23:58:03.1125437Z ##[group]GITHUB_TOKEN Permissions 2024-12-17T23:58:03.1128556Z Actions: read 2024-12-17T23:58:03.1129374Z Attestations: read 2024-12-17T23:58:03.1130084Z Checks: read 2024-12-17T23:58:03.1130788Z Contents: read 2024-12-17T23:58:03.1131564Z Deployments: read 2024-12-17T23:58:03.1132325Z Discussions: read 2024-12-17T23:58:03.1132993Z Issues: read 2024-12-17T23:58:03.1133713Z Metadata: read 2024-12-17T23:58:03.1134556Z Packages: read 2024-12-17T23:58:03.1135210Z Pages: read 2024-12-17T23:58:03.1135908Z PullRequests: read 2024-12-17T23:58:03.1136551Z RepositoryProjects: read 2024-12-17T23:58:03.1137283Z SecurityEvents: read 2024-12-17T23:58:03.1138022Z Statuses: read 2024-12-17T23:58:03.1138784Z ##[endgroup] 2024-12-17T23:58:03.1143262Z Secret source: Actions 2024-12-17T23:58:03.1144438Z Prepare workflow directory 2024-12-17T23:58:03.1686531Z Prepare all required actions 2024-12-17T23:58:03.1729195Z Getting action download info 2024-12-17T23:58:03.3686084Z Download action repository 'pytorch/test-infra@release/2.6' (SHA:eb0adf5a84668865394af69e26428b32c8105c1c) 2024-12-17T23:58:04.8974098Z Download action repository 'pytorch/pytorch@release/2.6' (SHA:0cdf8b1d09254cfda66191d1bd01e3041c3c76f7) 2024-12-17T23:58:18.2449139Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-12-17T23:58:18.4848444Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-12-17T23:58:18.7709996Z Getting action download info 2024-12-17T23:58:18.9193144Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-12-17T23:58:19.1794552Z Getting action download info 2024-12-17T23:58:19.2944769Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2024-12-17T23:58:19.4799260Z Getting action download info 2024-12-17T23:58:19.6109004Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-12-17T23:58:19.7541772Z Getting action download info 2024-12-17T23:58:19.8841181Z Download action repository 'pytorch/test-infra@main' (SHA:a07505a74641a4ff5123d635defac481ef28ef1e) 2024-12-17T23:58:21.1396450Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/release/2.6 (0cdf8b1d09254cfda66191d1bd01e3041c3c76f7) 2024-12-17T23:58:21.1398699Z ##[group] Inputs 2024-12-17T23:58:21.1399129Z build-environment: linux-focal-cuda11.8-py3.10-gcc9 2024-12-17T23:58:21.1400697Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}]} 2024-12-17T23:58:21.1402681Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-17T23:58:21.1403589Z sync-tag: 2024-12-17T23:58:21.1404424Z timeout-minutes: 360 2024-12-17T23:58:21.1404734Z use-gha: 2024-12-17T23:58:21.1404999Z dashboard-tag: 2024-12-17T23:58:21.1405290Z s3-bucket: gha-artifacts 2024-12-17T23:58:21.1405607Z aws-role-to-assume: 2024-12-17T23:58:21.1406238Z disable-monitor: false 2024-12-17T23:58:21.1406571Z ##[endgroup] 2024-12-17T23:58:21.1407169Z Complete job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-17T23:58:21.2101190Z A job started hook has been configured by the self-hosted runner administrator 2024-12-17T23:58:21.2207430Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-12-17T23:58:21.2216536Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-17T23:58:21.2217297Z ##[endgroup] 2024-12-17T23:58:22.3580628Z Runner Type: linux.g4dn.12xlarge.nvidia.gpu 2024-12-17T23:58:22.3581246Z Instance Type: g4dn.12xlarge 2024-12-17T23:58:22.3581595Z AMI Name: unknown 2024-12-17T23:58:22.3611645Z AMI ID: ami-0fff1b9a61dec8a5f 2024-12-17T23:58:28.2438592Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@release/2.6 2024-12-17T23:58:28.2439098Z with: 2024-12-17T23:58:28.2439836Z github-secret: *** 2024-12-17T23:58:28.2440588Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-12-17T23:58:28.2441412Z activate-with-label: false 2024-12-17T23:58:28.2441718Z label: with-ssh 2024-12-17T23:58:28.2441982Z remove-existing-keys: true 2024-12-17T23:58:28.2442283Z fail-silently: true 2024-12-17T23:58:28.2442532Z env: 2024-12-17T23:58:28.2442761Z GIT_DEFAULT_BRANCH: main 2024-12-17T23:58:28.2443082Z ##[endgroup] 2024-12-17T23:58:28.3679228Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-12-17T23:58:28.3680637Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys 2024-12-17T23:58:28.3810929Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@release/2.6 2024-12-17T23:58:28.3811425Z with: 2024-12-17T23:58:28.3811647Z no-sudo: true 2024-12-17T23:58:28.3811913Z submodules: recursive 2024-12-17T23:58:28.3812189Z fetch-depth: 0 2024-12-17T23:58:28.3812435Z env: 2024-12-17T23:58:28.3812666Z GIT_DEFAULT_BRANCH: main 2024-12-17T23:58:28.3812936Z ##[endgroup] 2024-12-17T23:58:28.3896928Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-17T23:58:28.3897946Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-17T23:58:28.3908527Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-17T23:58:28.3908984Z env: 2024-12-17T23:58:28.3909252Z GIT_DEFAULT_BRANCH: main 2024-12-17T23:58:28.3909604Z ##[endgroup] 2024-12-17T23:58:28.4015030Z ##[group]Run retry () { 2024-12-17T23:58:28.4015344Z retry () { 2024-12-17T23:58:28.4015694Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-12-17T23:58:28.4016110Z } 2024-12-17T23:58:28.4016364Z echo "${GITHUB_WORKSPACE}" 2024-12-17T23:58:28.4016711Z if [ -z "${NO_SUDO}" ]; then 2024-12-17T23:58:28.4017082Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-12-17T23:58:28.4017426Z else 2024-12-17T23:58:28.4017702Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-12-17T23:58:28.4018044Z fi 2024-12-17T23:58:28.4018300Z mkdir "${GITHUB_WORKSPACE}" 2024-12-17T23:58:28.4025098Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-17T23:58:28.4025581Z env: 2024-12-17T23:58:28.4025829Z GIT_DEFAULT_BRANCH: main 2024-12-17T23:58:28.4026147Z NO_SUDO: true 2024-12-17T23:58:28.4026418Z ##[endgroup] 2024-12-17T23:58:28.4050882Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-17T23:58:28.4140760Z ##[group]Run malfet/checkout@silent-checkout 2024-12-17T23:58:28.4141201Z with: 2024-12-17T23:58:28.4141504Z ref: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-17T23:58:28.4141902Z fetch-depth: 0 2024-12-17T23:58:28.4142189Z submodules: recursive 2024-12-17T23:58:28.4142481Z quiet-checkout: true 2024-12-17T23:58:28.4142795Z repository: pytorch/pytorch 2024-12-17T23:58:28.4143253Z token: *** 2024-12-17T23:58:28.4143518Z ssh-strict: true 2024-12-17T23:58:28.4143814Z persist-credentials: true 2024-12-17T23:58:28.4144136Z clean: true 2024-12-17T23:58:28.4144414Z sparse-checkout-cone-mode: true 2024-12-17T23:58:28.4144767Z lfs: false 2024-12-17T23:58:28.4145045Z set-safe-directory: true 2024-12-17T23:58:28.4145533Z env: 2024-12-17T23:58:28.4145782Z GIT_DEFAULT_BRANCH: main 2024-12-17T23:58:28.4146097Z ##[endgroup] 2024-12-17T23:58:28.5017941Z Syncing repository: pytorch/pytorch 2024-12-17T23:58:28.5020092Z ##[group]Getting Git version info 2024-12-17T23:58:28.5020688Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-12-17T23:58:28.5021505Z [command]/usr/bin/git version 2024-12-17T23:58:28.5022062Z git version 2.40.1 2024-12-17T23:58:28.5032966Z ##[endgroup] 2024-12-17T23:58:28.5048600Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/c4572818-5c90-43bb-bb74-d87319c623b9' before making global git config changes 2024-12-17T23:58:28.5049692Z Adding repository directory to the temporary git global config as a safe directory 2024-12-17T23:58:28.5051826Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-17T23:58:28.5080694Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-12-17T23:58:28.5083857Z ##[group]Initializing the repository 2024-12-17T23:58:28.5086117Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-17T23:58:28.5113776Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-12-17T23:58:28.5114523Z hint: is subject to change. To configure the initial branch name to use in all 2024-12-17T23:58:28.5115195Z hint: of your new repositories, which will suppress this warning, call: 2024-12-17T23:58:28.5115674Z hint: 2024-12-17T23:58:28.5116011Z hint: git config --global init.defaultBranch 2024-12-17T23:58:28.5116419Z hint: 2024-12-17T23:58:28.5116807Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-12-17T23:58:28.5117450Z hint: 'development'. The just-created branch can be renamed via this command: 2024-12-17T23:58:28.5117965Z hint: 2024-12-17T23:58:28.5118224Z hint: git branch -m 2024-12-17T23:58:28.5118814Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-12-17T23:58:28.5123784Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-12-17T23:58:28.5147110Z ##[endgroup] 2024-12-17T23:58:28.5147642Z ##[group]Disabling automatic garbage collection 2024-12-17T23:58:28.5149637Z [command]/usr/bin/git config --local gc.auto 0 2024-12-17T23:58:28.5174407Z ##[endgroup] 2024-12-17T23:58:28.5174890Z ##[group]Setting up auth 2024-12-17T23:58:28.5177740Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-12-17T23:58:28.5201047Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-12-17T23:58:28.5524144Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-12-17T23:58:28.5546361Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-12-17T23:58:28.5834698Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-12-17T23:58:28.5874475Z ##[endgroup] 2024-12-17T23:58:28.5874992Z ##[group]Fetching the repository 2024-12-17T23:58:28.5879201Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-12-17T23:58:31.3979521Z remote: Enumerating objects: 1056385 2024-12-17T23:58:31.3980122Z remote: Enumerating objects: 1057261, done. 2024-12-17T23:58:31.3980602Z remote: Counting objects: 0% (1/876) 2024-12-17T23:58:31.3981043Z remote: Counting objects: 1% (9/876) 2024-12-17T23:58:31.3981494Z remote: Counting objects: 2% (18/876) 2024-12-17T23:58:31.3981948Z remote: Counting objects: 3% (27/876) 2024-12-17T23:58:31.3982691Z remote: Counting objects: 4% (36/876) 2024-12-17T23:58:31.3983139Z remote: Counting objects: 5% (44/876) 2024-12-17T23:58:31.3983588Z remote: Counting objects: 6% (53/876) 2024-12-17T23:58:31.3984041Z remote: Counting objects: 7% (62/876) 2024-12-17T23:58:31.3984488Z remote: Counting objects: 8% (71/876) 2024-12-17T23:58:31.3985060Z remote: Counting objects: 9% (79/876) 2024-12-17T23:58:31.3985511Z remote: Counting objects: 10% (88/876) 2024-12-17T23:58:31.3985944Z remote: Counting objects: 11% (97/876) 2024-12-17T23:58:31.3986362Z remote: Counting objects: 12% (106/876) 2024-12-17T23:58:31.3986801Z remote: Counting objects: 13% (114/876) 2024-12-17T23:58:31.3987243Z remote: Counting objects: 14% (123/876) 2024-12-17T23:58:31.3987677Z remote: Counting objects: 15% (132/876) 2024-12-17T23:58:31.3988113Z remote: Counting objects: 16% (141/876) 2024-12-17T23:58:31.3988561Z remote: Counting objects: 17% (149/876) 2024-12-17T23:58:31.3988977Z remote: Counting objects: 18% (158/876) 2024-12-17T23:58:31.3989412Z remote: Counting objects: 19% (167/876) 2024-12-17T23:58:31.3990004Z remote: Counting objects: 20% (176/876) 2024-12-17T23:58:31.3990447Z remote: Counting objects: 21% (184/876) 2024-12-17T23:58:31.3990888Z remote: Counting objects: 22% (193/876) 2024-12-17T23:58:31.3991308Z remote: Counting objects: 23% (202/876) 2024-12-17T23:58:31.3991747Z remote: Counting objects: 24% (211/876) 2024-12-17T23:58:31.3992267Z remote: Counting objects: 25% (219/876) 2024-12-17T23:58:31.3992704Z remote: Counting objects: 26% (228/876) 2024-12-17T23:58:31.3993196Z remote: Counting objects: 27% (237/876) 2024-12-17T23:58:31.3993641Z remote: Counting objects: 28% (246/876) 2024-12-17T23:58:31.3994058Z remote: Counting objects: 29% (255/876) 2024-12-17T23:58:31.3994503Z remote: Counting objects: 30% (263/876) 2024-12-17T23:58:31.3994941Z remote: Counting objects: 31% (272/876) 2024-12-17T23:58:31.3995378Z remote: Counting objects: 32% (281/876) 2024-12-17T23:58:31.3995821Z remote: Counting objects: 33% (290/876) 2024-12-17T23:58:31.3996239Z remote: Counting objects: 34% (298/876) 2024-12-17T23:58:31.3996678Z remote: Counting objects: 35% (307/876) 2024-12-17T23:58:31.3997118Z remote: Counting objects: 36% (316/876) 2024-12-17T23:58:31.3997559Z remote: Counting objects: 37% (325/876) 2024-12-17T23:58:31.3998004Z remote: Counting objects: 38% (333/876) 2024-12-17T23:58:31.3998448Z remote: Counting objects: 39% (342/876) 2024-12-17T23:58:31.3998899Z remote: Counting objects: 40% (351/876) 2024-12-17T23:58:31.3999340Z remote: Counting objects: 41% (360/876) 2024-12-17T23:58:31.4038450Z remote: Counting objects: 42% (368/876) 2024-12-17T23:58:31.4038967Z remote: Counting objects: 43% (377/876) 2024-12-17T23:58:31.4039400Z remote: Counting objects: 44% (386/876) 2024-12-17T23:58:31.4039850Z remote: Counting objects: 45% (395/876) 2024-12-17T23:58:31.4040303Z remote: Counting objects: 46% (403/876) 2024-12-17T23:58:31.4040749Z remote: Counting objects: 47% (412/876) 2024-12-17T23:58:31.4041191Z remote: Counting objects: 48% (421/876) 2024-12-17T23:58:31.4041610Z remote: Counting objects: 49% (430/876) 2024-12-17T23:58:31.4042057Z remote: Counting objects: 50% (438/876) 2024-12-17T23:58:31.4042498Z remote: Counting objects: 51% (447/876) 2024-12-17T23:58:31.4042943Z remote: Counting objects: 52% (456/876) 2024-12-17T23:58:31.4043386Z remote: Counting objects: 53% (465/876) 2024-12-17T23:58:31.4043828Z remote: Counting objects: 54% (474/876) 2024-12-17T23:58:31.4044250Z remote: Counting objects: 55% (482/876) 2024-12-17T23:58:31.4044928Z remote: Counting objects: 56% (491/876) 2024-12-17T23:58:31.4045379Z remote: Counting objects: 57% (500/876) 2024-12-17T23:58:31.4045821Z remote: Counting objects: 58% (509/876) 2024-12-17T23:58:31.4046269Z remote: Counting objects: 59% (517/876) 2024-12-17T23:58:31.4046689Z remote: Counting objects: 60% (526/876) 2024-12-17T23:58:31.4047131Z remote: Counting objects: 61% (535/876) 2024-12-17T23:58:31.4047577Z remote: Counting objects: 62% (544/876) 2024-12-17T23:58:31.4048051Z remote: Counting objects: 63% (552/876) 2024-12-17T23:58:31.4048496Z remote: Counting objects: 64% (561/876) 2024-12-17T23:58:31.4048938Z remote: Counting objects: 65% (570/876) 2024-12-17T23:58:31.4049373Z remote: Counting objects: 66% (579/876) 2024-12-17T23:58:31.4049788Z remote: Counting objects: 67% (587/876) 2024-12-17T23:58:31.4050376Z remote: Counting objects: 68% (596/876) 2024-12-17T23:58:31.4050874Z remote: Counting objects: 69% (605/876) 2024-12-17T23:58:31.4051303Z remote: Counting objects: 70% (614/876) 2024-12-17T23:58:31.4051732Z remote: Counting objects: 71% (622/876) 2024-12-17T23:58:31.4052139Z remote: Counting objects: 72% (631/876) 2024-12-17T23:58:31.4052674Z remote: Counting objects: 73% (640/876) 2024-12-17T23:58:31.4053104Z remote: Counting objects: 74% (649/876) 2024-12-17T23:58:31.4053535Z remote: Counting objects: 75% (657/876) 2024-12-17T23:58:31.4053965Z remote: Counting objects: 76% (666/876) 2024-12-17T23:58:31.4054376Z remote: Counting objects: 77% (675/876) 2024-12-17T23:58:31.4054809Z remote: Counting objects: 78% (684/876) 2024-12-17T23:58:31.4055239Z remote: Counting objects: 79% (693/876) 2024-12-17T23:58:31.4055673Z remote: Counting objects: 80% (701/876) 2024-12-17T23:58:31.4056105Z remote: Counting objects: 81% (710/876) 2024-12-17T23:58:31.4056516Z remote: Counting objects: 82% (719/876) 2024-12-17T23:58:31.4056953Z remote: Counting objects: 83% (728/876) 2024-12-17T23:58:31.4057387Z remote: Counting objects: 84% (736/876) 2024-12-17T23:58:31.4057812Z remote: Counting objects: 85% (745/876) 2024-12-17T23:58:31.4058245Z remote: Counting objects: 86% (754/876) 2024-12-17T23:58:31.4059030Z remote: Counting objects: 87% (763/876) 2024-12-17T23:58:31.4059469Z remote: Counting objects: 88% (771/876) 2024-12-17T23:58:31.4059980Z remote: Counting objects: 89% (780/876) 2024-12-17T23:58:31.4060435Z remote: Counting objects: 90% (789/876) 2024-12-17T23:58:31.4060895Z remote: Counting objects: 91% (798/876) 2024-12-17T23:58:31.4061352Z remote: Counting objects: 92% (806/876) 2024-12-17T23:58:31.4061785Z remote: Counting objects: 93% (815/876) 2024-12-17T23:58:31.4062243Z remote: Counting objects: 94% (824/876) 2024-12-17T23:58:31.4062691Z remote: Counting objects: 95% (833/876) 2024-12-17T23:58:31.4063149Z remote: Counting objects: 96% (841/876) 2024-12-17T23:58:31.4063603Z remote: Counting objects: 97% (850/876) 2024-12-17T23:58:31.4064060Z remote: Counting objects: 98% (859/876) 2024-12-17T23:58:31.4064498Z remote: Counting objects: 99% (868/876) 2024-12-17T23:58:31.4065068Z remote: Counting objects: 100% (876/876) 2024-12-17T23:58:31.4065630Z remote: Counting objects: 100% (876/876), done. 2024-12-17T23:58:31.4101145Z remote: Compressing objects: 0% (1/352) 2024-12-17T23:58:31.4237446Z remote: Compressing objects: 1% (4/352) 2024-12-17T23:58:31.4317819Z remote: Compressing objects: 2% (8/352) 2024-12-17T23:58:31.4454945Z remote: Compressing objects: 3% (11/352) 2024-12-17T23:58:31.4539410Z remote: Compressing objects: 4% (15/352) 2024-12-17T23:58:31.4612237Z remote: Compressing objects: 5% (18/352) 2024-12-17T23:58:31.4765757Z remote: Compressing objects: 6% (22/352) 2024-12-17T23:58:31.5199169Z remote: Compressing objects: 7% (25/352) 2024-12-17T23:58:31.5453788Z remote: Compressing objects: 8% (29/352) 2024-12-17T23:58:31.6434736Z remote: Compressing objects: 9% (32/352) 2024-12-17T23:58:31.6856174Z remote: Compressing objects: 10% (36/352) 2024-12-17T23:58:31.7317723Z remote: Compressing objects: 11% (39/352) 2024-12-17T23:58:31.7422459Z remote: Compressing objects: 12% (43/352) 2024-12-17T23:58:31.7576479Z remote: Compressing objects: 13% (46/352) 2024-12-17T23:58:31.7644242Z remote: Compressing objects: 14% (50/352) 2024-12-17T23:58:31.7647914Z remote: Compressing objects: 15% (53/352) 2024-12-17T23:58:31.7648819Z remote: Compressing objects: 16% (57/352) 2024-12-17T23:58:31.7649712Z remote: Compressing objects: 17% (60/352) 2024-12-17T23:58:31.7653725Z remote: Compressing objects: 18% (64/352) 2024-12-17T23:58:31.7656314Z remote: Compressing objects: 19% (67/352) 2024-12-17T23:58:31.7657893Z remote: Compressing objects: 20% (71/352) 2024-12-17T23:58:31.7660013Z remote: Compressing objects: 21% (74/352) 2024-12-17T23:58:31.7662561Z remote: Compressing objects: 22% (78/352) 2024-12-17T23:58:31.7675713Z remote: Compressing objects: 23% (81/352) 2024-12-17T23:58:31.7676991Z remote: Compressing objects: 24% (85/352) 2024-12-17T23:58:31.7684395Z remote: Compressing objects: 25% (88/352) 2024-12-17T23:58:31.7684846Z remote: Compressing objects: 26% (92/352) 2024-12-17T23:58:31.7685320Z remote: Compressing objects: 27% (96/352) 2024-12-17T23:58:31.7689369Z remote: Compressing objects: 28% (99/352) 2024-12-17T23:58:31.7701165Z remote: Compressing objects: 29% (103/352) 2024-12-17T23:58:31.7711808Z remote: Compressing objects: 30% (106/352) 2024-12-17T23:58:31.7712300Z remote: Compressing objects: 31% (110/352) 2024-12-17T23:58:31.7712769Z remote: Compressing objects: 32% (113/352) 2024-12-17T23:58:31.7713272Z remote: Compressing objects: 33% (117/352) 2024-12-17T23:58:31.7724278Z remote: Compressing objects: 34% (120/352) 2024-12-17T23:58:31.7725515Z remote: Compressing objects: 35% (124/352) 2024-12-17T23:58:31.7730620Z remote: Compressing objects: 36% (127/352) 2024-12-17T23:58:31.7737639Z remote: Compressing objects: 37% (131/352) 2024-12-17T23:58:31.7745064Z remote: Compressing objects: 38% (134/352) 2024-12-17T23:58:31.7747091Z remote: Compressing objects: 39% (138/352) 2024-12-17T23:58:31.7752076Z remote: Compressing objects: 40% (141/352) 2024-12-17T23:58:31.7756921Z remote: Compressing objects: 41% (145/352) 2024-12-17T23:58:31.7766460Z remote: Compressing objects: 42% (148/352) 2024-12-17T23:58:31.7768049Z remote: Compressing objects: 43% (152/352) 2024-12-17T23:58:31.7770445Z remote: Compressing objects: 44% (155/352) 2024-12-17T23:58:31.7777918Z remote: Compressing objects: 45% (159/352) 2024-12-17T23:58:31.7783627Z remote: Compressing objects: 46% (162/352) 2024-12-17T23:58:31.7787529Z remote: Compressing objects: 47% (166/352) 2024-12-17T23:58:31.7790958Z remote: Compressing objects: 48% (169/352) 2024-12-17T23:58:31.7793743Z remote: Compressing objects: 49% (173/352) 2024-12-17T23:58:31.7795299Z remote: Compressing objects: 50% (176/352) 2024-12-17T23:58:31.7798290Z remote: Compressing objects: 51% (180/352) 2024-12-17T23:58:31.7800225Z remote: Compressing objects: 52% (184/352) 2024-12-17T23:58:31.7803179Z remote: Compressing objects: 53% (187/352) 2024-12-17T23:58:31.7805737Z remote: Compressing objects: 54% (191/352) 2024-12-17T23:58:31.7809064Z remote: Compressing objects: 55% (194/352) 2024-12-17T23:58:31.7811705Z remote: Compressing objects: 56% (198/352) 2024-12-17T23:58:31.7814637Z remote: Compressing objects: 57% (201/352) 2024-12-17T23:58:31.7825847Z remote: Compressing objects: 58% (205/352) 2024-12-17T23:58:31.7826376Z remote: Compressing objects: 59% (208/352) 2024-12-17T23:58:31.7827062Z remote: Compressing objects: 60% (212/352) 2024-12-17T23:58:31.7827531Z remote: Compressing objects: 61% (215/352) 2024-12-17T23:58:31.7829214Z remote: Compressing objects: 62% (219/352) 2024-12-17T23:58:31.7833833Z remote: Compressing objects: 63% (222/352) 2024-12-17T23:58:31.7838281Z remote: Compressing objects: 64% (226/352) 2024-12-17T23:58:31.7842460Z remote: Compressing objects: 65% (229/352) 2024-12-17T23:58:31.7845380Z remote: Compressing objects: 66% (233/352) 2024-12-17T23:58:31.7847805Z remote: Compressing objects: 67% (236/352) 2024-12-17T23:58:31.7848874Z remote: Compressing objects: 68% (240/352) 2024-12-17T23:58:31.7854221Z remote: Compressing objects: 69% (243/352) 2024-12-17T23:58:31.7859582Z remote: Compressing objects: 70% (247/352) 2024-12-17T23:58:31.7860038Z remote: Compressing objects: 71% (250/352) 2024-12-17T23:58:31.7860512Z remote: Compressing objects: 72% (254/352) 2024-12-17T23:58:31.7862324Z remote: Compressing objects: 73% (257/352) 2024-12-17T23:58:31.7863949Z remote: Compressing objects: 74% (261/352) 2024-12-17T23:58:31.7865368Z remote: Compressing objects: 75% (264/352) 2024-12-17T23:58:31.7866943Z remote: Compressing objects: 76% (268/352) 2024-12-17T23:58:31.7867397Z remote: Compressing objects: 77% (272/352) 2024-12-17T23:58:31.7868707Z remote: Compressing objects: 78% (275/352) 2024-12-17T23:58:31.7870220Z remote: Compressing objects: 79% (279/352) 2024-12-17T23:58:31.7872250Z remote: Compressing objects: 80% (282/352) 2024-12-17T23:58:31.7874100Z remote: Compressing objects: 81% (286/352) 2024-12-17T23:58:31.7876893Z remote: Compressing objects: 82% (289/352) 2024-12-17T23:58:31.7879131Z remote: Compressing objects: 83% (293/352) 2024-12-17T23:58:31.7881800Z remote: Compressing objects: 84% (296/352) 2024-12-17T23:58:31.7883681Z remote: Compressing objects: 85% (300/352) 2024-12-17T23:58:31.7886242Z remote: Compressing objects: 86% (303/352) 2024-12-17T23:58:31.7887254Z remote: Compressing objects: 87% (307/352) 2024-12-17T23:58:31.7889243Z remote: Compressing objects: 88% (310/352) 2024-12-17T23:58:31.7890803Z remote: Compressing objects: 89% (314/352) 2024-12-17T23:58:31.7892197Z remote: Compressing objects: 90% (317/352) 2024-12-17T23:58:31.7892963Z remote: Compressing objects: 91% (321/352) 2024-12-17T23:58:31.7895087Z remote: Compressing objects: 92% (324/352) 2024-12-17T23:58:31.7896086Z remote: Compressing objects: 93% (328/352) 2024-12-17T23:58:31.7897052Z remote: Compressing objects: 94% (331/352) 2024-12-17T23:58:31.7897894Z remote: Compressing objects: 95% (335/352) 2024-12-17T23:58:31.7899866Z remote: Compressing objects: 96% (338/352) 2024-12-17T23:58:31.7900633Z remote: Compressing objects: 97% (342/352) 2024-12-17T23:58:31.7901396Z remote: Compressing objects: 98% (345/352) 2024-12-17T23:58:31.7901873Z remote: Compressing objects: 99% (349/352) 2024-12-17T23:58:31.7903350Z remote: Compressing objects: 100% (352/352) 2024-12-17T23:58:31.7903861Z remote: Compressing objects: 100% (352/352), done. 2024-12-17T23:58:55.3810190Z remote: Total 1057261 (delta 697), reused 548 (delta 524), pack-reused 1056385 (from 3) 2024-12-17T23:59:12.2651921Z [command]/usr/bin/git rev-parse --verify --quiet 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7^{object} 2024-12-17T23:59:12.2674513Z 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-17T23:59:12.2676802Z ##[endgroup] 2024-12-17T23:59:12.2677280Z ##[group]Determining the checkout info 2024-12-17T23:59:12.2678095Z ##[endgroup] 2024-12-17T23:59:12.2678829Z ##[group]Checking out the ref 2024-12-17T23:59:12.2681286Z [command]/usr/bin/git checkout --quiet --force 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-17T23:59:13.7761452Z ##[endgroup] 2024-12-17T23:59:13.7762036Z ##[group]Setting up auth for fetching submodules 2024-12-17T23:59:13.7763834Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-12-17T23:59:13.7812689Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-12-17T23:59:13.7834804Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-12-17T23:59:13.7857910Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-12-17T23:59:13.7883935Z ##[endgroup] 2024-12-17T23:59:13.7884415Z ##[group]Fetching submodules 2024-12-17T23:59:13.7884794Z [command]/usr/bin/git submodule sync --recursive 2024-12-17T23:59:13.8208573Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-12-17T23:59:13.8515928Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-12-17T23:59:13.8517430Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-12-17T23:59:13.8518599Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-12-17T23:59:13.8520788Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-12-17T23:59:13.8523869Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2024-12-17T23:59:13.8527105Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-12-17T23:59:13.8529758Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-12-17T23:59:13.8532791Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-12-17T23:59:13.8536138Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel' 2024-12-17T23:59:13.8539541Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-12-17T23:59:13.8542940Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-12-17T23:59:13.8546549Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-12-17T23:59:13.8550054Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-12-17T23:59:13.8553795Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-12-17T23:59:13.8557503Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-12-17T23:59:13.8563080Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-12-17T23:59:13.8568690Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-12-17T23:59:13.8573024Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-12-17T23:59:13.8577012Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-12-17T23:59:13.8581888Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-12-17T23:59:13.8586231Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-12-17T23:59:13.8591045Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-12-17T23:59:13.8595555Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-12-17T23:59:13.8600275Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-12-17T23:59:13.8604933Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-12-17T23:59:13.8609835Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-12-17T23:59:13.8614726Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-12-17T23:59:13.8620540Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-12-17T23:59:13.8626018Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-12-17T23:59:13.8631294Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-12-17T23:59:13.8636843Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-12-17T23:59:13.8642413Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-12-17T23:59:13.8647815Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-12-17T23:59:13.8655020Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-12-17T23:59:13.8661166Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-12-17T23:59:13.8667397Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-12-17T23:59:13.8698110Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-12-17T23:59:14.1843208Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-12-17T23:59:14.4201475Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-12-17T23:59:14.6204765Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-12-17T23:59:14.9015520Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2024-12-17T23:59:15.2280318Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-12-17T23:59:17.0503073Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-12-17T23:59:26.5877020Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-12-17T23:59:27.0141844Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'... 2024-12-17T23:59:29.0998205Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-12-17T23:59:29.5757663Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-12-17T23:59:30.1999168Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-12-17T23:59:31.4070887Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-12-17T23:59:33.5021115Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-12-17T23:59:38.8265178Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-12-17T23:59:40.3061843Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-12-17T23:59:41.4000646Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-12-17T23:59:42.4491048Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-12-17T23:59:42.9703260Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-12-17T23:59:43.3492226Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-12-17T23:59:44.3220440Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-12-17T23:59:44.7179346Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-12-17T23:59:45.0390110Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-12-17T23:59:46.7197538Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-12-17T23:59:47.5532346Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-12-17T23:59:47.9227044Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-12-17T23:59:54.3975456Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-12-17T23:59:56.4491719Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-12-18T00:00:00.0062857Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-12-18T00:00:00.2579430Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-12-18T00:00:08.5982544Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-12-18T00:00:08.8044755Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-12-18T00:00:09.3444184Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-12-18T00:00:10.3550867Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-12-18T00:00:10.6655715Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-12-18T00:00:11.3401693Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-12-18T00:00:11.7536580Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-12-18T00:00:11.7659149Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-12-18T00:00:11.7756145Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-12-18T00:00:11.8011566Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-12-18T00:00:11.8360325Z Submodule path 'third_party/NVTX': checked out 'e170594ac7cf1dac584da473d4ca9301087090c1' 2024-12-18T00:00:11.8730688Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-12-18T00:00:12.6027471Z Submodule path 'third_party/XNNPACK': checked out '4ea82e595b36106653175dcb04b2aa532660d0d8' 2024-12-18T00:00:12.6268215Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-12-18T00:00:12.8624515Z Submodule path 'third_party/composable_kernel': checked out '50ee4267e27b875d149e642f4cebd47be1dc3b57' 2024-12-18T00:00:12.9111434Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-12-18T00:00:13.0109896Z Submodule path 'third_party/cpuinfo': checked out '1e83a2fdd3102f65c6f1fb602c1b320486218a99' 2024-12-18T00:00:13.0446773Z Submodule path 'third_party/cudnn_frontend': checked out '936021bfed8c91dc416af1588b2c4eca631a9e45' 2024-12-18T00:00:13.5617397Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-12-18T00:00:13.8243320Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-12-18T00:00:13.9206115Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-12-18T00:00:13.9223183Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:00:13.9225923Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:00:13.9228613Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:00:13.9231370Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:00:13.9234222Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:00:13.9262554Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-12-18T00:00:14.7467393Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-12-18T00:00:15.3553788Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-12-18T00:00:17.4042128Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-12-18T00:00:18.3643688Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-12-18T00:00:18.7283885Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-12-18T00:00:18.8278945Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-12-18T00:00:19.2393709Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-12-18T00:00:19.3033944Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-12-18T00:00:19.3160722Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-12-18T00:00:19.4493139Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-12-18T00:00:19.4895466Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-12-18T00:00:19.5297400Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-12-18T00:00:19.5562986Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-12-18T00:00:19.6006571Z Submodule path 'third_party/googletest': checked out 'b514bdc898e2951020cbdca1304b75f5950d1f59' 2024-12-18T00:00:19.6141097Z Submodule path 'third_party/ideep': checked out 'c7ccd5bdbe5434ba156f4e856dcef0601637334b' 2024-12-18T00:00:19.6159064Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-12-18T00:00:19.6182705Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-12-18T00:00:32.6220304Z Submodule path 'third_party/ideep/mkl-dnn': checked out '66f0cb9eb66affd2da3bf5f8d897376f04aae6af' 2024-12-18T00:00:32.6414544Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-12-18T00:00:32.7307407Z Submodule path 'third_party/kineto': checked out '338140f58a28d599da3434ced4fd2d75dd1a213d' 2024-12-18T00:00:32.7325730Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:00:32.7327576Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:00:32.7330300Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:00:32.7359008Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-12-18T00:00:33.5677198Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-12-18T00:00:34.6653683Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-12-18T00:00:35.7665452Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-12-18T00:00:35.7684651Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:00:35.7686346Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:00:35.7687732Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:00:35.7689945Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:00:35.7692699Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:00:35.7695816Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:00:35.7698802Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:00:35.7702309Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:00:35.7734466Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-12-18T00:00:36.7233272Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-12-18T00:00:37.0922084Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-12-18T00:00:38.1599993Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-12-18T00:00:38.4718707Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-12-18T00:00:39.0014155Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-12-18T00:00:39.9276447Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-12-18T00:00:46.4019924Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-12-18T00:00:46.8560103Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-12-18T00:00:46.8756927Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-12-18T00:00:46.9149443Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-12-18T00:00:46.9293679Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-12-18T00:00:46.9309584Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:00:46.9338923Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-12-18T00:00:47.6702787Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-12-18T00:00:47.6904240Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-12-18T00:00:47.7330762Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-12-18T00:00:47.8464948Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-12-18T00:00:47.8645722Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-12-18T00:00:47.9065564Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-12-18T00:00:47.9669756Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-12-18T00:00:48.0049081Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-12-18T00:00:48.0357904Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-12-18T00:00:48.1522249Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-12-18T00:00:48.5284340Z Submodule path 'third_party/onnx': checked out 'b8baa8446686496da4cc8fda09f2b6fe65c2a02c' 2024-12-18T00:00:48.5321384Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-12-18T00:00:48.5348820Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-12-18T00:00:50.6412235Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-12-18T00:00:50.7150194Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-12-18T00:00:50.7170816Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:00:50.7172297Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:00:50.7173653Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:00:50.7175995Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:00:50.7179066Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:00:50.7182207Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:00:50.7185198Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:00:50.7188212Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:00:50.7218907Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-12-18T00:00:51.1604995Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-12-18T00:00:52.1062576Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-12-18T00:00:52.4791789Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-12-18T00:00:58.9919606Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-12-18T00:00:59.3751620Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-12-18T00:00:59.5961176Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-12-18T00:00:59.9194806Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-12-18T00:01:06.3582253Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-12-18T00:01:06.4022834Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-12-18T00:01:06.4192450Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-12-18T00:01:06.5288660Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-12-18T00:01:06.5424416Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-12-18T00:01:06.5582172Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-12-18T00:01:06.5745613Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-12-18T00:01:06.5763684Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:01:06.5765283Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:01:06.5792721Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-12-18T00:01:08.2574243Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-12-18T00:01:09.4684340Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-12-18T00:01:09.5161420Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-12-18T00:01:09.9786541Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-12-18T00:01:09.9918465Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-12-18T00:01:10.2775076Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-12-18T00:01:10.2798933Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:01:10.2800328Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-12-18T00:01:10.2828179Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-12-18T00:01:10.7393991Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-12-18T00:01:11.6724929Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-12-18T00:01:11.7462603Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-12-18T00:01:11.7555260Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-12-18T00:01:11.7684933Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-12-18T00:01:11.8048114Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2024-12-18T00:01:11.8345870Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-12-18T00:01:11.8779706Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-12-18T00:01:11.9046950Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-12-18T00:01:11.9063079Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:01:11.9065246Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:01:11.9067793Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:01:11.9070593Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:01:11.9099341Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-12-18T00:01:12.8526982Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-12-18T00:01:13.0958142Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-12-18T00:01:15.2529421Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-12-18T00:01:16.3401386Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-12-18T00:01:16.3559319Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-12-18T00:01:16.4180173Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-12-18T00:01:16.4470048Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-12-18T00:01:16.4487594Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:01:16.4515629Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-12-18T00:01:16.6944757Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-12-18T00:01:16.6986676Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-12-18T00:01:16.7319190Z Entering 'android/libs/fbjni' 2024-12-18T00:01:16.7363985Z Entering 'third_party/FP16' 2024-12-18T00:01:16.7407565Z Entering 'third_party/FXdiv' 2024-12-18T00:01:16.7453180Z Entering 'third_party/NNPACK' 2024-12-18T00:01:16.7497660Z Entering 'third_party/NVTX' 2024-12-18T00:01:16.7541084Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T00:01:16.7583378Z Entering 'third_party/XNNPACK' 2024-12-18T00:01:16.7646230Z Entering 'third_party/benchmark' 2024-12-18T00:01:16.7691166Z Entering 'third_party/composable_kernel' 2024-12-18T00:01:16.7741578Z Entering 'third_party/cpp-httplib' 2024-12-18T00:01:16.7783771Z Entering 'third_party/cpuinfo' 2024-12-18T00:01:16.7826190Z Entering 'third_party/cudnn_frontend' 2024-12-18T00:01:16.7869406Z Entering 'third_party/cutlass' 2024-12-18T00:01:16.7922674Z Entering 'third_party/eigen' 2024-12-18T00:01:16.7970292Z Entering 'third_party/fbgemm' 2024-12-18T00:01:16.8016876Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:01:16.8060890Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:01:16.8102933Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:01:16.8153482Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:01:16.8196383Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:01:16.8240187Z Entering 'third_party/flatbuffers' 2024-12-18T00:01:16.8285025Z Entering 'third_party/fmt' 2024-12-18T00:01:16.8333852Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:01:16.8377525Z Entering 'third_party/gloo' 2024-12-18T00:01:16.8420192Z Entering 'third_party/googletest' 2024-12-18T00:01:16.8463335Z Entering 'third_party/ideep' 2024-12-18T00:01:16.8504173Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T00:01:16.8557345Z Entering 'third_party/ittapi' 2024-12-18T00:01:16.8598758Z Entering 'third_party/kineto' 2024-12-18T00:01:16.8643273Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:01:16.8685696Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:01:16.8727906Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:01:16.8771984Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:01:16.8814395Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:01:16.8856737Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:01:16.8900820Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:01:16.8942264Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:01:16.8987138Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:01:16.9030440Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:01:16.9078520Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:01:16.9134099Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:01:16.9179083Z Entering 'third_party/mimalloc' 2024-12-18T00:01:16.9220865Z Entering 'third_party/nccl/nccl' 2024-12-18T00:01:16.9262770Z Entering 'third_party/nlohmann' 2024-12-18T00:01:16.9305068Z Entering 'third_party/onnx' 2024-12-18T00:01:16.9366582Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T00:01:16.9412683Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T00:01:16.9466792Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:01:16.9506889Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:01:16.9547936Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:01:16.9588485Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:01:16.9632038Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:01:16.9675802Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:01:16.9717224Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:01:16.9758545Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:01:16.9803156Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:01:16.9848155Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:01:16.9919298Z Entering 'third_party/pocketfft' 2024-12-18T00:01:16.9963094Z Entering 'third_party/protobuf' 2024-12-18T00:01:17.0007834Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:01:17.0053258Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T00:01:17.0098334Z Entering 'third_party/psimd' 2024-12-18T00:01:17.0140854Z Entering 'third_party/pthreadpool' 2024-12-18T00:01:17.0181676Z Entering 'third_party/pybind11' 2024-12-18T00:01:17.0224196Z Entering 'third_party/python-peachpy' 2024-12-18T00:01:17.0265769Z Entering 'third_party/sleef' 2024-12-18T00:01:17.0307793Z Entering 'third_party/tensorpipe' 2024-12-18T00:01:17.0357433Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:01:17.0398959Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:01:17.0439883Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:01:17.0480906Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:01:17.0520964Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:01:17.0578048Z ##[endgroup] 2024-12-18T00:01:17.0578800Z ##[group]Persisting credentials for submodules 2024-12-18T00:01:17.0582198Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-12-18T00:01:17.0894976Z Entering 'android/libs/fbjni' 2024-12-18T00:01:17.0955883Z Entering 'third_party/FP16' 2024-12-18T00:01:17.1014320Z Entering 'third_party/FXdiv' 2024-12-18T00:01:17.1070645Z Entering 'third_party/NNPACK' 2024-12-18T00:01:17.1128469Z Entering 'third_party/NVTX' 2024-12-18T00:01:17.1186283Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T00:01:17.1244985Z Entering 'third_party/XNNPACK' 2024-12-18T00:01:17.1322672Z Entering 'third_party/benchmark' 2024-12-18T00:01:17.1380188Z Entering 'third_party/composable_kernel' 2024-12-18T00:01:17.1445800Z Entering 'third_party/cpp-httplib' 2024-12-18T00:01:17.1500674Z Entering 'third_party/cpuinfo' 2024-12-18T00:01:17.1562332Z Entering 'third_party/cudnn_frontend' 2024-12-18T00:01:17.1619040Z Entering 'third_party/cutlass' 2024-12-18T00:01:17.1687175Z Entering 'third_party/eigen' 2024-12-18T00:01:17.1756080Z Entering 'third_party/fbgemm' 2024-12-18T00:01:17.1813774Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:01:17.1869182Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:01:17.1924783Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:01:17.1987857Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:01:17.2045665Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:01:17.2106978Z Entering 'third_party/flatbuffers' 2024-12-18T00:01:17.2167804Z Entering 'third_party/fmt' 2024-12-18T00:01:17.2226302Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:01:17.2281845Z Entering 'third_party/gloo' 2024-12-18T00:01:17.2340614Z Entering 'third_party/googletest' 2024-12-18T00:01:17.2399104Z Entering 'third_party/ideep' 2024-12-18T00:01:17.2456404Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T00:01:17.2526304Z Entering 'third_party/ittapi' 2024-12-18T00:01:17.2583447Z Entering 'third_party/kineto' 2024-12-18T00:01:17.2639439Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:01:17.2700020Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:01:17.2763512Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:01:17.2818429Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:01:17.2878021Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:01:17.2935051Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:01:17.2990793Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:01:17.3047040Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:01:17.3103607Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:01:17.3162061Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:01:17.3220977Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:01:17.3277935Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:01:17.3341458Z Entering 'third_party/mimalloc' 2024-12-18T00:01:17.3398113Z Entering 'third_party/nccl/nccl' 2024-12-18T00:01:17.3456397Z Entering 'third_party/nlohmann' 2024-12-18T00:01:17.3517899Z Entering 'third_party/onnx' 2024-12-18T00:01:17.3591618Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T00:01:17.3649428Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T00:01:17.3706255Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:01:17.3762089Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:01:17.3817928Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:01:17.3877009Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:01:17.3939168Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:01:17.3997974Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:01:17.4058844Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:01:17.4117464Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:01:17.4176121Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:01:17.4238709Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:01:17.4318542Z Entering 'third_party/pocketfft' 2024-12-18T00:01:17.4374852Z Entering 'third_party/protobuf' 2024-12-18T00:01:17.4433289Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:01:17.4491505Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T00:01:17.4548130Z Entering 'third_party/psimd' 2024-12-18T00:01:17.4603870Z Entering 'third_party/pthreadpool' 2024-12-18T00:01:17.4657104Z Entering 'third_party/pybind11' 2024-12-18T00:01:17.4714729Z Entering 'third_party/python-peachpy' 2024-12-18T00:01:17.4772129Z Entering 'third_party/sleef' 2024-12-18T00:01:17.4839052Z Entering 'third_party/tensorpipe' 2024-12-18T00:01:17.4896070Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:01:17.4950754Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:01:17.5005176Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:01:17.5063212Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:01:17.5117551Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:01:17.5192339Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-12-18T00:01:17.5497850Z Entering 'android/libs/fbjni' 2024-12-18T00:01:17.5550206Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-12-18T00:01:17.5568671Z Entering 'third_party/FP16' 2024-12-18T00:01:17.5619767Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-12-18T00:01:17.5637209Z Entering 'third_party/FXdiv' 2024-12-18T00:01:17.5688767Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-12-18T00:01:17.5703972Z Entering 'third_party/NNPACK' 2024-12-18T00:01:17.5757210Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-12-18T00:01:17.5775384Z Entering 'third_party/NVTX' 2024-12-18T00:01:17.5830753Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2024-12-18T00:01:17.5847905Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T00:01:17.5903177Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-12-18T00:01:17.5921267Z Entering 'third_party/XNNPACK' 2024-12-18T00:01:17.5973891Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-12-18T00:01:17.6008444Z Entering 'third_party/benchmark' 2024-12-18T00:01:17.6060204Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-12-18T00:01:17.6078177Z Entering 'third_party/composable_kernel' 2024-12-18T00:01:17.6129388Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2024-12-18T00:01:17.6155302Z Entering 'third_party/cpp-httplib' 2024-12-18T00:01:17.6207698Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-12-18T00:01:17.6223324Z Entering 'third_party/cpuinfo' 2024-12-18T00:01:17.6276732Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-12-18T00:01:17.6294506Z Entering 'third_party/cudnn_frontend' 2024-12-18T00:01:17.6348183Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-12-18T00:01:17.6365659Z Entering 'third_party/cutlass' 2024-12-18T00:01:17.6417057Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-12-18T00:01:17.6444122Z Entering 'third_party/eigen' 2024-12-18T00:01:17.6495081Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-12-18T00:01:17.6516526Z Entering 'third_party/fbgemm' 2024-12-18T00:01:17.6571901Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-12-18T00:01:17.6586601Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:01:17.6641228Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-12-18T00:01:17.6659570Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:01:17.6714310Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-12-18T00:01:17.6730516Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:01:17.6781570Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-12-18T00:01:17.6806313Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:01:17.6857889Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:01:17.6876631Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:01:17.6927458Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-12-18T00:01:17.6944538Z Entering 'third_party/flatbuffers' 2024-12-18T00:01:17.6997735Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-12-18T00:01:17.7018179Z Entering 'third_party/fmt' 2024-12-18T00:01:17.7070280Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-12-18T00:01:17.7087948Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:01:17.7140575Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-12-18T00:01:17.7156814Z Entering 'third_party/gloo' 2024-12-18T00:01:17.7208338Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-12-18T00:01:17.7224639Z Entering 'third_party/googletest' 2024-12-18T00:01:17.7276976Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:01:17.7294206Z Entering 'third_party/ideep' 2024-12-18T00:01:17.7345818Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-12-18T00:01:17.7363062Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T00:01:17.7423280Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-12-18T00:01:17.7450561Z Entering 'third_party/ittapi' 2024-12-18T00:01:17.7501160Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-12-18T00:01:17.7518820Z Entering 'third_party/kineto' 2024-12-18T00:01:17.7575278Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-12-18T00:01:17.7592450Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:01:17.7645895Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-12-18T00:01:17.7660656Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:01:17.7711616Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-12-18T00:01:17.7729493Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:01:17.7781659Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-12-18T00:01:17.7800860Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:01:17.7852686Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-12-18T00:01:17.7868919Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:01:17.7921342Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-12-18T00:01:17.7938973Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:01:17.7990484Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-12-18T00:01:17.8009663Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:01:17.8060899Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-12-18T00:01:17.8078622Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:01:17.8128669Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:01:17.8145959Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:01:17.8197967Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-12-18T00:01:17.8216312Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:01:17.8266751Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-12-18T00:01:17.8286901Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:01:17.8338115Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-12-18T00:01:17.8354850Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:01:17.8408141Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-12-18T00:01:17.8425381Z Entering 'third_party/mimalloc' 2024-12-18T00:01:17.8477967Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-12-18T00:01:17.8495830Z Entering 'third_party/nccl/nccl' 2024-12-18T00:01:17.8549009Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-12-18T00:01:17.8567138Z Entering 'third_party/nlohmann' 2024-12-18T00:01:17.8618773Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-12-18T00:01:17.8639797Z Entering 'third_party/onnx' 2024-12-18T00:01:17.8690599Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-12-18T00:01:17.8723614Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T00:01:17.8775412Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-12-18T00:01:17.8798529Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T00:01:17.8851951Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-12-18T00:01:17.8869364Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:01:17.8921172Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-12-18T00:01:17.8938982Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:01:17.8989252Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:01:17.9007325Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:01:17.9058607Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-12-18T00:01:17.9077142Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:01:17.9126934Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-12-18T00:01:17.9144012Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:01:17.9196283Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-12-18T00:01:17.9213228Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:01:17.9263835Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-12-18T00:01:17.9282073Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:01:17.9331560Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-12-18T00:01:17.9346583Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:01:17.9400715Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-12-18T00:01:17.9419196Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:01:17.9471379Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-12-18T00:01:17.9492279Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:01:17.9543412Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-12-18T00:01:17.9580780Z Entering 'third_party/pocketfft' 2024-12-18T00:01:17.9631443Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-12-18T00:01:17.9648185Z Entering 'third_party/protobuf' 2024-12-18T00:01:17.9701145Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-12-18T00:01:17.9721095Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:01:17.9772911Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-12-18T00:01:17.9787899Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T00:01:17.9841457Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:01:17.9860584Z Entering 'third_party/psimd' 2024-12-18T00:01:17.9912369Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-12-18T00:01:17.9927476Z Entering 'third_party/pthreadpool' 2024-12-18T00:01:17.9980988Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-12-18T00:01:17.9996900Z Entering 'third_party/pybind11' 2024-12-18T00:01:18.0048162Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-12-18T00:01:18.0063974Z Entering 'third_party/python-peachpy' 2024-12-18T00:01:18.0115730Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-12-18T00:01:18.0131794Z Entering 'third_party/sleef' 2024-12-18T00:01:18.0182950Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-12-18T00:01:18.0200763Z Entering 'third_party/tensorpipe' 2024-12-18T00:01:18.0253294Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-12-18T00:01:18.0269330Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:01:18.0320875Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:01:18.0339249Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:01:18.0388787Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-12-18T00:01:18.0406589Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:01:18.0458842Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-12-18T00:01:18.0478782Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:01:18.0527420Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-12-18T00:01:18.0542343Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:01:18.0594242Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-12-18T00:01:18.1263879Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-12-18T00:01:18.1570524Z Entering 'android/libs/fbjni' 2024-12-18T00:01:18.1613035Z Entering 'third_party/FP16' 2024-12-18T00:01:18.1656474Z Entering 'third_party/FXdiv' 2024-12-18T00:01:18.1698956Z Entering 'third_party/NNPACK' 2024-12-18T00:01:18.1740506Z Entering 'third_party/NVTX' 2024-12-18T00:01:18.1782070Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T00:01:18.1823882Z Entering 'third_party/XNNPACK' 2024-12-18T00:01:18.1883652Z Entering 'third_party/benchmark' 2024-12-18T00:01:18.1924285Z Entering 'third_party/composable_kernel' 2024-12-18T00:01:18.1976459Z Entering 'third_party/cpp-httplib' 2024-12-18T00:01:18.2020897Z Entering 'third_party/cpuinfo' 2024-12-18T00:01:18.2064562Z Entering 'third_party/cudnn_frontend' 2024-12-18T00:01:18.2107047Z Entering 'third_party/cutlass' 2024-12-18T00:01:18.2160665Z Entering 'third_party/eigen' 2024-12-18T00:01:18.2205240Z Entering 'third_party/fbgemm' 2024-12-18T00:01:18.2249063Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:01:18.2291245Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:01:18.2341510Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:01:18.2390793Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:01:18.2433052Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:01:18.2478569Z Entering 'third_party/flatbuffers' 2024-12-18T00:01:18.2525293Z Entering 'third_party/fmt' 2024-12-18T00:01:18.2575969Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:01:18.2617176Z Entering 'third_party/gloo' 2024-12-18T00:01:18.2659058Z Entering 'third_party/googletest' 2024-12-18T00:01:18.2700873Z Entering 'third_party/ideep' 2024-12-18T00:01:18.2742996Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T00:01:18.2796195Z Entering 'third_party/ittapi' 2024-12-18T00:01:18.2839574Z Entering 'third_party/kineto' 2024-12-18T00:01:18.2881188Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:01:18.2923671Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:01:18.2969290Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:01:18.3015689Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:01:18.3058581Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:01:18.3103564Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:01:18.3148024Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:01:18.3195176Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:01:18.3239069Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:01:18.3281207Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:01:18.3324720Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:01:18.3367345Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:01:18.3412284Z Entering 'third_party/mimalloc' 2024-12-18T00:01:18.3456543Z Entering 'third_party/nccl/nccl' 2024-12-18T00:01:18.3500519Z Entering 'third_party/nlohmann' 2024-12-18T00:01:18.3543606Z Entering 'third_party/onnx' 2024-12-18T00:01:18.3603544Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T00:01:18.3648344Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T00:01:18.3693750Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:01:18.3737222Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:01:18.3779567Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:01:18.3820755Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:01:18.3866014Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:01:18.3906701Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:01:18.3948290Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:01:18.3988791Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:01:18.4032201Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:01:18.4080641Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:01:18.4142440Z Entering 'third_party/pocketfft' 2024-12-18T00:01:18.4183648Z Entering 'third_party/protobuf' 2024-12-18T00:01:18.4228571Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:01:18.4269990Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T00:01:18.4322412Z Entering 'third_party/psimd' 2024-12-18T00:01:18.4365203Z Entering 'third_party/pthreadpool' 2024-12-18T00:01:18.4406801Z Entering 'third_party/pybind11' 2024-12-18T00:01:18.4449185Z Entering 'third_party/python-peachpy' 2024-12-18T00:01:18.4493103Z Entering 'third_party/sleef' 2024-12-18T00:01:18.4537100Z Entering 'third_party/tensorpipe' 2024-12-18T00:01:18.4579363Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:01:18.4620246Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:01:18.4661918Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:01:18.4706523Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:01:18.4747237Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:01:18.4804080Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-12-18T00:01:18.5107916Z Entering 'android/libs/fbjni' 2024-12-18T00:01:18.5154448Z Entering 'third_party/FP16' 2024-12-18T00:01:18.5196597Z Entering 'third_party/FXdiv' 2024-12-18T00:01:18.5238690Z Entering 'third_party/NNPACK' 2024-12-18T00:01:18.5280829Z Entering 'third_party/NVTX' 2024-12-18T00:01:18.5323538Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T00:01:18.5366547Z Entering 'third_party/XNNPACK' 2024-12-18T00:01:18.5421992Z Entering 'third_party/benchmark' 2024-12-18T00:01:18.5464585Z Entering 'third_party/composable_kernel' 2024-12-18T00:01:18.5515805Z Entering 'third_party/cpp-httplib' 2024-12-18T00:01:18.5560507Z Entering 'third_party/cpuinfo' 2024-12-18T00:01:18.5602638Z Entering 'third_party/cudnn_frontend' 2024-12-18T00:01:18.5645164Z Entering 'third_party/cutlass' 2024-12-18T00:01:18.5695973Z Entering 'third_party/eigen' 2024-12-18T00:01:18.5742354Z Entering 'third_party/fbgemm' 2024-12-18T00:01:18.5784809Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:01:18.5828100Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:01:18.5879060Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:01:18.5937300Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:01:18.5980176Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:01:18.6023056Z Entering 'third_party/flatbuffers' 2024-12-18T00:01:18.6067835Z Entering 'third_party/fmt' 2024-12-18T00:01:18.6115450Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:01:18.6159209Z Entering 'third_party/gloo' 2024-12-18T00:01:18.6202307Z Entering 'third_party/googletest' 2024-12-18T00:01:18.6245027Z Entering 'third_party/ideep' 2024-12-18T00:01:18.6285854Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T00:01:18.6334220Z Entering 'third_party/ittapi' 2024-12-18T00:01:18.6377593Z Entering 'third_party/kineto' 2024-12-18T00:01:18.6419801Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:01:18.6465659Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:01:18.6516025Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:01:18.6560271Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:01:18.6602185Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:01:18.6709362Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:01:18.6762445Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:01:18.6805057Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:01:18.6846848Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:01:18.6891018Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:01:18.6937982Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:01:18.6980347Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:01:18.7022370Z Entering 'third_party/mimalloc' 2024-12-18T00:01:18.7064872Z Entering 'third_party/nccl/nccl' 2024-12-18T00:01:18.7107237Z Entering 'third_party/nlohmann' 2024-12-18T00:01:18.7150978Z Entering 'third_party/onnx' 2024-12-18T00:01:18.7210831Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T00:01:18.7261349Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T00:01:18.7304796Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:01:18.7347296Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:01:18.7388682Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:01:18.7430045Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:01:18.7475800Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:01:18.7516580Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:01:18.7560072Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:01:18.7601191Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:01:18.7644136Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:01:18.7687538Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:01:18.7753962Z Entering 'third_party/pocketfft' 2024-12-18T00:01:18.7798710Z Entering 'third_party/protobuf' 2024-12-18T00:01:18.7847824Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:01:18.7893793Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T00:01:18.7946551Z Entering 'third_party/psimd' 2024-12-18T00:01:18.7998241Z Entering 'third_party/pthreadpool' 2024-12-18T00:01:18.8040767Z Entering 'third_party/pybind11' 2024-12-18T00:01:18.8082598Z Entering 'third_party/python-peachpy' 2024-12-18T00:01:18.8123749Z Entering 'third_party/sleef' 2024-12-18T00:01:18.8166446Z Entering 'third_party/tensorpipe' 2024-12-18T00:01:18.8209744Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:01:18.8253907Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:01:18.8295498Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:01:18.8339660Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:01:18.8380230Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:01:18.8433748Z ##[endgroup] 2024-12-18T00:01:18.8464195Z [command]/usr/bin/git log -1 --format='%H' 2024-12-18T00:01:18.8486254Z '0cdf8b1d09254cfda66191d1bd01e3041c3c76f7' 2024-12-18T00:01:18.8657695Z Prepare all required actions 2024-12-18T00:01:18.8658264Z Getting action download info 2024-12-18T00:01:19.0524710Z ##[group]Run ./.github/actions/setup-linux 2024-12-18T00:01:19.0525095Z env: 2024-12-18T00:01:19.0525341Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:19.0525653Z ##[endgroup] 2024-12-18T00:01:19.0580998Z ##[group]Run set -euo pipefail 2024-12-18T00:01:19.0581429Z set -euo pipefail 2024-12-18T00:01:19.0581775Z function get_ec2_metadata() { 2024-12-18T00:01:19.0582240Z  # Pulled from instance metadata endpoint for EC2 2024-12-18T00:01:19.0583003Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-12-18T00:01:19.0583680Z  category=$1 2024-12-18T00:01:19.0584118Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-12-18T00:01:19.0584658Z  runner_name_str=i-0a62689d000b23974 2024-12-18T00:01:19.0585107Z  if [[ -f /.inarc ]]; then 2024-12-18T00:01:19.0585530Z  echo "ARC Runner, no info on ec2 metadata" 2024-12-18T00:01:19.0585986Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-12-18T00:01:19.0586556Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-12-18T00:01:19.0587079Z  else 2024-12-18T00:01:19.0588103Z  curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-12-18T00:01:19.0589201Z  fi 2024-12-18T00:01:19.0589469Z } 2024-12-18T00:01:19.0589774Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-12-18T00:01:19.0590289Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-12-18T00:01:19.0590866Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-12-18T00:01:19.0591497Z echo "system info $(uname -a)" 2024-12-18T00:01:19.0599768Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:19.0600181Z env: 2024-12-18T00:01:19.0600407Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:19.0600691Z ##[endgroup] 2024-12-18T00:01:19.0766457Z ami-id: ami-0fff1b9a61dec8a5f 2024-12-18T00:01:19.0877968Z instance-id: i-0a62689d000b23974 2024-12-18T00:01:19.0982542Z instance-type: g4dn.12xlarge 2024-12-18T00:01:19.0993806Z system info Linux ip-10-0-44-187.ec2.internal 6.1.109-118.189.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 10 08:59:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-12-18T00:01:19.1026533Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T00:01:19.1027637Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T00:01:19.1034968Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:19.1035389Z env: 2024-12-18T00:01:19.1035608Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:19.1035893Z ##[endgroup] 2024-12-18T00:01:19.1097623Z ##[group]Run if systemctl is-active --quiet docker; then 2024-12-18T00:01:19.1098097Z if systemctl is-active --quiet docker; then 2024-12-18T00:01:19.1098510Z  echo "Docker daemon is running..."; 2024-12-18T00:01:19.1098946Z else 2024-12-18T00:01:19.1099525Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-12-18T00:01:19.1100053Z fi 2024-12-18T00:01:19.1105826Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:19.1106279Z env: 2024-12-18T00:01:19.1106537Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:19.1106833Z ##[endgroup] 2024-12-18T00:01:19.1189073Z Docker daemon is running... 2024-12-18T00:01:19.1251555Z ##[group]Run nick-fields/retry@v3.0.0 2024-12-18T00:01:19.1252034Z with: 2024-12-18T00:01:19.1252264Z shell: bash 2024-12-18T00:01:19.1252638Z timeout_minutes: 5 2024-12-18T00:01:19.1252913Z max_attempts: 3 2024-12-18T00:01:19.1253165Z retry_wait_seconds: 30 2024-12-18T00:01:19.1255695Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-12-18T00:01:19.1258254Z polling_interval_seconds: 1 2024-12-18T00:01:19.1258575Z warning_on_retry: true 2024-12-18T00:01:19.1258962Z continue_on_error: false 2024-12-18T00:01:19.1259432Z env: 2024-12-18T00:01:19.1259685Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:19.1260043Z AWS_RETRY_MODE: standard 2024-12-18T00:01:19.1260356Z AWS_MAX_ATTEMPTS: 5 2024-12-18T00:01:19.1260649Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T00:01:19.1260977Z ##[endgroup] 2024-12-18T00:01:20.4456452Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-12-18T00:01:20.4457217Z Configure a credential helper to remove this warning. See 2024-12-18T00:01:20.4457897Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-12-18T00:01:20.4458348Z 2024-12-18T00:01:20.4458480Z Login Succeeded 2024-12-18T00:01:21.3290020Z Command completed after 1 attempt(s). 2024-12-18T00:01:21.3390893Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-12-18T00:01:21.3391764Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-12-18T00:01:21.3392288Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-12-18T00:01:21.3401410Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:21.3401839Z env: 2024-12-18T00:01:21.3402072Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:21.3402370Z ##[endgroup] 2024-12-18T00:01:21.3499975Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T00:01:21.3500673Z # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T00:01:21.3501201Z # shellcheck disable=SC2046 2024-12-18T00:01:21.3501625Z docker stop $(docker ps -q) || true 2024-12-18T00:01:21.3502042Z # Prune all of the docker images 2024-12-18T00:01:21.3502452Z docker system prune -af 2024-12-18T00:01:21.3508771Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:21.3509235Z env: 2024-12-18T00:01:21.3509510Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:21.3509817Z ##[endgroup] 2024-12-18T00:01:21.3816898Z "docker stop" requires at least 1 argument. 2024-12-18T00:01:21.3817975Z See 'docker stop --help'. 2024-12-18T00:01:21.3818277Z 2024-12-18T00:01:21.3818504Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-12-18T00:01:21.3818960Z 2024-12-18T00:01:21.3819294Z Stop one or more running containers 2024-12-18T00:01:21.3994779Z Total reclaimed space: 0B 2024-12-18T00:01:21.4033494Z ##[group]Run set +e 2024-12-18T00:01:21.4033943Z set +e 2024-12-18T00:01:21.4034206Z set -x 2024-12-18T00:01:21.4034458Z  2024-12-18T00:01:21.4034722Z PT_DOMAIN=download.pytorch.org 2024-12-18T00:01:21.4035373Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-12-18T00:01:21.4036229Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-12-18T00:01:21.4036940Z # one is returned at random 2024-12-18T00:01:21.4037361Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-12-18T00:01:21.4037906Z  2024-12-18T00:01:21.4038274Z if [ -z "${RESOLVED_IP}" ]; then 2024-12-18T00:01:21.4038744Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-12-18T00:01:21.4039313Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-12-18T00:01:21.4039739Z  2024-12-18T00:01:21.4040001Z  if [ -z "${RESOLVED_IP}" ]; then 2024-12-18T00:01:21.4040417Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-12-18T00:01:21.4040792Z  exit 1 2024-12-18T00:01:21.4041045Z  fi 2024-12-18T00:01:21.4041279Z fi 2024-12-18T00:01:21.4041505Z  2024-12-18T00:01:21.4041782Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-12-18T00:01:21.4042158Z  # Clean up any old records first 2024-12-18T00:01:21.4042541Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-12-18T00:01:21.4042890Z fi 2024-12-18T00:01:21.4043121Z  2024-12-18T00:01:21.4043457Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-12-18T00:01:21.4043867Z cat /etc/hosts 2024-12-18T00:01:21.4049953Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:21.4050385Z env: 2024-12-18T00:01:21.4050630Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:21.4050925Z ##[endgroup] 2024-12-18T00:01:21.4074822Z + PT_DOMAIN=download.pytorch.org 2024-12-18T00:01:21.4081406Z ++ dig -4 +short download.pytorch.org 2024-12-18T00:01:21.4082350Z ++ tail -n1 2024-12-18T00:01:21.4512994Z + RESOLVED_IP=3.167.99.70 2024-12-18T00:01:21.4513388Z + '[' -z 3.167.99.70 ']' 2024-12-18T00:01:21.4513753Z + grep -r download.pytorch.org /etc/hosts 2024-12-18T00:01:21.4530908Z + echo '3.167.99.70 download.pytorch.org' 2024-12-18T00:01:21.4531317Z + sudo tee -a /etc/hosts 2024-12-18T00:01:21.7101040Z 3.167.99.70 download.pytorch.org 2024-12-18T00:01:21.7118332Z + cat /etc/hosts 2024-12-18T00:01:21.7126499Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-12-18T00:01:21.7131743Z ::1 localhost6 localhost6.localdomain6 2024-12-18T00:01:21.7132352Z 3.167.99.70 download.pytorch.org 2024-12-18T00:01:21.7304758Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@release/2.6 2024-12-18T00:01:21.7305332Z with: 2024-12-18T00:01:21.7306187Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:21.7307157Z docker-build-dir: .ci/docker 2024-12-18T00:01:21.7307499Z working-directory: . 2024-12-18T00:01:21.7307914Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:21.7308371Z force-push: false 2024-12-18T00:01:21.7308643Z env: 2024-12-18T00:01:21.7308892Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:21.7309200Z ##[endgroup] 2024-12-18T00:01:21.7343822Z ##[group]Run set -ex 2024-12-18T00:01:21.7344196Z set -ex 2024-12-18T00:01:21.7344462Z  2024-12-18T00:01:21.7344927Z # If the docker build directory or the build script doesn't exist, the action will 2024-12-18T00:01:21.7345765Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-12-18T00:01:21.7346429Z # job could then download the pre-built image as usual 2024-12-18T00:01:21.7347039Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-12-18T00:01:21.7347605Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:21.7348142Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:21.7348629Z  2024-12-18T00:01:21.7349048Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-12-18T00:01:21.7349558Z  exit 0 2024-12-18T00:01:21.7349830Z else 2024-12-18T00:01:21.7350143Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:21.7350657Z fi 2024-12-18T00:01:21.7350909Z  2024-12-18T00:01:21.7351404Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-12-18T00:01:21.7352102Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-12-18T00:01:21.7352720Z  # use it as it is, but first let's extract the tag 2024-12-18T00:01:21.7353374Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-12-18T00:01:21.7353942Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:21.7354487Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:21.7354944Z else 2024-12-18T00:01:21.7355278Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-12-18T00:01:21.7355789Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:21.7356501Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:21.7357128Z fi 2024-12-18T00:01:21.7363955Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:21.7364359Z env: 2024-12-18T00:01:21.7364573Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:21.7364856Z REPO_NAME: pytorch 2024-12-18T00:01:21.7365639Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:21.7366487Z DOCKER_BUILD_DIR: .ci/docker 2024-12-18T00:01:21.7366878Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:21.7367271Z ##[endgroup] 2024-12-18T00:01:21.7392015Z + [[ ! -d .ci/docker ]] 2024-12-18T00:01:21.7392351Z + [[ ! -f .ci/docker/build.sh ]] 2024-12-18T00:01:21.7393033Z + echo skip=false 2024-12-18T00:01:21.7394296Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-12-18T00:01:21.7399860Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:21.7400738Z ++ awk -F '[:,]' '{print $2}' 2024-12-18T00:01:21.7421083Z + DOCKER_TAG=45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:21.7421619Z + echo docker-tag=45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:21.7422976Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:21.7456922Z ##[group]Run set +e 2024-12-18T00:01:21.7457275Z set +e 2024-12-18T00:01:21.7457533Z set -x 2024-12-18T00:01:21.7457776Z  2024-12-18T00:01:21.7457991Z login() { 2024-12-18T00:01:21.7458510Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-12-18T00:01:21.7459340Z } 2024-12-18T00:01:21.7459591Z  2024-12-18T00:01:21.7459851Z retry () { 2024-12-18T00:01:21.7460164Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-12-18T00:01:21.7460542Z } 2024-12-18T00:01:21.7460794Z  2024-12-18T00:01:21.7461072Z retry login "${DOCKER_REGISTRY}" 2024-12-18T00:01:21.7461444Z  2024-12-18T00:01:21.7461694Z START_TIME=$(date +%s) 2024-12-18T00:01:21.7462044Z # Wait up to 90 minutes 2024-12-18T00:01:21.7462480Z while [[ $(( $(date +%s) - 5400 )) -lt $START_TIME ]]; do 2024-12-18T00:01:21.7463064Z  # Check if image already exists, if it does then skip building it 2024-12-18T00:01:21.7463657Z  if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-12-18T00:01:21.7464094Z  exit 0 2024-12-18T00:01:21.7464515Z  fi 2024-12-18T00:01:21.7464776Z  2024-12-18T00:01:21.7465235Z  # NB: This flag is used by Docker build workflow to push the image to ECR, so we can 2024-12-18T00:01:21.7466031Z  # use this to differentiate between the Docker build and regular build jobs. For the 2024-12-18T00:01:21.7466819Z  # latter, it will wait for the Docker images to become available before continuing 2024-12-18T00:01:21.7467438Z  if [ "${DOCKER_PUSH:-false}" == "true" ]; then 2024-12-18T00:01:21.7467901Z  # It's a Docker build job, let's build the image 2024-12-18T00:01:21.7468317Z  break 2024-12-18T00:01:21.7468599Z  else 2024-12-18T00:01:21.7469007Z  # It's a regular build job, wait for the image to become available 2024-12-18T00:01:21.7469508Z  sleep 300 2024-12-18T00:01:21.7469794Z  fi 2024-12-18T00:01:21.7470056Z done 2024-12-18T00:01:21.7470321Z  2024-12-18T00:01:21.7470738Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-12-18T00:01:21.7471633Z # be empty. The default action would be to continue rebuild the image 2024-12-18T00:01:21.7472169Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-12-18T00:01:21.7472654Z  # if we're on the base branch then use the parent commit 2024-12-18T00:01:21.7473088Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-12-18T00:01:21.7473427Z else 2024-12-18T00:01:21.7473775Z  # otherwise we're on a PR, so use the most recent base commit 2024-12-18T00:01:21.7474281Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-12-18T00:01:21.7474656Z fi 2024-12-18T00:01:21.7474884Z  2024-12-18T00:01:21.7475133Z if [[ -z "${MERGE_BASE}" ]]; then 2024-12-18T00:01:21.7475517Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:21.7475879Z  2024-12-18T00:01:21.7476359Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-12-18T00:01:21.7476952Z  exit 0 2024-12-18T00:01:21.7477300Z fi 2024-12-18T00:01:21.7477539Z  2024-12-18T00:01:21.7477873Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-12-18T00:01:21.7478600Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-12-18T00:01:21.7479226Z  exit 1 2024-12-18T00:01:21.7479463Z fi 2024-12-18T00:01:21.7479698Z  2024-12-18T00:01:21.7480089Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-12-18T00:01:21.7480792Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-12-18T00:01:21.7481422Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-12-18T00:01:21.7482152Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-12-18T00:01:21.7482958Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-12-18T00:01:21.7483451Z fi 2024-12-18T00:01:21.7483687Z  2024-12-18T00:01:21.7483970Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-12-18T00:01:21.7489385Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:21.7489795Z env: 2024-12-18T00:01:21.7490015Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:21.7490311Z DOCKER_BUILD_DIR: .ci/docker 2024-12-18T00:01:21.7490678Z BASE_REVISION: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:01:21.7491570Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:21.7492440Z DOCKER_TAG: 45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:21.7492989Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:21.7493388Z DOCKER_PUSH: 2024-12-18T00:01:21.7493636Z ##[endgroup] 2024-12-18T00:01:21.7516450Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:21.7516967Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:21.7519409Z + aws ecr get-login-password --region us-east-1 2024-12-18T00:01:21.7520879Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:22.3197496Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-12-18T00:01:22.3198457Z Configure a credential helper to remove this warning. See 2024-12-18T00:01:22.3199103Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-12-18T00:01:22.3199560Z 2024-12-18T00:01:22.3199672Z Login Succeeded 2024-12-18T00:01:22.3216334Z ++ date +%s 2024-12-18T00:01:22.3226211Z + START_TIME=1734480082 2024-12-18T00:01:22.3229480Z ++ date +%s 2024-12-18T00:01:22.3238561Z + [[ 1734474682 -lt 1734480082 ]] 2024-12-18T00:01:22.3239784Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:22.5757730Z { 2024-12-18T00:01:22.5758159Z "schemaVersion": 2, 2024-12-18T00:01:22.5758670Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-12-18T00:01:22.5759200Z "config": { 2024-12-18T00:01:22.5759603Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-12-18T00:01:22.5760085Z "size": 50450, 2024-12-18T00:01:22.5760564Z "digest": "sha256:166dd14a00627fbd0d4e062a733b69b733b63b1ac12ec3bd5285db8a8befaaf5" 2024-12-18T00:01:22.5761132Z }, 2024-12-18T00:01:22.5761367Z "layers": [ 2024-12-18T00:01:22.5761615Z { 2024-12-18T00:01:22.5762000Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5762510Z "size": 28580681, 2024-12-18T00:01:22.5763007Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-12-18T00:01:22.5763562Z }, 2024-12-18T00:01:22.5763793Z { 2024-12-18T00:01:22.5764422Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5764915Z "size": 7943353, 2024-12-18T00:01:22.5765407Z "digest": "sha256:09d415c238d76b32a7ea4a6e6add9542db9a5641f7f183af70aae185d0709e58" 2024-12-18T00:01:22.5765960Z }, 2024-12-18T00:01:22.5766186Z { 2024-12-18T00:01:22.5766568Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5767040Z "size": 56226389, 2024-12-18T00:01:22.5767533Z "digest": "sha256:9fe6e2e61518cba6844870c03b285737daec35e62baf25ae7744629ed3a7b470" 2024-12-18T00:01:22.5768196Z }, 2024-12-18T00:01:22.5768412Z { 2024-12-18T00:01:22.5768780Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5769237Z "size": 187, 2024-12-18T00:01:22.5769699Z "digest": "sha256:41f16248e682693ff20b3032c1d5e5541cc87c5af898ae2ff9b24d2940e59100" 2024-12-18T00:01:22.5770227Z }, 2024-12-18T00:01:22.5770444Z { 2024-12-18T00:01:22.5770817Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5771274Z "size": 6885, 2024-12-18T00:01:22.5771858Z "digest": "sha256:95d7b781703928cf3c4eece39d800cccb76728c375fedf51ecd83833fb25e458" 2024-12-18T00:01:22.5772400Z }, 2024-12-18T00:01:22.5772622Z { 2024-12-18T00:01:22.5773139Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5774006Z "size": 1383662475, 2024-12-18T00:01:22.5774608Z "digest": "sha256:8f6c9048534734f4c873935293b7296225846ceb31c1a158400a67ea170dde7f" 2024-12-18T00:01:22.5775132Z }, 2024-12-18T00:01:22.5775410Z { 2024-12-18T00:01:22.5776027Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5776692Z "size": 62282, 2024-12-18T00:01:22.5777153Z "digest": "sha256:ab17245097e491b9368790714f9d90ed447bf0973bd677cfe6f2456d62b72a13" 2024-12-18T00:01:22.5777862Z }, 2024-12-18T00:01:22.5778085Z { 2024-12-18T00:01:22.5778465Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5779067Z "size": 1684, 2024-12-18T00:01:22.5779759Z "digest": "sha256:dfecd7e9912b76ed460b8edd5a85f1943666e38a973ab5458177cf2c7c3110e3" 2024-12-18T00:01:22.5780323Z }, 2024-12-18T00:01:22.5780553Z { 2024-12-18T00:01:22.5780973Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5781449Z "size": 1522, 2024-12-18T00:01:22.5781954Z "digest": "sha256:464a8f74544589bf7b57f9a4cadcb6681e5ed00758f6c35025e691df4e88e890" 2024-12-18T00:01:22.5782527Z }, 2024-12-18T00:01:22.5782752Z { 2024-12-18T00:01:22.5783132Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5783606Z "size": 2445884909, 2024-12-18T00:01:22.5784091Z "digest": "sha256:d67c111fa58886d8904335b5122039816c8e51af5844cb6fe4e14154d9a7c016" 2024-12-18T00:01:22.5784638Z }, 2024-12-18T00:01:22.5784863Z { 2024-12-18T00:01:22.5785246Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5785723Z "size": 85882, 2024-12-18T00:01:22.5786202Z "digest": "sha256:0723abde19c09431db80b5d0f0e003541f6232c2da5c4f35c635c861875937ae" 2024-12-18T00:01:22.5786754Z }, 2024-12-18T00:01:22.5786979Z { 2024-12-18T00:01:22.5787356Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5787828Z "size": 1825, 2024-12-18T00:01:22.5788306Z "digest": "sha256:38b3bd834f10691245159178ede4a97949d80aafef88c4da88bc8c59b128c195" 2024-12-18T00:01:22.5788855Z }, 2024-12-18T00:01:22.5789079Z { 2024-12-18T00:01:22.5789460Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5789937Z "size": 655189618, 2024-12-18T00:01:22.5790532Z "digest": "sha256:e45523662aa5eb3b174fdd800641a054a960911c585cea9b0782baf969a4dfa5" 2024-12-18T00:01:22.5791067Z }, 2024-12-18T00:01:22.5791289Z { 2024-12-18T00:01:22.5791655Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5792113Z "size": 704, 2024-12-18T00:01:22.5792684Z "digest": "sha256:5a6d97ee6e47c9a76f0de956b2f0e28bfc3947fdf421470b6202b5ba2702f3a9" 2024-12-18T00:01:22.5793227Z }, 2024-12-18T00:01:22.5793447Z { 2024-12-18T00:01:22.5793801Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5794273Z "size": 1260, 2024-12-18T00:01:22.5794747Z "digest": "sha256:8b9cee92f4081f7ae7b4ac3842ed028a3595044d35c3aabdfdf61fdf414b422c" 2024-12-18T00:01:22.5795292Z }, 2024-12-18T00:01:22.5795512Z { 2024-12-18T00:01:22.5795897Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5796378Z "size": 484, 2024-12-18T00:01:22.5796843Z "digest": "sha256:05a25e0e092fec6d2dd4a72016ea313485058594415703c2be2cc9fb041d483b" 2024-12-18T00:01:22.5797366Z }, 2024-12-18T00:01:22.5797597Z { 2024-12-18T00:01:22.5797968Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5798445Z "size": 91714827, 2024-12-18T00:01:22.5798913Z "digest": "sha256:32f9bd3454b2990eb47fa1b21e333f7e8b560c5c086f531c5ee9389676c218ac" 2024-12-18T00:01:22.5799449Z }, 2024-12-18T00:01:22.5799670Z { 2024-12-18T00:01:22.5800046Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5800522Z "size": 3707, 2024-12-18T00:01:22.5800985Z "digest": "sha256:7cef0be11b192948fbbde9ab3ef1a6d84a3008372f6c8dd6b2bc29b049f76e74" 2024-12-18T00:01:22.5801538Z }, 2024-12-18T00:01:22.5801763Z { 2024-12-18T00:01:22.5802136Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5802613Z "size": 1860, 2024-12-18T00:01:22.5803065Z "digest": "sha256:3a7f27881c57c1035aa6d2109802ac6da11afd93eaa81d48ba8e07f910242248" 2024-12-18T00:01:22.5803606Z }, 2024-12-18T00:01:22.5803830Z { 2024-12-18T00:01:22.5804270Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5804743Z "size": 700, 2024-12-18T00:01:22.5805201Z "digest": "sha256:7feada69620ed9505fc525cf4e6d81f7d25589cb3c26fe8fef7479807372f91a" 2024-12-18T00:01:22.5805739Z }, 2024-12-18T00:01:22.5805964Z { 2024-12-18T00:01:22.5806334Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5806807Z "size": 478, 2024-12-18T00:01:22.5807259Z "digest": "sha256:1f019fb4d944db0c3f6f320a880353d7ebeae6df3456f7e492b7a6c19e7430f2" 2024-12-18T00:01:22.5807802Z }, 2024-12-18T00:01:22.5808020Z { 2024-12-18T00:01:22.5808391Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5808871Z "size": 2953611806, 2024-12-18T00:01:22.5809332Z "digest": "sha256:f900f32845825b74c796aa12c8ca7412b38ae05813a53257f53b49bb2b501bcd" 2024-12-18T00:01:22.5809860Z }, 2024-12-18T00:01:22.5810079Z { 2024-12-18T00:01:22.5810449Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5810924Z "size": 380, 2024-12-18T00:01:22.5811375Z "digest": "sha256:03ef53c6b863b93f161c2bf5eb906c9cae887d4e3cb793fa06212162c09b3584" 2024-12-18T00:01:22.5811917Z }, 2024-12-18T00:01:22.5812133Z { 2024-12-18T00:01:22.5812500Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5812973Z "size": 12909, 2024-12-18T00:01:22.5813430Z "digest": "sha256:8c33d01f9e39d69c625c8a507af96fef2fa8f95ff019b7e060062fbd98da0a3b" 2024-12-18T00:01:22.5813967Z }, 2024-12-18T00:01:22.5814184Z { 2024-12-18T00:01:22.5814555Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5815031Z "size": 863, 2024-12-18T00:01:22.5815480Z "digest": "sha256:49be5867b6ab28064a789b446635d6ec60a1c639deffec9c581499d09a20d31a" 2024-12-18T00:01:22.5816014Z }, 2024-12-18T00:01:22.5816233Z { 2024-12-18T00:01:22.5816600Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5817076Z "size": 106, 2024-12-18T00:01:22.5817522Z "digest": "sha256:2c921c2b926223806ddf443e5a591ae34c19c50f553adea16a2ae986a821b47e" 2024-12-18T00:01:22.5818051Z }, 2024-12-18T00:01:22.5818342Z { 2024-12-18T00:01:22.5818814Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5819464Z "size": 504, 2024-12-18T00:01:22.5819925Z "digest": "sha256:f276569c48f345d048de43d8296869ea278af215142ba5741b765adeac6c238f" 2024-12-18T00:01:22.5820468Z }, 2024-12-18T00:01:22.5820696Z { 2024-12-18T00:01:22.5821079Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5821567Z "size": 121477378, 2024-12-18T00:01:22.5822384Z "digest": "sha256:cac6488770e66c796a9bae0e25b007efd1ff47a360bc5714b7fd2946a39ff59b" 2024-12-18T00:01:22.5822951Z }, 2024-12-18T00:01:22.5823180Z { 2024-12-18T00:01:22.5823561Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5824049Z "size": 109, 2024-12-18T00:01:22.5824517Z "digest": "sha256:9695d05a4951d9c024dd2d0726270c2acd5c81cbb1986ae03f60e167357f43cf" 2024-12-18T00:01:22.5825061Z }, 2024-12-18T00:01:22.5825285Z { 2024-12-18T00:01:22.5825667Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5826148Z "size": 489, 2024-12-18T00:01:22.5826614Z "digest": "sha256:2857ab4d43aab8ad9d8ba6af295f20bfb83a30f5442e8502e1db44c385916241" 2024-12-18T00:01:22.5827167Z }, 2024-12-18T00:01:22.5827392Z { 2024-12-18T00:01:22.5827770Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5828257Z "size": 296, 2024-12-18T00:01:22.5828723Z "digest": "sha256:fcdec1f5e2fde1f1945361f0d3c56b33a33f65d42378ff4ca797060040518bf0" 2024-12-18T00:01:22.5829284Z }, 2024-12-18T00:01:22.5829512Z { 2024-12-18T00:01:22.5829810Z + exit 0 2024-12-18T00:01:22.5830199Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5830689Z "size": 103, 2024-12-18T00:01:22.5831314Z "digest": "sha256:ff1478fbefd979ac80d1b176b2c3f6ddcf4cecf685433398b03bc4f0cbd06110" 2024-12-18T00:01:22.5831879Z }, 2024-12-18T00:01:22.5871067Z { 2024-12-18T00:01:22.5871673Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5872143Z "size": 1476, 2024-12-18T00:01:22.5872601Z "digest": "sha256:f8a6d90f0b3ef982ba39428cb0d40dc4c9e03df7c4011258916d4d5dab1bef27" 2024-12-18T00:01:22.5873134Z }, 2024-12-18T00:01:22.5873339Z { 2024-12-18T00:01:22.5873697Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5874160Z "size": 430258066, 2024-12-18T00:01:22.5874608Z "digest": "sha256:011542937262656639e466e38068de398e4565825a5b1d02acc425bd4da3ed3d" 2024-12-18T00:01:22.5875112Z }, 2024-12-18T00:01:22.5875317Z { 2024-12-18T00:01:22.5875672Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5876133Z "size": 162, 2024-12-18T00:01:22.5876579Z "digest": "sha256:812caf3e433816d6c7bea249be5ff35aa976d1d044d32c02741ac161aade0327" 2024-12-18T00:01:22.5877110Z }, 2024-12-18T00:01:22.5877313Z { 2024-12-18T00:01:22.5877674Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5878129Z "size": 564, 2024-12-18T00:01:22.5878566Z "digest": "sha256:7e2aa240b70cc8e1f87f625b28c79751d71f025760c2402d0c695d6333847c71" 2024-12-18T00:01:22.5879084Z }, 2024-12-18T00:01:22.5879290Z { 2024-12-18T00:01:22.5879641Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5880092Z "size": 35806330, 2024-12-18T00:01:22.5880542Z "digest": "sha256:a0e5fd2630de64c400e6cc5c43094e8ac4550325904f24db8f710842f33cc348" 2024-12-18T00:01:22.5881054Z }, 2024-12-18T00:01:22.5881255Z { 2024-12-18T00:01:22.5881733Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5882188Z "size": 104, 2024-12-18T00:01:22.5882684Z "digest": "sha256:a7253f4f13102b2c05f098066e2c11157d6b24c4cb879b3f3f9a5e29e73aa482" 2024-12-18T00:01:22.5883427Z }, 2024-12-18T00:01:22.5883714Z { 2024-12-18T00:01:22.5884326Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5885280Z "size": 425, 2024-12-18T00:01:22.5886113Z "digest": "sha256:d4bbe33feb5a0018de6d8445ad79aa990043b624c90752a4f264235b00ce9c71" 2024-12-18T00:01:22.5887076Z }, 2024-12-18T00:01:22.5887413Z { 2024-12-18T00:01:22.5888094Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5888865Z "size": 20262180, 2024-12-18T00:01:22.5889598Z "digest": "sha256:c7da3e1fda207aae43390fd25dc953ee6bb0fb29d6f92fe8ea45e25af003040e" 2024-12-18T00:01:22.5890135Z }, 2024-12-18T00:01:22.5890332Z { 2024-12-18T00:01:22.5890690Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5891148Z "size": 643, 2024-12-18T00:01:22.5891601Z "digest": "sha256:a1fa96c8688be0cd5c869e56cb39672774ca1466e3b55b296c07e6ddafc196db" 2024-12-18T00:01:22.5892135Z }, 2024-12-18T00:01:22.5892328Z { 2024-12-18T00:01:22.5892773Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5893237Z "size": 700, 2024-12-18T00:01:22.5893756Z "digest": "sha256:7feada69620ed9505fc525cf4e6d81f7d25589cb3c26fe8fef7479807372f91a" 2024-12-18T00:01:22.5894274Z }, 2024-12-18T00:01:22.5894465Z { 2024-12-18T00:01:22.5894815Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5895268Z "size": 141, 2024-12-18T00:01:22.5895718Z "digest": "sha256:600e7beb6ed65b70903fb3d10a7720f76a0e7ed4225e6a2d974edab4ded9985c" 2024-12-18T00:01:22.5896239Z }, 2024-12-18T00:01:22.5896431Z { 2024-12-18T00:01:22.5896781Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5897237Z "size": 136, 2024-12-18T00:01:22.5897737Z "digest": "sha256:7fee2a426d6e9e46b8fa661021c0cb6cbe7f52186061b61d7f32df2ec7f1dbae" 2024-12-18T00:01:22.5898255Z }, 2024-12-18T00:01:22.5898552Z { 2024-12-18T00:01:22.5899013Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5899652Z "size": 32, 2024-12-18T00:01:22.5900124Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:01:22.5900673Z }, 2024-12-18T00:01:22.5900934Z { 2024-12-18T00:01:22.5901302Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5901774Z "size": 196, 2024-12-18T00:01:22.5902237Z "digest": "sha256:321081f536b035434bb8104a9d93a930eba4415dbeffb4ce926a7c9560a818a9" 2024-12-18T00:01:22.5902775Z }, 2024-12-18T00:01:22.5902982Z { 2024-12-18T00:01:22.5903349Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5903821Z "size": 565, 2024-12-18T00:01:22.5904267Z "digest": "sha256:b2671014f386836740538595eb3d3d98580b1d88d3d63ade2ac59313e53526bd" 2024-12-18T00:01:22.5904777Z }, 2024-12-18T00:01:22.5904982Z { 2024-12-18T00:01:22.5905342Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5905813Z "size": 43163802, 2024-12-18T00:01:22.5906282Z "digest": "sha256:94a3028693ada8037297f5512a3a90c4d307bf06337ef43fcbce578505dbb02d" 2024-12-18T00:01:22.5906805Z }, 2024-12-18T00:01:22.5907029Z { 2024-12-18T00:01:22.5907409Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5907895Z "size": 106, 2024-12-18T00:01:22.5908365Z "digest": "sha256:838f3043a48a3fccad46b59399809bd6a67048404a30c44f8d04272c826634c4" 2024-12-18T00:01:22.5908895Z }, 2024-12-18T00:01:22.5909125Z { 2024-12-18T00:01:22.5909511Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5909997Z "size": 1402, 2024-12-18T00:01:22.5910478Z "digest": "sha256:3bf4f9678a1173c728cf0aa3632bbc5de29b674a2c547de331ecd9d3dd86e8c4" 2024-12-18T00:01:22.5911126Z }, 2024-12-18T00:01:22.5911345Z { 2024-12-18T00:01:22.5911717Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5912193Z "size": 700, 2024-12-18T00:01:22.5912659Z "digest": "sha256:7feada69620ed9505fc525cf4e6d81f7d25589cb3c26fe8fef7479807372f91a" 2024-12-18T00:01:22.5913180Z }, 2024-12-18T00:01:22.5913487Z { 2024-12-18T00:01:22.5913856Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5914333Z "size": 137, 2024-12-18T00:01:22.5914811Z "digest": "sha256:cbd527c6febd14371f782c2752d8947f86adeca1e46e6e1c9dfc0dff6e32df6f" 2024-12-18T00:01:22.5915342Z }, 2024-12-18T00:01:22.5915562Z { 2024-12-18T00:01:22.5915933Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5916409Z "size": 120, 2024-12-18T00:01:22.5916862Z "digest": "sha256:70c75642ca1573a70d7ab208940c68c9ec0518797e55e05b621cef283f348bbd" 2024-12-18T00:01:22.5917381Z }, 2024-12-18T00:01:22.5917601Z { 2024-12-18T00:01:22.5917971Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5918452Z "size": 4886117515, 2024-12-18T00:01:22.5918944Z "digest": "sha256:975ffa249994efd3380d6baae778fb7f9ae31ef5208ceb19da52ab66f447401d" 2024-12-18T00:01:22.5919475Z }, 2024-12-18T00:01:22.5919695Z { 2024-12-18T00:01:22.5920068Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5920541Z "size": 176, 2024-12-18T00:01:22.5921006Z "digest": "sha256:1de45071cffa26dd3b9959d220f37f09e3ebc5282b49c8910ca1b7045f37c763" 2024-12-18T00:01:22.5921528Z }, 2024-12-18T00:01:22.5921747Z { 2024-12-18T00:01:22.5922291Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5922942Z "size": 908, 2024-12-18T00:01:22.5923419Z "digest": "sha256:6f1f1c73799125dbfc36617f78ae22c9507955fb3870b11886da704f63cd79b9" 2024-12-18T00:01:22.5923949Z }, 2024-12-18T00:01:22.5924177Z { 2024-12-18T00:01:22.5924560Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5925049Z "size": 700, 2024-12-18T00:01:22.5925649Z "digest": "sha256:7feada69620ed9505fc525cf4e6d81f7d25589cb3c26fe8fef7479807372f91a" 2024-12-18T00:01:22.5926194Z }, 2024-12-18T00:01:22.5926422Z { 2024-12-18T00:01:22.5926808Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5927295Z "size": 135, 2024-12-18T00:01:22.5927773Z "digest": "sha256:8da387044f457792831b74e2facde5dc612cb058a3fcc22e503a778ad0e9125e" 2024-12-18T00:01:22.5928309Z }, 2024-12-18T00:01:22.5928535Z { 2024-12-18T00:01:22.5928913Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5929398Z "size": 32, 2024-12-18T00:01:22.5929873Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:01:22.5930415Z }, 2024-12-18T00:01:22.5930636Z { 2024-12-18T00:01:22.5931013Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5931496Z "size": 159, 2024-12-18T00:01:22.5931973Z "digest": "sha256:25b1aa58f5caff295c81a716174c70d1207b7efc44a594cb4b4c730d11ca6f58" 2024-12-18T00:01:22.5932517Z }, 2024-12-18T00:01:22.5932742Z { 2024-12-18T00:01:22.5933120Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5933612Z "size": 1931, 2024-12-18T00:01:22.5934193Z "digest": "sha256:f4c8ff23e407e7f06a6b9e83db042522c9cb14a2872e033bf0aec10e0cd68af8" 2024-12-18T00:01:22.5934719Z }, 2024-12-18T00:01:22.5934937Z { 2024-12-18T00:01:22.5935301Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5935775Z "size": 196807254, 2024-12-18T00:01:22.5936263Z "digest": "sha256:d9a1d16ac9fb7fb0a5b04cf9bd8a051fedd0f6ef1467103fc8796790e941e1bb" 2024-12-18T00:01:22.5936795Z }, 2024-12-18T00:01:22.5937009Z { 2024-12-18T00:01:22.5937377Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5937847Z "size": 163, 2024-12-18T00:01:22.5938295Z "digest": "sha256:4c24861203037ee0fc72cda05c5446739232f8fc027622a4982a392434699737" 2024-12-18T00:01:22.5938890Z }, 2024-12-18T00:01:22.5939280Z { 2024-12-18T00:01:22.5939667Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5940162Z "size": 7942, 2024-12-18T00:01:22.5940786Z "digest": "sha256:1c1a1b6b59a8cfdf0a6b0dd0fbf35ea01e8d3d9fbfbf7a1cc3c5129697842953" 2024-12-18T00:01:22.5941346Z }, 2024-12-18T00:01:22.5941577Z { 2024-12-18T00:01:22.5941962Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5942451Z "size": 8071, 2024-12-18T00:01:22.5942942Z "digest": "sha256:65ab75609fc9c1a8878dee49da05ea4c3cae04ec5ecd96bff7a59ba5f6716161" 2024-12-18T00:01:22.5943492Z }, 2024-12-18T00:01:22.5943716Z { 2024-12-18T00:01:22.5944094Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5944582Z "size": 304, 2024-12-18T00:01:22.5945054Z "digest": "sha256:41fe6d82c37a17ff1a0e760821e5ffd58423b2c50138348b2d99214fa8c7f7b9" 2024-12-18T00:01:22.5945587Z }, 2024-12-18T00:01:22.5945822Z { 2024-12-18T00:01:22.5946203Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5946690Z "size": 7632783, 2024-12-18T00:01:22.5947177Z "digest": "sha256:9456ad12e83f1f01e0a64ef29e45b8ce1657f0c45a5f334956fed566782f34d9" 2024-12-18T00:01:22.5947713Z }, 2024-12-18T00:01:22.5947939Z { 2024-12-18T00:01:22.5948331Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5948806Z "size": 108, 2024-12-18T00:01:22.5949292Z "digest": "sha256:7dd812a149578ae7aa4d747b9c0cfdd91a1edc15b63e7ec826ca42a601c7c03f" 2024-12-18T00:01:22.5949852Z }, 2024-12-18T00:01:22.5950079Z { 2024-12-18T00:01:22.5950461Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5951043Z "size": 54145664, 2024-12-18T00:01:22.5951532Z "digest": "sha256:e47d2dd30bb9d7d5f7295bbf4b68683cf7e0a3ec443e3d47696213edb61bb0ad" 2024-12-18T00:01:22.5952077Z }, 2024-12-18T00:01:22.5952300Z { 2024-12-18T00:01:22.5952741Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5953201Z "size": 486, 2024-12-18T00:01:22.5953683Z "digest": "sha256:9a255e925aee3dd6711c86af0e487d1c5ff1b2cde447bd2f64f5cc6aa08a3666" 2024-12-18T00:01:22.5954229Z }, 2024-12-18T00:01:22.5954450Z { 2024-12-18T00:01:22.5954821Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5955283Z "size": 1373229207, 2024-12-18T00:01:22.5955764Z "digest": "sha256:266e2409e545ef9c13ef48d69773c27f5ebe37e17c77808c473870b99ccaceca" 2024-12-18T00:01:22.5956301Z }, 2024-12-18T00:01:22.5956520Z { 2024-12-18T00:01:22.5956891Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5957348Z "size": 106, 2024-12-18T00:01:22.5957801Z "digest": "sha256:100d759b6098f93451cdf0ab691600451c05cbf8de2592d370f32f6be20b6127" 2024-12-18T00:01:22.5958330Z }, 2024-12-18T00:01:22.5958544Z { 2024-12-18T00:01:22.5958913Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5959375Z "size": 568, 2024-12-18T00:01:22.5959835Z "digest": "sha256:a3111847bfa2e9e8629c6c7a6cbdd52dde3359882add44762805cb47464c285f" 2024-12-18T00:01:22.5960375Z }, 2024-12-18T00:01:22.5960597Z { 2024-12-18T00:01:22.5960964Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5961420Z "size": 36532350, 2024-12-18T00:01:22.5961908Z "digest": "sha256:7e6ea32f1b1cfa63f3df243c6bfa8b2659be396b488adcbdf9a9c0d9a1e65193" 2024-12-18T00:01:22.5962454Z }, 2024-12-18T00:01:22.5962670Z { 2024-12-18T00:01:22.5963040Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5963499Z "size": 111, 2024-12-18T00:01:22.5963964Z "digest": "sha256:38bf4ff6a6aca492ad60288413a8a2462e07f715d05252617bb4fa942c0ca565" 2024-12-18T00:01:22.5964492Z }, 2024-12-18T00:01:22.5964698Z { 2024-12-18T00:01:22.5965051Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5965512Z "size": 531, 2024-12-18T00:01:22.5965956Z "digest": "sha256:231d0f8577895b4225081015421c272a6f6c601e9b367fcf49106b9a8e3ff74c" 2024-12-18T00:01:22.5966505Z }, 2024-12-18T00:01:22.5966795Z { 2024-12-18T00:01:22.5967173Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5967636Z "size": 26033, 2024-12-18T00:01:22.5968120Z "digest": "sha256:0fe4c18889ad4b2d23fac8e5c1a1bba4cc65ed4abf30d84ab871f507fd27ee03" 2024-12-18T00:01:22.5968675Z }, 2024-12-18T00:01:22.5968899Z { 2024-12-18T00:01:22.5969272Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5969726Z "size": 106, 2024-12-18T00:01:22.5970186Z "digest": "sha256:c5d910c7fb9c4e239cfeafa9695f9e8b9a2892da466239b362451bb0d9a539df" 2024-12-18T00:01:22.5970725Z }, 2024-12-18T00:01:22.5970947Z { 2024-12-18T00:01:22.5971311Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5971774Z "size": 32, 2024-12-18T00:01:22.5972235Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:01:22.5972771Z }, 2024-12-18T00:01:22.5972987Z { 2024-12-18T00:01:22.5973359Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5973816Z "size": 32, 2024-12-18T00:01:22.5974280Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:01:22.5974817Z }, 2024-12-18T00:01:22.5975036Z { 2024-12-18T00:01:22.5975401Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5975859Z "size": 32, 2024-12-18T00:01:22.5976315Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:01:22.5976848Z }, 2024-12-18T00:01:22.5977067Z { 2024-12-18T00:01:22.5977431Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:01:22.5977887Z "size": 32, 2024-12-18T00:01:22.5978348Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:01:22.5979216Z } 2024-12-18T00:01:22.5979440Z ] 2024-12-18T00:01:22.5979649Z } 2024-12-18T00:01:22.6022506Z ##[group]Run set -eux 2024-12-18T00:01:22.6022831Z set -eux 2024-12-18T00:01:22.6023854Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin 2024-12-18T00:01:22.6031037Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:22.6031472Z env: 2024-12-18T00:01:22.6031713Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:22.6032006Z ##[endgroup] 2024-12-18T00:01:22.6059796Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token 2024-12-18T00:01:22.6060383Z + jq --raw-output .SecretString 2024-12-18T00:01:22.6061550Z + jq -r .docker_hub_readonly_token 2024-12-18T00:01:22.6062670Z + docker login --username pytorchbot --password-stdin 2024-12-18T00:01:23.2263716Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-12-18T00:01:23.2264505Z Configure a credential helper to remove this warning. See 2024-12-18T00:01:23.2265197Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-12-18T00:01:23.2265830Z 2024-12-18T00:01:23.2265953Z Login Succeeded 2024-12-18T00:01:23.2367579Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-12-18T00:01:23.2367974Z tag=${ECR_DOCKER_IMAGE##*/} 2024-12-18T00:01:23.2368410Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-12-18T00:01:23.2374309Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:23.2374723Z env: 2024-12-18T00:01:23.2374963Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:23.2375771Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:23.2376606Z ##[endgroup] 2024-12-18T00:01:23.2404274Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9-45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:23.2457270Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@release/2.6 2024-12-18T00:01:23.2457793Z with: 2024-12-18T00:01:23.2458580Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:23.2459861Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:23.2460323Z env: 2024-12-18T00:01:23.2460586Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:23.2460902Z ##[endgroup] 2024-12-18T00:01:23.2482475Z ##[group]Run set -x 2024-12-18T00:01:23.2482773Z set -x 2024-12-18T00:01:23.2483030Z set +e 2024-12-18T00:01:23.2483266Z  2024-12-18T00:01:23.2483506Z login() { 2024-12-18T00:01:23.2484039Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-12-18T00:01:23.2484634Z } 2024-12-18T00:01:23.2484870Z  2024-12-18T00:01:23.2485150Z retry () { 2024-12-18T00:01:23.2485465Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-12-18T00:01:23.2485934Z } 2024-12-18T00:01:23.2486165Z  2024-12-18T00:01:23.2486419Z retry login "${DOCKER_REGISTRY}" 2024-12-18T00:01:23.2486750Z  2024-12-18T00:01:23.2486976Z set -e 2024-12-18T00:01:23.2487326Z # ignore output since only exit code is used for conditional 2024-12-18T00:01:23.2487847Z # only pull docker image if it's not available locally 2024-12-18T00:01:23.2488422Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-12-18T00:01:23.2488950Z  retry docker pull "${DOCKER_IMAGE}" 2024-12-18T00:01:23.2489290Z fi 2024-12-18T00:01:23.2494892Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:01:23.2495460Z env: 2024-12-18T00:01:23.2495707Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:01:23.2496553Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:23.2497523Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:23.2497953Z ##[endgroup] 2024-12-18T00:01:23.2521120Z + set +e 2024-12-18T00:01:23.2522134Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:23.2522641Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:23.2524993Z + aws ecr get-login-password --region us-east-1 2024-12-18T00:01:23.2526618Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:01:23.8052691Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-12-18T00:01:23.8054385Z Configure a credential helper to remove this warning. See 2024-12-18T00:01:23.8055070Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-12-18T00:01:23.8055515Z 2024-12-18T00:01:23.8055651Z Login Succeeded 2024-12-18T00:01:23.8067325Z + set -e 2024-12-18T00:01:23.8068605Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:23.8206436Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:23.8207916Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:01:24.0918899Z 45e1356b47a284893081276eff3000b7b534f3b1: Pulling from pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9 2024-12-18T00:01:24.0919759Z 7a2c55901189: Pulling fs layer 2024-12-18T00:01:24.0920402Z 09d415c238d7: Pulling fs layer 2024-12-18T00:01:24.0920758Z 9fe6e2e61518: Pulling fs layer 2024-12-18T00:01:24.0921176Z 41f16248e682: Pulling fs layer 2024-12-18T00:01:24.0922181Z 95d7b7817039: Pulling fs layer 2024-12-18T00:01:24.0922865Z 8f6c90485347: Pulling fs layer 2024-12-18T00:01:24.0923227Z ab17245097e4: Pulling fs layer 2024-12-18T00:01:24.0923653Z dfecd7e9912b: Pulling fs layer 2024-12-18T00:01:24.0924045Z 464a8f745445: Pulling fs layer 2024-12-18T00:01:24.0924388Z d67c111fa588: Pulling fs layer 2024-12-18T00:01:24.0924845Z 0723abde19c0: Pulling fs layer 2024-12-18T00:01:24.0925242Z 38b3bd834f10: Pulling fs layer 2024-12-18T00:01:24.0925669Z e45523662aa5: Pulling fs layer 2024-12-18T00:01:24.0926033Z 5a6d97ee6e47: Pulling fs layer 2024-12-18T00:01:24.0926432Z 8b9cee92f408: Pulling fs layer 2024-12-18T00:01:24.0926765Z 05a25e0e092f: Pulling fs layer 2024-12-18T00:01:24.0927185Z 32f9bd3454b2: Pulling fs layer 2024-12-18T00:01:24.0927536Z 7cef0be11b19: Pulling fs layer 2024-12-18T00:01:24.0927985Z 41f16248e682: Waiting 2024-12-18T00:01:24.0928304Z 3a7f27881c57: Pulling fs layer 2024-12-18T00:01:24.0928680Z 95d7b7817039: Waiting 2024-12-18T00:01:24.0928995Z 8f6c90485347: Waiting 2024-12-18T00:01:24.0929323Z 7feada69620e: Pulling fs layer 2024-12-18T00:01:24.0929767Z ab17245097e4: Waiting 2024-12-18T00:01:24.0930095Z 1f019fb4d944: Pulling fs layer 2024-12-18T00:01:24.0930460Z dfecd7e9912b: Waiting 2024-12-18T00:01:24.0930802Z f900f3284582: Pulling fs layer 2024-12-18T00:01:24.0931157Z 03ef53c6b863: Pulling fs layer 2024-12-18T00:01:24.0931545Z 464a8f745445: Waiting 2024-12-18T00:01:24.0931836Z 8c33d01f9e39: Pulling fs layer 2024-12-18T00:01:24.0932215Z 32f9bd3454b2: Waiting 2024-12-18T00:01:24.0932514Z d67c111fa588: Waiting 2024-12-18T00:01:24.0932818Z 7cef0be11b19: Waiting 2024-12-18T00:01:24.0933276Z 49be5867b6ab: Pulling fs layer 2024-12-18T00:01:24.0934227Z 05a25e0e092f: Waiting 2024-12-18T00:01:24.0934656Z 3a7f27881c57: Waiting 2024-12-18T00:01:24.0935122Z 2c921c2b9262: Pulling fs layer 2024-12-18T00:01:24.0935854Z 8b9cee92f408: Waiting 2024-12-18T00:01:24.0936219Z 38b3bd834f10: Waiting 2024-12-18T00:01:24.0936673Z f276569c48f3: Pulling fs layer 2024-12-18T00:01:24.0937211Z cac6488770e6: Pulling fs layer 2024-12-18T00:01:24.0937680Z 0723abde19c0: Waiting 2024-12-18T00:01:24.0938147Z 9695d05a4951: Pulling fs layer 2024-12-18T00:01:24.0938545Z 1f019fb4d944: Waiting 2024-12-18T00:01:24.0939099Z 2857ab4d43aa: Pulling fs layer 2024-12-18T00:01:24.0939609Z fcdec1f5e2fd: Pulling fs layer 2024-12-18T00:01:24.0940066Z 7feada69620e: Waiting 2024-12-18T00:01:24.0940461Z 5a6d97ee6e47: Waiting 2024-12-18T00:01:24.0940942Z ff1478fbefd9: Pulling fs layer 2024-12-18T00:01:24.0941343Z 03ef53c6b863: Waiting 2024-12-18T00:01:24.0941738Z e45523662aa5: Waiting 2024-12-18T00:01:24.0942238Z f900f3284582: Waiting 2024-12-18T00:01:24.0942659Z 8c33d01f9e39: Waiting 2024-12-18T00:01:24.0943137Z f8a6d90f0b3e: Pulling fs layer 2024-12-18T00:01:24.0943562Z 2c921c2b9262: Waiting 2024-12-18T00:01:24.0943994Z 011542937262: Pulling fs layer 2024-12-18T00:01:24.0944470Z 49be5867b6ab: Waiting 2024-12-18T00:01:24.0944932Z 812caf3e4338: Pulling fs layer 2024-12-18T00:01:24.0945383Z cac6488770e6: Waiting 2024-12-18T00:01:24.0945756Z f276569c48f3: Waiting 2024-12-18T00:01:24.0946210Z 7e2aa240b70c: Pulling fs layer 2024-12-18T00:01:24.0946668Z 2857ab4d43aa: Waiting 2024-12-18T00:01:24.0947154Z f8a6d90f0b3e: Waiting 2024-12-18T00:01:24.0947592Z a0e5fd2630de: Pulling fs layer 2024-12-18T00:01:24.0948003Z a7253f4f1310: Pulling fs layer 2024-12-18T00:01:24.0948487Z 011542937262: Waiting 2024-12-18T00:01:24.0948931Z d4bbe33feb5a: Pulling fs layer 2024-12-18T00:01:24.0949366Z 812caf3e4338: Waiting 2024-12-18T00:01:24.0949857Z c7da3e1fda20: Pulling fs layer 2024-12-18T00:01:24.0950264Z ff1478fbefd9: Waiting 2024-12-18T00:01:24.0950670Z a1fa96c8688b: Pulling fs layer 2024-12-18T00:01:24.0951188Z d4bbe33feb5a: Waiting 2024-12-18T00:01:24.0951739Z 7e2aa240b70c: Waiting 2024-12-18T00:01:24.0952191Z a0e5fd2630de: Waiting 2024-12-18T00:01:24.0952599Z 600e7beb6ed6: Pulling fs layer 2024-12-18T00:01:24.0953050Z fcdec1f5e2fd: Waiting 2024-12-18T00:01:24.0953492Z 7fee2a426d6e: Pulling fs layer 2024-12-18T00:01:24.0954126Z 4f4fb700ef54: Pulling fs layer 2024-12-18T00:01:24.0954600Z 321081f536b0: Pulling fs layer 2024-12-18T00:01:24.0955010Z b2671014f386: Pulling fs layer 2024-12-18T00:01:24.0955486Z 94a3028693ad: Pulling fs layer 2024-12-18T00:01:24.0955981Z 600e7beb6ed6: Waiting 2024-12-18T00:01:24.0956435Z 7fee2a426d6e: Waiting 2024-12-18T00:01:24.0956856Z 838f3043a48a: Pulling fs layer 2024-12-18T00:01:24.0957317Z a7253f4f1310: Waiting 2024-12-18T00:01:24.0957723Z 94a3028693ad: Waiting 2024-12-18T00:01:24.0958127Z a1fa96c8688b: Waiting 2024-12-18T00:01:24.0958526Z 3bf4f9678a11: Pulling fs layer 2024-12-18T00:01:24.0959040Z cbd527c6febd: Pulling fs layer 2024-12-18T00:01:24.0959499Z 70c75642ca15: Pulling fs layer 2024-12-18T00:01:24.0959865Z 975ffa249994: Pulling fs layer 2024-12-18T00:01:24.0960367Z 1de45071cffa: Pulling fs layer 2024-12-18T00:01:24.0960819Z 321081f536b0: Waiting 2024-12-18T00:01:24.0961224Z b2671014f386: Waiting 2024-12-18T00:01:24.0961700Z 6f1f1c737991: Pulling fs layer 2024-12-18T00:01:24.0962133Z 8da387044f45: Pulling fs layer 2024-12-18T00:01:24.0962586Z 6f1f1c737991: Waiting 2024-12-18T00:01:24.0963022Z 8da387044f45: Waiting 2024-12-18T00:01:24.0963443Z 25b1aa58f5ca: Pulling fs layer 2024-12-18T00:01:24.0963924Z f4c8ff23e407: Pulling fs layer 2024-12-18T00:01:24.0964332Z d9a1d16ac9fb: Pulling fs layer 2024-12-18T00:01:24.0964774Z 25b1aa58f5ca: Waiting 2024-12-18T00:01:24.0965201Z f4c8ff23e407: Waiting 2024-12-18T00:01:24.0965624Z 4c2486120303: Pulling fs layer 2024-12-18T00:01:24.0966047Z 1c1a1b6b59a8: Pulling fs layer 2024-12-18T00:01:24.0966499Z 65ab75609fc9: Pulling fs layer 2024-12-18T00:01:24.0966996Z 41fe6d82c37a: Pulling fs layer 2024-12-18T00:01:24.0967413Z 1c1a1b6b59a8: Waiting 2024-12-18T00:01:24.0967886Z d9a1d16ac9fb: Waiting 2024-12-18T00:01:24.0968295Z 65ab75609fc9: Waiting 2024-12-18T00:01:24.0968629Z 9456ad12e83f: Pulling fs layer 2024-12-18T00:01:24.0969246Z 4c2486120303: Waiting 2024-12-18T00:01:24.0969661Z 7dd812a14957: Pulling fs layer 2024-12-18T00:01:24.0970130Z e47d2dd30bb9: Pulling fs layer 2024-12-18T00:01:24.0970599Z 41fe6d82c37a: Waiting 2024-12-18T00:01:24.0970958Z 9a255e925aee: Pulling fs layer 2024-12-18T00:01:24.0971403Z 9456ad12e83f: Waiting 2024-12-18T00:01:24.0971862Z e47d2dd30bb9: Waiting 2024-12-18T00:01:24.0972280Z 266e2409e545: Pulling fs layer 2024-12-18T00:01:24.0988491Z 9a255e925aee: Waiting 2024-12-18T00:01:24.0988874Z 100d759b6098: Pulling fs layer 2024-12-18T00:01:24.0989238Z 7dd812a14957: Waiting 2024-12-18T00:01:24.0989541Z a3111847bfa2: Pulling fs layer 2024-12-18T00:01:24.0989889Z 266e2409e545: Waiting 2024-12-18T00:01:24.0990198Z 100d759b6098: Waiting 2024-12-18T00:01:24.0990511Z 7e6ea32f1b1c: Pulling fs layer 2024-12-18T00:01:24.0991015Z 38bf4ff6a6ac: Pulling fs layer 2024-12-18T00:01:24.0991348Z 231d0f857789: Pulling fs layer 2024-12-18T00:01:24.0991683Z a3111847bfa2: Waiting 2024-12-18T00:01:24.0992027Z 0fe4c18889ad: Pulling fs layer 2024-12-18T00:01:24.0992378Z c5d910c7fb9c: Pulling fs layer 2024-12-18T00:01:24.0992712Z 231d0f857789: Waiting 2024-12-18T00:01:24.0993002Z c5d910c7fb9c: Waiting 2024-12-18T00:01:24.0993302Z 0fe4c18889ad: Waiting 2024-12-18T00:01:24.2167030Z 09d415c238d7: Verifying Checksum 2024-12-18T00:01:24.2167467Z 09d415c238d7: Download complete 2024-12-18T00:01:24.2908822Z 41f16248e682: Verifying Checksum 2024-12-18T00:01:24.2909209Z 41f16248e682: Download complete 2024-12-18T00:01:24.3755509Z 95d7b7817039: Verifying Checksum 2024-12-18T00:01:24.3755896Z 95d7b7817039: Download complete 2024-12-18T00:01:24.4467368Z 7a2c55901189: Download complete 2024-12-18T00:01:24.5288033Z ab17245097e4: Download complete 2024-12-18T00:01:24.6257266Z dfecd7e9912b: Download complete 2024-12-18T00:01:24.7002852Z 464a8f745445: Download complete 2024-12-18T00:01:24.7202511Z 9fe6e2e61518: Verifying Checksum 2024-12-18T00:01:24.7202921Z 9fe6e2e61518: Download complete 2024-12-18T00:01:24.8202955Z 0723abde19c0: Verifying Checksum 2024-12-18T00:01:24.8203367Z 0723abde19c0: Download complete 2024-12-18T00:01:24.9016079Z 38b3bd834f10: Verifying Checksum 2024-12-18T00:01:24.9017012Z 38b3bd834f10: Download complete 2024-12-18T00:01:25.1765587Z 7a2c55901189: Pull complete 2024-12-18T00:01:25.3856987Z 09d415c238d7: Pull complete 2024-12-18T00:01:26.1271437Z 9fe6e2e61518: Pull complete 2024-12-18T00:01:26.1420487Z 41f16248e682: Pull complete 2024-12-18T00:01:26.1567838Z 95d7b7817039: Pull complete 2024-12-18T00:01:31.5145474Z e45523662aa5: Verifying Checksum 2024-12-18T00:01:31.5145935Z e45523662aa5: Download complete 2024-12-18T00:01:31.5916777Z 5a6d97ee6e47: Verifying Checksum 2024-12-18T00:01:31.5917189Z 5a6d97ee6e47: Download complete 2024-12-18T00:01:31.6762595Z 8b9cee92f408: Verifying Checksum 2024-12-18T00:01:31.6762998Z 8b9cee92f408: Download complete 2024-12-18T00:01:31.7747625Z 05a25e0e092f: Verifying Checksum 2024-12-18T00:01:31.7748033Z 05a25e0e092f: Download complete 2024-12-18T00:01:32.7604434Z 32f9bd3454b2: Verifying Checksum 2024-12-18T00:01:32.7604910Z 32f9bd3454b2: Download complete 2024-12-18T00:01:32.8510547Z 7cef0be11b19: Verifying Checksum 2024-12-18T00:01:32.8511005Z 7cef0be11b19: Download complete 2024-12-18T00:01:32.9464600Z 3a7f27881c57: Download complete 2024-12-18T00:01:33.0460837Z 7feada69620e: Download complete 2024-12-18T00:01:33.1327701Z 1f019fb4d944: Download complete 2024-12-18T00:01:38.2810337Z 8f6c90485347: Verifying Checksum 2024-12-18T00:01:38.2810903Z 8f6c90485347: Download complete 2024-12-18T00:01:38.4082485Z 03ef53c6b863: Verifying Checksum 2024-12-18T00:01:38.4082920Z 03ef53c6b863: Download complete 2024-12-18T00:01:38.4926100Z 8c33d01f9e39: Verifying Checksum 2024-12-18T00:01:38.4926512Z 8c33d01f9e39: Download complete 2024-12-18T00:01:38.5742297Z 49be5867b6ab: Verifying Checksum 2024-12-18T00:01:38.6863020Z 2c921c2b9262: Download complete 2024-12-18T00:01:38.7774579Z f276569c48f3: Download complete 2024-12-18T00:01:40.0848385Z cac6488770e6: Verifying Checksum 2024-12-18T00:01:40.0849185Z cac6488770e6: Download complete 2024-12-18T00:01:40.1797787Z 9695d05a4951: Verifying Checksum 2024-12-18T00:01:40.1798214Z 9695d05a4951: Download complete 2024-12-18T00:01:40.2589968Z 2857ab4d43aa: Verifying Checksum 2024-12-18T00:01:40.2590428Z 2857ab4d43aa: Download complete 2024-12-18T00:01:40.3416402Z fcdec1f5e2fd: Verifying Checksum 2024-12-18T00:01:40.3416843Z fcdec1f5e2fd: Download complete 2024-12-18T00:01:40.4150369Z ff1478fbefd9: Download complete 2024-12-18T00:01:40.5114313Z f8a6d90f0b3e: Verifying Checksum 2024-12-18T00:01:40.5114827Z f8a6d90f0b3e: Download complete 2024-12-18T00:01:44.8660839Z 011542937262: Download complete 2024-12-18T00:01:44.9584325Z 812caf3e4338: Verifying Checksum 2024-12-18T00:01:44.9584747Z 812caf3e4338: Download complete 2024-12-18T00:01:45.0571801Z 7e2aa240b70c: Verifying Checksum 2024-12-18T00:01:45.0572260Z 7e2aa240b70c: Download complete 2024-12-18T00:01:45.4836583Z a0e5fd2630de: Verifying Checksum 2024-12-18T00:01:45.4837178Z a0e5fd2630de: Download complete 2024-12-18T00:01:45.5582483Z a7253f4f1310: Verifying Checksum 2024-12-18T00:01:45.5582925Z a7253f4f1310: Download complete 2024-12-18T00:01:45.6499292Z d4bbe33feb5a: Verifying Checksum 2024-12-18T00:01:45.6500177Z d4bbe33feb5a: Download complete 2024-12-18T00:01:45.9006909Z c7da3e1fda20: Verifying Checksum 2024-12-18T00:01:45.9007345Z c7da3e1fda20: Download complete 2024-12-18T00:01:45.9777767Z a1fa96c8688b: Verifying Checksum 2024-12-18T00:01:45.9778211Z a1fa96c8688b: Download complete 2024-12-18T00:01:46.0755647Z 600e7beb6ed6: Verifying Checksum 2024-12-18T00:01:46.0756073Z 600e7beb6ed6: Download complete 2024-12-18T00:01:46.1599046Z 7fee2a426d6e: Verifying Checksum 2024-12-18T00:01:46.1599470Z 7fee2a426d6e: Download complete 2024-12-18T00:01:46.1695570Z 4f4fb700ef54: Verifying Checksum 2024-12-18T00:01:46.1695989Z 4f4fb700ef54: Download complete 2024-12-18T00:01:46.2542368Z 321081f536b0: Verifying Checksum 2024-12-18T00:01:46.2542798Z 321081f536b0: Download complete 2024-12-18T00:01:46.3434176Z b2671014f386: Verifying Checksum 2024-12-18T00:01:46.3434660Z b2671014f386: Download complete 2024-12-18T00:01:46.8484964Z 94a3028693ad: Verifying Checksum 2024-12-18T00:01:46.8485738Z 94a3028693ad: Download complete 2024-12-18T00:01:46.9393759Z 838f3043a48a: Download complete 2024-12-18T00:01:47.0487030Z 3bf4f9678a11: Verifying Checksum 2024-12-18T00:01:47.0487468Z 3bf4f9678a11: Download complete 2024-12-18T00:01:47.1294465Z cbd527c6febd: Verifying Checksum 2024-12-18T00:01:47.1294913Z cbd527c6febd: Download complete 2024-12-18T00:01:47.2110084Z 70c75642ca15: Verifying Checksum 2024-12-18T00:01:47.2110514Z 70c75642ca15: Download complete 2024-12-18T00:01:49.2221058Z d67c111fa588: Verifying Checksum 2024-12-18T00:01:49.2221676Z d67c111fa588: Download complete 2024-12-18T00:01:49.3026963Z 1de45071cffa: Download complete 2024-12-18T00:01:49.4253989Z 6f1f1c737991: Verifying Checksum 2024-12-18T00:01:49.4254413Z 6f1f1c737991: Download complete 2024-12-18T00:01:49.5181494Z 8da387044f45: Verifying Checksum 2024-12-18T00:01:49.5181967Z 8da387044f45: Download complete 2024-12-18T00:01:49.6084376Z 25b1aa58f5ca: Download complete 2024-12-18T00:01:49.7029515Z f4c8ff23e407: Verifying Checksum 2024-12-18T00:01:49.7029978Z f4c8ff23e407: Download complete 2024-12-18T00:01:51.7360115Z d9a1d16ac9fb: Verifying Checksum 2024-12-18T00:01:51.7360535Z d9a1d16ac9fb: Download complete 2024-12-18T00:01:51.8186270Z 4c2486120303: Verifying Checksum 2024-12-18T00:01:51.8186687Z 4c2486120303: Download complete 2024-12-18T00:01:51.8987488Z 1c1a1b6b59a8: Download complete 2024-12-18T00:01:51.9743745Z 65ab75609fc9: Download complete 2024-12-18T00:01:52.0652941Z 41fe6d82c37a: Download complete 2024-12-18T00:01:52.1981274Z 8f6c90485347: Pull complete 2024-12-18T00:01:52.2006655Z 9456ad12e83f: Verifying Checksum 2024-12-18T00:01:52.2007629Z 9456ad12e83f: Download complete 2024-12-18T00:01:52.2715395Z 7dd812a14957: Download complete 2024-12-18T00:01:52.5893234Z ab17245097e4: Pull complete 2024-12-18T00:01:52.8836737Z e47d2dd30bb9: Verifying Checksum 2024-12-18T00:01:52.8837628Z e47d2dd30bb9: Download complete 2024-12-18T00:01:52.9880719Z 9a255e925aee: Verifying Checksum 2024-12-18T00:01:52.9881197Z 9a255e925aee: Download complete 2024-12-18T00:01:53.0297432Z dfecd7e9912b: Pull complete 2024-12-18T00:01:53.4223834Z 464a8f745445: Pull complete 2024-12-18T00:02:02.7291958Z f900f3284582: Verifying Checksum 2024-12-18T00:02:02.7292369Z f900f3284582: Download complete 2024-12-18T00:02:02.8219882Z 100d759b6098: Verifying Checksum 2024-12-18T00:02:02.8220327Z 100d759b6098: Download complete 2024-12-18T00:02:02.9177931Z a3111847bfa2: Verifying Checksum 2024-12-18T00:02:02.9178374Z a3111847bfa2: Download complete 2024-12-18T00:02:03.3451937Z 7e6ea32f1b1c: Verifying Checksum 2024-12-18T00:02:03.3452398Z 7e6ea32f1b1c: Download complete 2024-12-18T00:02:03.4435090Z 38bf4ff6a6ac: Download complete 2024-12-18T00:02:03.5281103Z 231d0f857789: Download complete 2024-12-18T00:02:03.6381546Z 0fe4c18889ad: Download complete 2024-12-18T00:02:03.7244436Z c5d910c7fb9c: Verifying Checksum 2024-12-18T00:02:03.7244898Z c5d910c7fb9c: Download complete 2024-12-18T00:02:06.7682887Z 266e2409e545: Verifying Checksum 2024-12-18T00:02:06.7683392Z 266e2409e545: Download complete 2024-12-18T00:02:18.2868172Z d67c111fa588: Pull complete 2024-12-18T00:02:18.7763309Z 0723abde19c0: Pull complete 2024-12-18T00:02:19.2542326Z 38b3bd834f10: Pull complete 2024-12-18T00:02:28.9353732Z e45523662aa5: Pull complete 2024-12-18T00:02:29.4168229Z 5a6d97ee6e47: Pull complete 2024-12-18T00:02:29.8894083Z 8b9cee92f408: Pull complete 2024-12-18T00:02:30.3618417Z 05a25e0e092f: Pull complete 2024-12-18T00:02:32.5402019Z 32f9bd3454b2: Pull complete 2024-12-18T00:02:32.9918203Z 7cef0be11b19: Pull complete 2024-12-18T00:02:33.4327113Z 3a7f27881c57: Pull complete 2024-12-18T00:02:33.8526081Z 7feada69620e: Pull complete 2024-12-18T00:02:34.2849149Z 1f019fb4d944: Pull complete 2024-12-18T00:02:36.1420763Z 975ffa249994: Verifying Checksum 2024-12-18T00:02:36.1421164Z 975ffa249994: Download complete 2024-12-18T00:03:16.2938748Z f900f3284582: Pull complete 2024-12-18T00:03:16.7217256Z 03ef53c6b863: Pull complete 2024-12-18T00:03:17.1593070Z 8c33d01f9e39: Pull complete 2024-12-18T00:03:17.4811451Z 49be5867b6ab: Pull complete 2024-12-18T00:03:17.9539862Z 2c921c2b9262: Pull complete 2024-12-18T00:03:18.4043419Z f276569c48f3: Pull complete 2024-12-18T00:03:21.3328597Z cac6488770e6: Pull complete 2024-12-18T00:03:21.4264173Z 9695d05a4951: Pull complete 2024-12-18T00:03:21.8789392Z 2857ab4d43aa: Pull complete 2024-12-18T00:03:22.2688058Z fcdec1f5e2fd: Pull complete 2024-12-18T00:03:22.6659516Z ff1478fbefd9: Pull complete 2024-12-18T00:03:23.0804530Z f8a6d90f0b3e: Pull complete 2024-12-18T00:03:30.3561742Z 011542937262: Pull complete 2024-12-18T00:03:30.7652506Z 812caf3e4338: Pull complete 2024-12-18T00:03:31.1943836Z 7e2aa240b70c: Pull complete 2024-12-18T00:03:32.2672236Z a0e5fd2630de: Pull complete 2024-12-18T00:03:32.7214191Z a7253f4f1310: Pull complete 2024-12-18T00:03:33.1652811Z d4bbe33feb5a: Pull complete 2024-12-18T00:03:33.8300212Z c7da3e1fda20: Pull complete 2024-12-18T00:03:34.2873041Z a1fa96c8688b: Pull complete 2024-12-18T00:03:35.1672530Z 600e7beb6ed6: Pull complete 2024-12-18T00:03:35.5659330Z 7fee2a426d6e: Pull complete 2024-12-18T00:03:35.9782565Z 4f4fb700ef54: Pull complete 2024-12-18T00:03:36.3729354Z 321081f536b0: Pull complete 2024-12-18T00:03:36.5848233Z b2671014f386: Pull complete 2024-12-18T00:03:38.6166638Z 94a3028693ad: Pull complete 2024-12-18T00:03:39.0404508Z 838f3043a48a: Pull complete 2024-12-18T00:03:39.4106618Z 3bf4f9678a11: Pull complete 2024-12-18T00:03:39.9600410Z cbd527c6febd: Pull complete 2024-12-18T00:03:40.2354334Z 70c75642ca15: Pull complete 2024-12-18T00:04:53.4575206Z 975ffa249994: Pull complete 2024-12-18T00:04:53.9057340Z 1de45071cffa: Pull complete 2024-12-18T00:04:54.3265820Z 6f1f1c737991: Pull complete 2024-12-18T00:04:55.2303124Z 8da387044f45: Pull complete 2024-12-18T00:04:56.0945326Z 25b1aa58f5ca: Pull complete 2024-12-18T00:04:56.5599239Z f4c8ff23e407: Pull complete 2024-12-18T00:05:02.3520163Z d9a1d16ac9fb: Pull complete 2024-12-18T00:05:02.7009523Z 4c2486120303: Pull complete 2024-12-18T00:05:03.0256295Z 1c1a1b6b59a8: Pull complete 2024-12-18T00:05:03.4942305Z 65ab75609fc9: Pull complete 2024-12-18T00:05:03.9244234Z 41fe6d82c37a: Pull complete 2024-12-18T00:05:05.0770694Z 9456ad12e83f: Pull complete 2024-12-18T00:05:05.5518168Z 7dd812a14957: Pull complete 2024-12-18T00:05:07.5133974Z e47d2dd30bb9: Pull complete 2024-12-18T00:05:07.9227437Z 9a255e925aee: Pull complete 2024-12-18T00:05:20.5648509Z 266e2409e545: Pull complete 2024-12-18T00:05:20.9541186Z 100d759b6098: Pull complete 2024-12-18T00:05:21.3631283Z a3111847bfa2: Pull complete 2024-12-18T00:05:22.1914074Z 7e6ea32f1b1c: Pull complete 2024-12-18T00:05:22.6601724Z 38bf4ff6a6ac: Pull complete 2024-12-18T00:05:23.1170783Z 231d0f857789: Pull complete 2024-12-18T00:05:23.5419160Z 0fe4c18889ad: Pull complete 2024-12-18T00:05:23.9820595Z c5d910c7fb9c: Pull complete 2024-12-18T00:05:25.9622197Z Digest: sha256:52918bcdb0e370d1b9413099049464295e6c36e0a09cff6cff645b519e6da502 2024-12-18T00:05:26.0612014Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:05:26.1060701Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:05:26.1119615Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T00:05:26.1120703Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T00:05:26.1132267Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:05:26.1132707Z env: 2024-12-18T00:05:26.1132979Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:05:26.1133307Z ##[endgroup] 2024-12-18T00:05:26.1405186Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@release/2.6 2024-12-18T00:05:26.1405682Z with: 2024-12-18T00:05:26.1405931Z driver-version: 550.54.15 2024-12-18T00:05:26.1406202Z env: 2024-12-18T00:05:26.1406438Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:05:26.1406726Z ##[endgroup] 2024-12-18T00:05:26.1572344Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-12-18T00:05:26.1572820Z with: 2024-12-18T00:05:26.1573263Z timeout_minutes: 10 2024-12-18T00:05:26.1573551Z max_attempts: 3 2024-12-18T00:05:26.1604511Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 nvidia-container-toolkit-1.16.2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-12-18T00:05:26.1634795Z retry_wait_seconds: 10 2024-12-18T00:05:26.1635121Z polling_interval_seconds: 1 2024-12-18T00:05:26.1635460Z warning_on_retry: true 2024-12-18T00:05:26.1635778Z continue_on_error: false 2024-12-18T00:05:26.1636067Z env: 2024-12-18T00:05:26.1636326Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:05:26.1636642Z DRIVER_VERSION: 550.54.15 2024-12-18T00:05:26.1636954Z ##[endgroup] 2024-12-18T00:05:26.2809319Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-12-18T00:05:26.2810005Z + pre_install_nvidia_driver_amzn2 2024-12-18T00:05:26.2811330Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-12-18T00:05:26.7887489Z No match for argument: nvidia-driver-latest-dkms 2024-12-18T00:05:26.7888156Z No packages marked for removal. 2024-12-18T00:05:26.7957380Z Dependencies resolved. 2024-12-18T00:05:26.7965197Z Nothing to do. 2024-12-18T00:05:26.7965860Z Complete! 2024-12-18T00:05:26.9351069Z + install_nvidia_driver_common 2024-12-18T00:05:26.9353223Z + echo 'Before installing NVIDIA driver' 2024-12-18T00:05:26.9354775Z + lspci 2024-12-18T00:05:26.9356507Z Before installing NVIDIA driver 2024-12-18T00:05:27.0589658Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-12-18T00:05:27.0590378Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-12-18T00:05:27.0591097Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-12-18T00:05:27.0591761Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-12-18T00:05:27.0592354Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-12-18T00:05:27.0593024Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-12-18T00:05:27.0593656Z 00:1b.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-12-18T00:05:27.0594247Z 00:1c.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-12-18T00:05:27.0595084Z 00:1d.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-12-18T00:05:27.0595694Z 00:1e.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-12-18T00:05:27.0596304Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-12-18T00:05:27.0596926Z + lsmod 2024-12-18T00:05:27.0625711Z Module Size Used by 2024-12-18T00:05:27.0626116Z xt_conntrack 16384 1 2024-12-18T00:05:27.0626463Z nft_chain_nat 16384 3 2024-12-18T00:05:27.0626785Z xt_MASQUERADE 20480 1 2024-12-18T00:05:27.0627174Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-12-18T00:05:27.0627601Z nf_conntrack_netlink 57344 0 2024-12-18T00:05:27.0628108Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-12-18T00:05:27.0628672Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-12-18T00:05:27.0629063Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-12-18T00:05:27.0629463Z xfrm_user 57344 1 2024-12-18T00:05:27.0629814Z xfrm_algo 16384 1 xfrm_user 2024-12-18T00:05:27.0630193Z xt_addrtype 16384 2 2024-12-18T00:05:27.0630538Z nft_compat 20480 4 2024-12-18T00:05:27.0630936Z nf_tables 311296 57 nft_compat,nft_chain_nat 2024-12-18T00:05:27.0631564Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-12-18T00:05:27.0632034Z br_netfilter 36864 0 2024-12-18T00:05:27.0632383Z bridge 323584 1 br_netfilter 2024-12-18T00:05:27.0632756Z stp 16384 1 bridge 2024-12-18T00:05:27.0633115Z llc 16384 2 bridge,stp 2024-12-18T00:05:27.0633454Z overlay 167936 0 2024-12-18T00:05:27.0633767Z tls 114688 0 2024-12-18T00:05:27.0634078Z nls_ascii 16384 1 2024-12-18T00:05:27.0634390Z nls_cp437 20480 1 2024-12-18T00:05:27.0634699Z vfat 24576 1 2024-12-18T00:05:27.0635001Z fat 86016 1 vfat 2024-12-18T00:05:27.0635557Z sunrpc 692224 1 2024-12-18T00:05:27.0635869Z ena 167936 0 2024-12-18T00:05:27.0636180Z i8042 45056 0 2024-12-18T00:05:27.0636500Z serio 28672 3 i8042 2024-12-18T00:05:27.0636834Z skx_edac_common 28672 0 2024-12-18T00:05:27.0637160Z button 24576 0 2024-12-18T00:05:27.0637484Z ghash_clmulni_intel 16384 0 2024-12-18T00:05:27.0637814Z sch_fq_codel 20480 33 2024-12-18T00:05:27.0638136Z dm_mod 188416 0 2024-12-18T00:05:27.0638430Z fuse 163840 1 2024-12-18T00:05:27.0638742Z configfs 57344 1 2024-12-18T00:05:27.0639062Z dax 45056 1 dm_mod 2024-12-18T00:05:27.0639400Z loop 36864 0 2024-12-18T00:05:27.0639713Z dmi_sysfs 20480 0 2024-12-18T00:05:27.0640011Z crc32_pclmul 16384 0 2024-12-18T00:05:27.0640328Z crc32c_intel 24576 0 2024-12-18T00:05:27.0640650Z efivarfs 24576 1 2024-12-18T00:05:27.0640967Z + modinfo nvidia 2024-12-18T00:05:27.0646270Z filename: /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-12-18T00:05:27.0646836Z alias: char-major-195-* 2024-12-18T00:05:27.0647174Z version: 550.54.15 2024-12-18T00:05:27.0647496Z supported: external 2024-12-18T00:05:27.0647867Z license: NVIDIA 2024-12-18T00:05:27.0648195Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-12-18T00:05:27.0648639Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-12-18T00:05:27.0649024Z srcversion: 833721318DA517F0C2FEC97 2024-12-18T00:05:27.0649414Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-12-18T00:05:27.0649834Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-12-18T00:05:27.0650254Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-12-18T00:05:27.0650642Z depends: i2c-core,drm 2024-12-18T00:05:27.0650944Z retpoline: Y 2024-12-18T00:05:27.0651347Z name: nvidia 2024-12-18T00:05:27.0651805Z vermagic: 6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-12-18T00:05:27.0652393Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-12-18T00:05:27.0652944Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-12-18T00:05:27.0653450Z parm: NVreg_ResmanDebugLevel:int 2024-12-18T00:05:27.0653840Z parm: NVreg_RmLogonRC:int 2024-12-18T00:05:27.0654219Z parm: NVreg_ModifyDeviceFiles:int 2024-12-18T00:05:27.0654616Z parm: NVreg_DeviceFileUID:int 2024-12-18T00:05:27.0655000Z parm: NVreg_DeviceFileGID:int 2024-12-18T00:05:27.0655364Z parm: NVreg_DeviceFileMode:int 2024-12-18T00:05:27.0655804Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-12-18T00:05:27.0669829Z parm: NVreg_UsePageAttributeTable:int 2024-12-18T00:05:27.0670380Z parm: NVreg_EnablePCIeGen3:int 2024-12-18T00:05:27.0670789Z parm: NVreg_EnableMSI:int 2024-12-18T00:05:27.0671364Z parm: NVreg_TCEBypassMode:int 2024-12-18T00:05:27.0671768Z parm: NVreg_EnableStreamMemOPs:int 2024-12-18T00:05:27.0672223Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-12-18T00:05:27.0672716Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-12-18T00:05:27.0673169Z parm: NVreg_EnableS0ixPowerManagement:int 2024-12-18T00:05:27.0673676Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-12-18T00:05:27.0674183Z parm: NVreg_DynamicPowerManagement:int 2024-12-18T00:05:27.0674695Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-12-18T00:05:27.0675203Z parm: NVreg_EnableGpuFirmware:int 2024-12-18T00:05:27.0675622Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-12-18T00:05:27.0676062Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-12-18T00:05:27.0676527Z parm: NVreg_EnableUserNUMAManagement:int 2024-12-18T00:05:27.0676957Z parm: NVreg_MemoryPoolSize:int 2024-12-18T00:05:27.0677501Z parm: NVreg_KMallocHeapMaxSize:int 2024-12-18T00:05:27.0677916Z parm: NVreg_VMallocHeapMaxSize:int 2024-12-18T00:05:27.0678308Z parm: NVreg_IgnoreMMIOCheck:int 2024-12-18T00:05:27.0678701Z parm: NVreg_NvLinkDisable:int 2024-12-18T00:05:27.0679129Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-12-18T00:05:27.0679579Z parm: NVreg_RegisterPCIDriver:int 2024-12-18T00:05:27.0679987Z parm: NVreg_EnableResizableBar:int 2024-12-18T00:05:27.0680386Z parm: NVreg_EnableDbgBreakpoint:int 2024-12-18T00:05:27.0680815Z parm: NVreg_EnableNonblockingOpen:int 2024-12-18T00:05:27.0681236Z parm: NVreg_RegistryDwords:charp 2024-12-18T00:05:27.0681654Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-12-18T00:05:27.0682062Z parm: NVreg_RmMsg:charp 2024-12-18T00:05:27.0682408Z parm: NVreg_GpuBlacklist:charp 2024-12-18T00:05:27.0682819Z parm: NVreg_TemporaryFilePath:charp 2024-12-18T00:05:27.0683214Z parm: NVreg_ExcludedGpus:charp 2024-12-18T00:05:27.0683602Z parm: NVreg_DmaRemapPeerMmio:int 2024-12-18T00:05:27.0684002Z parm: NVreg_RmNvlinkBandwidth:charp 2024-12-18T00:05:27.0684507Z parm: NVreg_ImexChannelCount:int 2024-12-18T00:05:27.0684886Z parm: rm_firmware_active:charp 2024-12-18T00:05:27.0685240Z + HAS_NVIDIA_DRIVER=0 2024-12-18T00:05:27.0685551Z ++ command -v nvidia-smi 2024-12-18T00:05:27.0685864Z + '[' -x /usr/bin/nvidia-smi ']' 2024-12-18T00:05:27.0686164Z + set +e 2024-12-18T00:05:27.0686542Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-12-18T00:05:32.0130456Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-12-18T00:05:32.0130864Z + NVIDIA_SMI_STATUS=0 2024-12-18T00:05:32.0131160Z + '[' 0 -ne 0 ']' 2024-12-18T00:05:32.0131418Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-12-18T00:05:32.0131744Z + HAS_NVIDIA_DRIVER=1 2024-12-18T00:05:32.0132535Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-12-18T00:05:32.0133166Z + set -e 2024-12-18T00:05:32.0133405Z + '[' 1 -eq 0 ']' 2024-12-18T00:05:32.0133866Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-12-18T00:05:32.0134442Z + post_install_nvidia_driver_common 2024-12-18T00:05:32.0136583Z + sudo modprobe nvidia 2024-12-18T00:05:32.1804710Z + echo 'After installing NVIDIA driver' 2024-12-18T00:05:32.1805220Z + lspci 2024-12-18T00:05:32.1806095Z After installing NVIDIA driver 2024-12-18T00:05:32.1941036Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-12-18T00:05:32.1941728Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-12-18T00:05:32.1942450Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-12-18T00:05:32.1943134Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-12-18T00:05:32.1943777Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-12-18T00:05:32.1944472Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-12-18T00:05:32.1945115Z 00:1b.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-12-18T00:05:32.1945711Z 00:1c.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-12-18T00:05:32.1946306Z 00:1d.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-12-18T00:05:32.1946876Z 00:1e.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-12-18T00:05:32.1947512Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-12-18T00:05:32.1948027Z + lsmod 2024-12-18T00:05:32.1967637Z Module Size Used by 2024-12-18T00:05:32.1968133Z nvidia_uvm 4706304 0 2024-12-18T00:05:32.1968898Z nvidia 54071296 1 nvidia_uvm 2024-12-18T00:05:32.1969347Z drm 602112 1 nvidia 2024-12-18T00:05:32.1969752Z drm_panel_orientation_quirks 32768 1 drm 2024-12-18T00:05:32.1970360Z backlight 24576 1 drm 2024-12-18T00:05:32.1970722Z i2c_core 106496 2 nvidia,drm 2024-12-18T00:05:32.1971086Z xt_conntrack 16384 1 2024-12-18T00:05:32.1971414Z nft_chain_nat 16384 3 2024-12-18T00:05:32.1971743Z xt_MASQUERADE 20480 1 2024-12-18T00:05:32.1972103Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-12-18T00:05:32.1972518Z nf_conntrack_netlink 57344 0 2024-12-18T00:05:32.1973018Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-12-18T00:05:32.1973567Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-12-18T00:05:32.1973956Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-12-18T00:05:32.1974321Z xfrm_user 57344 1 2024-12-18T00:05:32.1974636Z xfrm_algo 16384 1 xfrm_user 2024-12-18T00:05:32.1975096Z xt_addrtype 16384 2 2024-12-18T00:05:32.1975418Z nft_compat 20480 4 2024-12-18T00:05:32.1975787Z nf_tables 311296 57 nft_compat,nft_chain_nat 2024-12-18T00:05:32.1976293Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-12-18T00:05:32.1976737Z br_netfilter 36864 0 2024-12-18T00:05:32.1977073Z bridge 323584 1 br_netfilter 2024-12-18T00:05:32.1977433Z stp 16384 1 bridge 2024-12-18T00:05:32.1977778Z llc 16384 2 bridge,stp 2024-12-18T00:05:32.1978122Z overlay 167936 0 2024-12-18T00:05:32.1978414Z tls 114688 0 2024-12-18T00:05:32.1978842Z nls_ascii 16384 1 2024-12-18T00:05:32.1979324Z nls_cp437 20480 1 2024-12-18T00:05:32.1979650Z vfat 24576 1 2024-12-18T00:05:32.1980005Z fat 86016 1 vfat 2024-12-18T00:05:32.1980331Z sunrpc 692224 1 2024-12-18T00:05:32.1980652Z ena 167936 0 2024-12-18T00:05:32.1981145Z i8042 45056 0 2024-12-18T00:05:32.1981475Z serio 28672 3 i8042 2024-12-18T00:05:32.1981837Z skx_edac_common 28672 0 2024-12-18T00:05:32.1982148Z button 24576 0 2024-12-18T00:05:32.1982478Z ghash_clmulni_intel 16384 0 2024-12-18T00:05:32.1982819Z sch_fq_codel 20480 33 2024-12-18T00:05:32.1983149Z dm_mod 188416 0 2024-12-18T00:05:32.1983456Z fuse 163840 1 2024-12-18T00:05:32.1983774Z configfs 57344 1 2024-12-18T00:05:32.1984101Z dax 45056 1 dm_mod 2024-12-18T00:05:32.1984446Z loop 36864 0 2024-12-18T00:05:32.1984765Z dmi_sysfs 20480 0 2024-12-18T00:05:32.1985184Z crc32_pclmul 16384 0 2024-12-18T00:05:32.1985498Z crc32c_intel 24576 0 2024-12-18T00:05:32.1985810Z efivarfs 24576 1 2024-12-18T00:05:32.1986119Z + modinfo nvidia 2024-12-18T00:05:32.1986619Z filename: /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-12-18T00:05:32.1987191Z alias: char-major-195-* 2024-12-18T00:05:32.1987510Z version: 550.54.15 2024-12-18T00:05:32.1987815Z supported: external 2024-12-18T00:05:32.1988117Z license: NVIDIA 2024-12-18T00:05:32.1988445Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-12-18T00:05:32.1988862Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-12-18T00:05:32.1989240Z srcversion: 833721318DA517F0C2FEC97 2024-12-18T00:05:32.1989629Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-12-18T00:05:32.1990052Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-12-18T00:05:32.1990466Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-12-18T00:05:32.1990851Z depends: i2c-core,drm 2024-12-18T00:05:32.1991153Z retpoline: Y 2024-12-18T00:05:32.1991421Z name: nvidia 2024-12-18T00:05:32.1991865Z vermagic: 6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-12-18T00:05:32.1992450Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-12-18T00:05:32.1993078Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-12-18T00:05:32.1993603Z parm: NVreg_ResmanDebugLevel:int 2024-12-18T00:05:32.1993974Z parm: NVreg_RmLogonRC:int 2024-12-18T00:05:32.1994352Z parm: NVreg_ModifyDeviceFiles:int 2024-12-18T00:05:32.1994746Z parm: NVreg_DeviceFileUID:int 2024-12-18T00:05:32.1995122Z parm: NVreg_DeviceFileGID:int 2024-12-18T00:05:32.1995501Z parm: NVreg_DeviceFileMode:int 2024-12-18T00:05:32.1995931Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-12-18T00:05:32.1996404Z parm: NVreg_UsePageAttributeTable:int 2024-12-18T00:05:32.1996815Z parm: NVreg_EnablePCIeGen3:int 2024-12-18T00:05:32.1997186Z parm: NVreg_EnableMSI:int 2024-12-18T00:05:32.1997547Z parm: NVreg_TCEBypassMode:int 2024-12-18T00:05:32.1997924Z parm: NVreg_EnableStreamMemOPs:int 2024-12-18T00:05:32.1998372Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-12-18T00:05:32.1998856Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-12-18T00:05:32.1999324Z parm: NVreg_EnableS0ixPowerManagement:int 2024-12-18T00:05:32.1999824Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-12-18T00:05:32.2000304Z parm: NVreg_DynamicPowerManagement:int 2024-12-18T00:05:32.2000813Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-12-18T00:05:32.2001314Z parm: NVreg_EnableGpuFirmware:int 2024-12-18T00:05:32.2001728Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-12-18T00:05:32.2002178Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-12-18T00:05:32.2002727Z parm: NVreg_EnableUserNUMAManagement:int 2024-12-18T00:05:32.2003132Z parm: NVreg_MemoryPoolSize:int 2024-12-18T00:05:32.2003519Z parm: NVreg_KMallocHeapMaxSize:int 2024-12-18T00:05:32.2003991Z parm: NVreg_VMallocHeapMaxSize:int 2024-12-18T00:05:32.2004385Z parm: NVreg_IgnoreMMIOCheck:int 2024-12-18T00:05:32.2004750Z parm: NVreg_NvLinkDisable:int 2024-12-18T00:05:32.2005162Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-12-18T00:05:32.2005592Z parm: NVreg_RegisterPCIDriver:int 2024-12-18T00:05:32.2005989Z parm: NVreg_EnableResizableBar:int 2024-12-18T00:05:32.2006392Z parm: NVreg_EnableDbgBreakpoint:int 2024-12-18T00:05:32.2006788Z parm: NVreg_EnableNonblockingOpen:int 2024-12-18T00:05:32.2007195Z parm: NVreg_RegistryDwords:charp 2024-12-18T00:05:32.2007600Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-12-18T00:05:32.2008002Z parm: NVreg_RmMsg:charp 2024-12-18T00:05:32.2008359Z parm: NVreg_GpuBlacklist:charp 2024-12-18T00:05:32.2008752Z parm: NVreg_TemporaryFilePath:charp 2024-12-18T00:05:32.2009130Z parm: NVreg_ExcludedGpus:charp 2024-12-18T00:05:32.2009515Z parm: NVreg_DmaRemapPeerMmio:int 2024-12-18T00:05:32.2009910Z parm: NVreg_RmNvlinkBandwidth:charp 2024-12-18T00:05:32.2010311Z parm: NVreg_ImexChannelCount:int 2024-12-18T00:05:32.2010701Z parm: rm_firmware_active:charp 2024-12-18T00:05:32.2011045Z + set +e 2024-12-18T00:05:32.2011283Z + nvidia-smi 2024-12-18T00:05:34.7359574Z Wed Dec 18 00:05:34 2024 2024-12-18T00:05:34.7360117Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:05:34.7360766Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-12-18T00:05:34.7361393Z |-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:05:34.7362031Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-12-18T00:05:34.7362704Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-12-18T00:05:34.7363272Z | | | MIG M. | 2024-12-18T00:05:34.7363982Z |=========================================+========================+======================| 2024-12-18T00:05:34.7448084Z | 0 Tesla T4 Off | 00000000:00:1B.0 Off | 0 | 2024-12-18T00:05:34.7448690Z | N/A 31C P0 26W / 70W | 0MiB / 15360MiB | 0% Default | 2024-12-18T00:05:34.7449180Z | | | N/A | 2024-12-18T00:05:34.7449684Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:05:34.7533166Z | 1 Tesla T4 Off | 00000000:00:1C.0 Off | 0 | 2024-12-18T00:05:34.7533729Z | N/A 32C P0 26W / 70W | 0MiB / 15360MiB | 6% Default | 2024-12-18T00:05:34.7534226Z | | | N/A | 2024-12-18T00:05:34.7534722Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:05:34.7623604Z | 2 Tesla T4 Off | 00000000:00:1D.0 Off | 0 | 2024-12-18T00:05:34.7624199Z | N/A 31C P0 26W / 70W | 0MiB / 15360MiB | 0% Default | 2024-12-18T00:05:34.7624699Z | | | N/A | 2024-12-18T00:05:34.7625205Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:05:34.7716676Z | 3 Tesla T4 Off | 00000000:00:1E.0 Off | 0 | 2024-12-18T00:05:34.7717254Z | N/A 30C P0 25W / 70W | 0MiB / 15360MiB | 0% Default | 2024-12-18T00:05:34.7717742Z | | | N/A | 2024-12-18T00:05:34.7718470Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:05:34.7720397Z 2024-12-18T00:05:34.7720892Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:05:34.7721418Z | Processes: | 2024-12-18T00:05:34.7722136Z | GPU GI CI PID Type Process name GPU Memory | 2024-12-18T00:05:34.7722669Z | ID ID Usage | 2024-12-18T00:05:34.7723100Z |=========================================================================================| 2024-12-18T00:05:34.7740195Z | No running processes found | 2024-12-18T00:05:34.7740906Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:05:37.1661031Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-12-18T00:05:39.6704980Z Tesla T4 2024-12-18T00:05:41.7257082Z + NVIDIA_SMI_STATUS=0 2024-12-18T00:05:41.7257601Z + '[' 0 -eq 0 ']' 2024-12-18T00:05:41.7258094Z + echo 'INFO: Ignoring allowed status 0' 2024-12-18T00:05:41.7258752Z + set -e 2024-12-18T00:05:41.7259263Z INFO: Ignoring allowed status 0 2024-12-18T00:05:41.7263542Z == Installing nvidia container toolkit for amzn2023 == 2024-12-18T00:05:41.7267241Z + sudo yum install -y yum-utils 2024-12-18T00:05:42.1778079Z Last metadata expiration check: 0:16:19 ago on Tue Dec 17 23:49:23 2024. 2024-12-18T00:05:42.2058464Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-12-18T00:05:42.2451911Z Dependencies resolved. 2024-12-18T00:05:42.2631432Z Nothing to do. 2024-12-18T00:05:42.2631823Z Complete! 2024-12-18T00:05:42.3918118Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-12-18T00:05:42.3918920Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-12-18T00:05:42.3920300Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-12-18T00:05:42.7228875Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-12-18T00:05:42.7974439Z + sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2 2024-12-18T00:05:43.3549849Z nvidia-container-toolkit 10 kB/s | 833 B 00:00 2024-12-18T00:05:43.3820924Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-12-18T00:05:43.4225808Z Dependencies resolved. 2024-12-18T00:05:43.4407954Z ================================================================================ 2024-12-18T00:05:43.4409081Z Package Arch Version Repository Size 2024-12-18T00:05:43.4409646Z ================================================================================ 2024-12-18T00:05:43.4410038Z Downgrading: 2024-12-18T00:05:43.4410490Z nvidia-container-toolkit x86_64 1.16.2-1 nvidia-container-toolkit 1.2 M 2024-12-18T00:05:43.4411225Z nvidia-container-toolkit-base x86_64 1.16.2-1 nvidia-container-toolkit 5.6 M 2024-12-18T00:05:43.4411675Z 2024-12-18T00:05:43.4411786Z Transaction Summary 2024-12-18T00:05:43.4412097Z ================================================================================ 2024-12-18T00:05:43.4412480Z Downgrade 2 Packages 2024-12-18T00:05:43.4412659Z 2024-12-18T00:05:43.4412775Z Total download size: 6.8 M 2024-12-18T00:05:43.4413095Z Downloading Packages: 2024-12-18T00:05:43.5601654Z (1/2): nvidia-container-toolkit-1.16.2-1.x86_64 10 MB/s | 1.2 MB 00:00 2024-12-18T00:05:43.7548149Z (2/2): nvidia-container-toolkit-base-1.16.2-1.x 18 MB/s | 5.6 MB 00:00 2024-12-18T00:05:43.7556566Z -------------------------------------------------------------------------------- 2024-12-18T00:05:43.7558302Z Total 22 MB/s | 6.8 MB 00:00 2024-12-18T00:05:43.7560721Z Running transaction check 2024-12-18T00:05:43.7692377Z Transaction check succeeded. 2024-12-18T00:05:43.7692809Z Running transaction test 2024-12-18T00:05:43.8012026Z Transaction test succeeded. 2024-12-18T00:05:43.8013291Z Running transaction 2024-12-18T00:05:44.7995292Z Preparing : 1/1 2024-12-18T00:05:45.0335390Z Downgrading : nvidia-container-toolkit-base-1.16.2-1.x86_64 1/4 2024-12-18T00:05:45.1044362Z Downgrading : nvidia-container-toolkit-1.16.2-1.x86_64 2/4 2024-12-18T00:05:45.1960995Z Running scriptlet: nvidia-container-toolkit-1.16.2-1.x86_64 2/4 2024-12-18T00:05:45.1961713Z Cleanup : nvidia-container-toolkit-1.17.3-1.x86_64 3/4 2024-12-18T00:05:45.2701660Z Running scriptlet: nvidia-container-toolkit-1.17.3-1.x86_64 3/4 2024-12-18T00:05:45.3300874Z Cleanup : nvidia-container-toolkit-base-1.17.3-1.x86_64 4/4 2024-12-18T00:07:14.1225449Z Running scriptlet: nvidia-container-toolkit-1.16.2-1.x86_64 4/4 2024-12-18T00:07:14.1227190Z Verifying : nvidia-container-toolkit-1.16.2-1.x86_64 1/4 2024-12-18T00:07:14.1227917Z Verifying : nvidia-container-toolkit-1.17.3-1.x86_64 2/4 2024-12-18T00:07:14.1228604Z Verifying : nvidia-container-toolkit-base-1.16.2-1.x86_64 3/4 2024-12-18T00:07:14.2773144Z Verifying : nvidia-container-toolkit-base-1.17.3-1.x86_64 4/4================================================================================ 2024-12-18T00:07:14.2773886Z WARNING: 2024-12-18T00:07:14.2774174Z A newer release of "Amazon Linux" is available. 2024-12-18T00:07:14.2774497Z 2024-12-18T00:07:14.2774607Z Available Versions: 2024-12-18T00:07:14.2774795Z 2024-12-18T00:07:14.2774903Z Version 2023.6.20241010: 2024-12-18T00:07:14.2775320Z Run the following command to upgrade to 2023.6.20241010: 2024-12-18T00:07:14.2775912Z 2024-12-18T00:07:14.2776078Z dnf upgrade --releasever=2023.6.20241010 2024-12-18T00:07:14.2776341Z 2024-12-18T00:07:14.2776458Z Release notes: 2024-12-18T00:07:14.2776951Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241010.html 2024-12-18T00:07:14.2777423Z 2024-12-18T00:07:14.2777532Z Version 2023.6.20241028: 2024-12-18T00:07:14.2777911Z Run the following command to upgrade to 2023.6.20241028: 2024-12-18T00:07:14.2778229Z 2024-12-18T00:07:14.2778365Z dnf upgrade --releasever=2023.6.20241028 2024-12-18T00:07:14.2778720Z 2024-12-18T00:07:14.2778840Z Release notes: 2024-12-18T00:07:14.2779511Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241028.html 2024-12-18T00:07:14.2779979Z 2024-12-18T00:07:14.2780088Z Version 2023.6.20241031: 2024-12-18T00:07:14.2780480Z Run the following command to upgrade to 2023.6.20241031: 2024-12-18T00:07:14.2780820Z 2024-12-18T00:07:14.2780966Z dnf upgrade --releasever=2023.6.20241031 2024-12-18T00:07:14.2781244Z 2024-12-18T00:07:14.2781349Z Release notes: 2024-12-18T00:07:14.2781845Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241031.html 2024-12-18T00:07:14.2782308Z 2024-12-18T00:07:14.2782429Z Version 2023.6.20241111: 2024-12-18T00:07:14.2782804Z Run the following command to upgrade to 2023.6.20241111: 2024-12-18T00:07:14.2783135Z 2024-12-18T00:07:14.2783273Z dnf upgrade --releasever=2023.6.20241111 2024-12-18T00:07:14.2783547Z 2024-12-18T00:07:14.2783652Z Release notes: 2024-12-18T00:07:14.2784150Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241111.html 2024-12-18T00:07:14.2784615Z 2024-12-18T00:07:14.2784735Z Version 2023.6.20241121: 2024-12-18T00:07:14.2785120Z Run the following command to upgrade to 2023.6.20241121: 2024-12-18T00:07:14.2785433Z 2024-12-18T00:07:14.2785726Z dnf upgrade --releasever=2023.6.20241121 2024-12-18T00:07:14.2786009Z 2024-12-18T00:07:14.2786128Z Release notes: 2024-12-18T00:07:14.2786630Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241121.html 2024-12-18T00:07:14.2787109Z 2024-12-18T00:07:14.2787217Z Version 2023.6.20241212: 2024-12-18T00:07:14.2787604Z Run the following command to upgrade to 2023.6.20241212: 2024-12-18T00:07:14.2787918Z 2024-12-18T00:07:14.2788070Z dnf upgrade --releasever=2023.6.20241212 2024-12-18T00:07:14.2788333Z 2024-12-18T00:07:14.2788457Z Release notes: 2024-12-18T00:07:14.2788942Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241212.html 2024-12-18T00:07:14.2789420Z 2024-12-18T00:07:14.2789560Z ================================================================================ 2024-12-18T00:07:14.3109509Z 2024-12-18T00:07:14.3109699Z 2024-12-18T00:07:14.3109807Z Downgraded: 2024-12-18T00:07:14.3110308Z nvidia-container-toolkit-1.16.2-1.x86_64 2024-12-18T00:07:14.3111164Z nvidia-container-toolkit-base-1.16.2-1.x86_64 2024-12-18T00:07:14.3111586Z 2024-12-18T00:07:14.3111686Z Complete! 2024-12-18T00:07:14.3620367Z + sudo systemctl restart docker 2024-12-18T00:07:22.9818084Z Wed Dec 18 00:07:22 2024 2024-12-18T00:07:22.9818750Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:07:22.9819593Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-12-18T00:07:22.9820228Z |-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:07:22.9820885Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-12-18T00:07:22.9821549Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-12-18T00:07:22.9822311Z | | | MIG M. | 2024-12-18T00:07:22.9823034Z |=========================================+========================+======================| 2024-12-18T00:07:22.9930370Z | 0 Tesla T4 On | 00000000:00:1B.0 Off | 0 | 2024-12-18T00:07:22.9930970Z | N/A 27C P0 25W / 70W | 0MiB / 15360MiB | 0% Default | 2024-12-18T00:07:22.9931460Z | | | N/A | 2024-12-18T00:07:22.9931953Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:07:23.0020620Z | 1 Tesla T4 On | 00000000:00:1C.0 Off | 0 | 2024-12-18T00:07:23.0021219Z | N/A 28C P0 26W / 70W | 0MiB / 15360MiB | 0% Default | 2024-12-18T00:07:23.0021724Z | | | N/A | 2024-12-18T00:07:23.0022418Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:07:23.0115737Z | 2 Tesla T4 On | 00000000:00:1D.0 Off | 0 | 2024-12-18T00:07:23.0116376Z | N/A 27C P0 25W / 70W | 0MiB / 15360MiB | 0% Default | 2024-12-18T00:07:23.0116864Z | | | N/A | 2024-12-18T00:07:23.0117354Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:07:23.0207138Z | 3 Tesla T4 On | 00000000:00:1E.0 Off | 0 | 2024-12-18T00:07:23.0207791Z | N/A 26C P0 25W / 70W | 0MiB / 15360MiB | 6% Default | 2024-12-18T00:07:23.0208285Z | | | N/A | 2024-12-18T00:07:23.0209006Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:07:23.0210975Z 2024-12-18T00:07:23.0211472Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:07:23.0211996Z | Processes: | 2024-12-18T00:07:23.0212562Z | GPU GI CI PID Type Process name GPU Memory | 2024-12-18T00:07:23.0213089Z | ID ID Usage | 2024-12-18T00:07:23.0213518Z |=========================================================================================| 2024-12-18T00:07:23.0229810Z | No running processes found | 2024-12-18T00:07:23.0230434Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:07:24.3947507Z Command completed after 1 attempt(s). 2024-12-18T00:07:24.4045812Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-12-18T00:07:24.4046495Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-12-18T00:07:24.4047084Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-12-18T00:07:24.4047622Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:07:24.4059266Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:24.4059763Z env: 2024-12-18T00:07:24.4060045Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:24.4060464Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:24.4060893Z ##[endgroup] 2024-12-18T00:07:24.7340582Z Defaulting to user installation because normal site-packages is not writeable 2024-12-18T00:07:25.1460373Z Collecting psutil==5.9.1 2024-12-18T00:07:25.1873909Z Downloading psutil-5.9.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2024-12-18T00:07:25.2394003Z Collecting nvidia-ml-py==11.525.84 2024-12-18T00:07:25.2433082Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2024-12-18T00:07:25.3382728Z Installing collected packages: psutil, nvidia-ml-py 2024-12-18T00:07:25.5069414Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1 2024-12-18T00:07:25.7088473Z Prepare all required actions 2024-12-18T00:07:25.7089320Z Getting action download info 2024-12-18T00:07:25.8525389Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-12-18T00:07:26.0886935Z Download action repository 'actions/download-artifact@v4' (SHA:fa0a91b85d4f404e444e00e005971372dc801d16) 2024-12-18T00:07:26.4040188Z ##[group]Run ./.github/actions/download-build-artifacts 2024-12-18T00:07:26.4040623Z with: 2024-12-18T00:07:26.4040897Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-12-18T00:07:26.4041287Z s3-bucket: gha-artifacts 2024-12-18T00:07:26.4041596Z env: 2024-12-18T00:07:26.4041886Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:26.4042280Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:26.4042686Z ##[endgroup] 2024-12-18T00:07:26.4104259Z ##[group]Run seemethere/download-artifact-s3@v4 2024-12-18T00:07:26.4104660Z with: 2024-12-18T00:07:26.4104988Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-12-18T00:07:26.4105371Z s3-bucket: gha-artifacts 2024-12-18T00:07:26.4105673Z region: us-east-1 2024-12-18T00:07:26.4105943Z env: 2024-12-18T00:07:26.4106190Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:26.4106579Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:26.4107004Z ##[endgroup] 2024-12-18T00:07:26.9176399Z (node:57304) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-12-18T00:07:26.9177152Z 2024-12-18T00:07:26.9177373Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-12-18T00:07:26.9177995Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-12-18T00:07:26.9178764Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-12-18T00:07:27.1315107Z Found 1 objects with prefix pytorch/pytorch/12383255652/linux-focal-cuda11.8-py3.10-gcc9/ 2024-12-18T00:07:27.1315980Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-12-18T00:07:38.4380018Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-12-18T00:07:38.4385165Z Artifact download has finished successfully 2024-12-18T00:07:38.4629579Z ##[group]Run unzip -o artifacts.zip 2024-12-18T00:07:38.4630000Z unzip -o artifacts.zip 2024-12-18T00:07:38.4636589Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:38.4636985Z env: 2024-12-18T00:07:38.4637228Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:38.4637606Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:38.4637996Z ##[endgroup] 2024-12-18T00:07:38.4708567Z Archive: artifacts.zip 2024-12-18T00:07:38.4709239Z creating: dist/ 2024-12-18T00:07:40.5100845Z inflating: dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl 2024-12-18T00:07:40.5101483Z creating: build/custom_test_artifacts/ 2024-12-18T00:07:40.5101961Z creating: build/custom_test_artifacts/custom-op-build/ 2024-12-18T00:07:40.5102577Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-12-18T00:07:40.5103384Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-12-18T00:07:40.5108937Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-12-18T00:07:40.5109738Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-12-18T00:07:40.5110529Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-12-18T00:07:40.5111470Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-12-18T00:07:40.5112271Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-12-18T00:07:40.5113459Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-12-18T00:07:40.5114564Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-12-18T00:07:40.5115386Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-12-18T00:07:40.5116211Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-12-18T00:07:40.5117208Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-12-18T00:07:40.5118661Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-12-18T00:07:40.5120326Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-12-18T00:07:40.5121293Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-12-18T00:07:40.5122756Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-12-18T00:07:40.5123912Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-12-18T00:07:40.5124762Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-12-18T00:07:40.5125591Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-12-18T00:07:40.5172322Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-12-18T00:07:40.5173715Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-12-18T00:07:40.5174935Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-12-18T00:07:40.5176191Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-12-18T00:07:40.5177429Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-12-18T00:07:40.5178707Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-12-18T00:07:40.5180068Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-12-18T00:07:40.5181292Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-12-18T00:07:40.5182498Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-12-18T00:07:40.5217934Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-12-18T00:07:40.5259700Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-12-18T00:07:40.5260971Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-12-18T00:07:40.5262106Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-12-18T00:07:40.5263193Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-12-18T00:07:40.5264253Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-12-18T00:07:40.5265322Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-12-18T00:07:40.5266361Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-12-18T00:07:40.5267605Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-12-18T00:07:40.5327812Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-12-18T00:07:40.5390964Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-12-18T00:07:40.5392262Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-12-18T00:07:40.5393080Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-12-18T00:07:40.5393789Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-12-18T00:07:40.5394542Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-12-18T00:07:40.5395310Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-12-18T00:07:40.5396172Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-12-18T00:07:40.5397143Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-12-18T00:07:40.5398077Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-12-18T00:07:40.5398931Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-12-18T00:07:40.5399826Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-12-18T00:07:40.5400729Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-12-18T00:07:40.5401633Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-12-18T00:07:40.5402535Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-12-18T00:07:40.5403557Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-12-18T00:07:40.5419517Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-12-18T00:07:40.5565666Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-12-18T00:07:40.5566694Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-12-18T00:07:40.5567614Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-12-18T00:07:40.5568644Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-12-18T00:07:40.5569630Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-12-18T00:07:40.5570542Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-12-18T00:07:40.5571497Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-12-18T00:07:40.5572445Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-12-18T00:07:40.5573397Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-12-18T00:07:40.5574348Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-12-18T00:07:40.5575284Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-12-18T00:07:40.5592449Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-12-18T00:07:40.5676663Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-12-18T00:07:40.5678080Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-12-18T00:07:40.5678997Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-12-18T00:07:40.5679905Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-12-18T00:07:40.5680664Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-12-18T00:07:40.5681414Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-12-18T00:07:40.5682168Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-12-18T00:07:40.5682875Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-12-18T00:07:40.5683519Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-12-18T00:07:40.5684166Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-12-18T00:07:40.5808760Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-12-18T00:07:40.5874959Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-12-18T00:07:40.5875741Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-12-18T00:07:40.5876312Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-12-18T00:07:40.5876978Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-12-18T00:07:40.5881733Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-12-18T00:07:40.5882486Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-12-18T00:07:40.5883241Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-12-18T00:07:40.5884053Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-12-18T00:07:40.5884847Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-12-18T00:07:40.5885768Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-12-18T00:07:40.5887026Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-12-18T00:07:40.5887881Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-12-18T00:07:40.5888695Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-12-18T00:07:40.5889742Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-12-18T00:07:40.5891020Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-12-18T00:07:40.5892704Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-12-18T00:07:40.5893653Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-12-18T00:07:40.5894842Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-12-18T00:07:40.5895870Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-12-18T00:07:40.5896731Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-12-18T00:07:40.5897549Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-12-18T00:07:40.5946061Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-12-18T00:07:40.5947307Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-12-18T00:07:40.5948532Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-12-18T00:07:40.5950008Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-12-18T00:07:40.5951357Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-12-18T00:07:40.5952605Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-12-18T00:07:40.5953768Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-12-18T00:07:40.5954947Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-12-18T00:07:40.5956110Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-12-18T00:07:40.5993447Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-12-18T00:07:40.6035396Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-12-18T00:07:40.6036766Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-12-18T00:07:40.6037836Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-12-18T00:07:40.6038878Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-12-18T00:07:40.6039904Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-12-18T00:07:40.6040936Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-12-18T00:07:40.6041933Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-12-18T00:07:40.6042947Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-12-18T00:07:40.6104173Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-12-18T00:07:40.6169306Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-12-18T00:07:40.6170478Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-12-18T00:07:40.6171302Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-12-18T00:07:40.6172001Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-12-18T00:07:40.6172733Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-12-18T00:07:40.6173508Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-12-18T00:07:40.6174398Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-12-18T00:07:40.6175392Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-12-18T00:07:40.6176358Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-12-18T00:07:40.6177242Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-12-18T00:07:40.6178163Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-12-18T00:07:40.6179365Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-12-18T00:07:40.6180322Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-12-18T00:07:40.6181275Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-12-18T00:07:40.6182380Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-12-18T00:07:40.6197709Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-12-18T00:07:40.6261837Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-12-18T00:07:40.6262899Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-12-18T00:07:40.6263823Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-12-18T00:07:40.6264660Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-12-18T00:07:40.6265424Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-12-18T00:07:40.6266163Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-12-18T00:07:40.6266926Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-12-18T00:07:40.6267633Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-12-18T00:07:40.6268535Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-12-18T00:07:40.6269207Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-12-18T00:07:40.6322727Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-12-18T00:07:40.6323373Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-12-18T00:07:40.6323975Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-12-18T00:07:40.6324698Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-12-18T00:07:40.6329512Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-12-18T00:07:40.6330339Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-12-18T00:07:40.6331183Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-12-18T00:07:40.6332065Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-12-18T00:07:40.6332933Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-12-18T00:07:40.6333898Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-12-18T00:07:40.6334896Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-12-18T00:07:40.6335783Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-12-18T00:07:40.6336664Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-12-18T00:07:40.6337685Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-12-18T00:07:40.6339268Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-12-18T00:07:40.6340905Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-12-18T00:07:40.6341967Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-12-18T00:07:40.6343089Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-12-18T00:07:40.6344467Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-12-18T00:07:40.6345460Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-12-18T00:07:40.6346380Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-12-18T00:07:40.6392804Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-12-18T00:07:40.6394252Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-12-18T00:07:40.6395621Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-12-18T00:07:40.6396940Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-12-18T00:07:40.6398235Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-12-18T00:07:40.6399466Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-12-18T00:07:40.6400701Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-12-18T00:07:40.6401951Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-12-18T00:07:40.6403195Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-12-18T00:07:40.6440437Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-12-18T00:07:40.6481683Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-12-18T00:07:40.6483130Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-12-18T00:07:40.6484289Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-12-18T00:07:40.6485417Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-12-18T00:07:40.6486515Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-12-18T00:07:40.6487628Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-12-18T00:07:40.6488702Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-12-18T00:07:40.6489777Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-12-18T00:07:40.6551433Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-12-18T00:07:40.6615158Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-12-18T00:07:40.6616527Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-12-18T00:07:40.6617441Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-12-18T00:07:40.6618208Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-12-18T00:07:40.6619294Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-12-18T00:07:40.6620164Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-12-18T00:07:40.6621132Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-12-18T00:07:40.6622422Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-12-18T00:07:40.6623499Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-12-18T00:07:40.6624502Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-12-18T00:07:40.6625718Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-12-18T00:07:40.6626855Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-12-18T00:07:40.6627898Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-12-18T00:07:40.6628932Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-12-18T00:07:40.6629961Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-12-18T00:07:40.6631171Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-12-18T00:07:40.6751492Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-12-18T00:07:40.6752748Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-12-18T00:07:40.6753779Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-12-18T00:07:40.6754914Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-12-18T00:07:40.6756002Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-12-18T00:07:40.6757021Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-12-18T00:07:40.6758076Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-12-18T00:07:40.6759146Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-12-18T00:07:40.6760213Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-12-18T00:07:40.6761281Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-12-18T00:07:40.6762331Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-12-18T00:07:40.6777863Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-12-18T00:07:40.6834016Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-12-18T00:07:40.6835308Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-12-18T00:07:40.6836285Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-12-18T00:07:40.6837170Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-12-18T00:07:40.6837974Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-12-18T00:07:40.6838766Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-12-18T00:07:40.6839571Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-12-18T00:07:40.6840322Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-12-18T00:07:40.6841028Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-12-18T00:07:40.6841737Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-12-18T00:07:40.6946156Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-12-18T00:07:40.6990073Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-12-18T00:07:40.6990682Z creating: build/lib/ 2024-12-18T00:07:40.7000453Z inflating: build/lib/libpthreadpool.a 2024-12-18T00:07:40.7008959Z inflating: build/lib/libcpuinfo.a 2024-12-18T00:07:40.7017222Z inflating: build/lib/libcpuinfo_internals.a 2024-12-18T00:07:40.7018036Z inflating: build/lib/libclog.a 2024-12-18T00:07:40.7112884Z inflating: build/lib/libprotobuf-lite.a 2024-12-18T00:07:40.7113386Z inflating: build/lib/libnnpack_reference_layers.a 2024-12-18T00:07:40.7579761Z inflating: build/lib/libprotobuf.a 2024-12-18T00:07:40.7649780Z inflating: build/lib/libgtest.a 2024-12-18T00:07:40.7726036Z inflating: build/lib/libbenchmark.a 2024-12-18T00:07:40.7733508Z inflating: build/lib/libittnotify.a 2024-12-18T00:07:40.7764897Z inflating: build/lib/libtensorpipe_uv.a 2024-12-18T00:07:40.7830093Z inflating: build/lib/libasmjit.a 2024-12-18T00:07:40.7850228Z inflating: build/lib/libnnpack.a 2024-12-18T00:07:40.7980704Z inflating: build/lib/libgloo.a 2024-12-18T00:07:40.8004343Z inflating: build/lib/libfmt.a 2024-12-18T00:07:40.8004777Z inflating: build/lib/libcaffe2_nvrtc.so 2024-12-18T00:07:40.8108399Z inflating: build/lib/libc10.so 2024-12-18T00:07:40.8300800Z inflating: build/lib/libmicrokernels-prod.a 2024-12-18T00:07:40.8301789Z inflating: build/lib/libtorch_global_deps.so 2024-12-18T00:07:40.8817517Z inflating: build/lib/libprotoc.a 2024-12-18T00:07:40.8838874Z inflating: build/lib/libpytorch_qnnpack.a 2024-12-18T00:07:40.8856687Z inflating: build/lib/libgmock.a 2024-12-18T00:07:40.8857122Z inflating: build/lib/libgtest_main.a 2024-12-18T00:07:40.8857854Z inflating: build/lib/libbenchmark_main.a 2024-12-18T00:07:41.9193913Z inflating: build/lib/libdnnl.a 2024-12-18T00:07:41.9774915Z inflating: build/lib/libtensorpipe.a 2024-12-18T00:07:41.9837384Z inflating: build/lib/libc10_cuda.so 2024-12-18T00:07:42.1092135Z inflating: build/lib/libfbgemm.a 2024-12-18T00:07:42.1092559Z inflating: build/lib/libgmock_main.a 2024-12-18T00:07:42.1617035Z inflating: build/lib/libkineto.a 2024-12-18T00:07:42.1877362Z inflating: build/lib/libtensorpipe_cuda.a 2024-12-18T00:07:42.1969924Z inflating: build/lib/libXNNPACK.a 2024-12-18T00:07:42.2355417Z inflating: build/lib/libgloo_cuda.a 2024-12-18T00:07:42.2398868Z inflating: build/lib/libonnx_proto.a 2024-12-18T00:07:42.3140422Z inflating: build/lib/libonnx.a 2024-12-18T00:07:42.4031006Z inflating: build/lib/libmicrokernels-all.a 2024-12-18T00:07:44.9716356Z inflating: build/lib/libtorch_cpu.so 2024-12-18T00:07:44.9719458Z inflating: build/lib/libunbox_lib.a 2024-12-18T00:07:44.9724071Z inflating: build/lib/libshm.so 2024-12-18T00:07:46.6857421Z inflating: build/lib/libtorch_cuda.so 2024-12-18T00:07:46.6857872Z inflating: build/lib/libtorch.so 2024-12-18T00:07:46.6859745Z inflating: build/lib/libc10d_cuda_test.so 2024-12-18T00:07:47.6480308Z inflating: build/lib/libtorch_cuda_linalg.so 2024-12-18T00:07:47.6500270Z inflating: build/lib/libjitbackend_test.so 2024-12-18T00:07:47.6573781Z inflating: build/lib/libtorchbind_test.so 2024-12-18T00:07:47.6600518Z inflating: build/lib/libbackend_with_compiler.so 2024-12-18T00:07:47.6626220Z inflating: build/lib/libaoti_custom_ops.so 2024-12-18T00:07:47.8759116Z inflating: build/lib/libtorch_python.so 2024-12-18T00:07:47.8795968Z inflating: build/lib/libnnapi_backend.so 2024-12-18T00:07:47.8796635Z creating: build/bin/ 2024-12-18T00:07:47.8852712Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-12-18T00:07:47.8908744Z inflating: build/bin/c10_DeviceGuard_test 2024-12-18T00:07:47.8969108Z inflating: build/bin/c10_Device_test 2024-12-18T00:07:47.9035810Z inflating: build/bin/c10_DispatchKeySet_test 2024-12-18T00:07:47.9095075Z inflating: build/bin/c10_Scalar_test 2024-12-18T00:07:47.9150200Z inflating: build/bin/c10_StreamGuard_test 2024-12-18T00:07:47.9207956Z inflating: build/bin/c10_SymInt_test 2024-12-18T00:07:47.9268762Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-12-18T00:07:47.9330555Z inflating: build/bin/c10_InlineStreamGuard_test 2024-12-18T00:07:47.9394539Z inflating: build/bin/c10_SizesAndStrides_test 2024-12-18T00:07:47.9449904Z inflating: build/bin/c10_ArrayRef_test 2024-12-18T00:07:47.9528135Z inflating: build/bin/c10_cow_test 2024-12-18T00:07:47.9581476Z inflating: build/bin/c10_ConstexprCrc_test 2024-12-18T00:07:47.9641685Z inflating: build/bin/c10_Bitset_test 2024-12-18T00:07:47.9696798Z inflating: build/bin/c10_DeadlockDetection_test 2024-12-18T00:07:47.9755441Z inflating: build/bin/c10_Half_test 2024-12-18T00:07:47.9816701Z inflating: build/bin/c10_LeftRight_test 2024-12-18T00:07:47.9878363Z inflating: build/bin/c10_Metaprogramming_test 2024-12-18T00:07:47.9938232Z inflating: build/bin/c10_NetworkFlow_test 2024-12-18T00:07:47.9994772Z inflating: build/bin/c10_Synchronized_test 2024-12-18T00:07:48.0055661Z inflating: build/bin/c10_ThreadLocal_test 2024-12-18T00:07:48.0113814Z inflating: build/bin/c10_TypeIndex_test 2024-12-18T00:07:48.0171212Z inflating: build/bin/c10_TypeList_test 2024-12-18T00:07:48.0224380Z inflating: build/bin/c10_TypeTraits_test 2024-12-18T00:07:48.0283506Z inflating: build/bin/c10_accumulate_test 2024-12-18T00:07:48.0346127Z inflating: build/bin/c10_bfloat16_test 2024-12-18T00:07:48.0403684Z inflating: build/bin/c10_bit_cast_test 2024-12-18T00:07:48.0465897Z inflating: build/bin/c10_complex_math_test 2024-12-18T00:07:48.0526866Z inflating: build/bin/c10_complex_test 2024-12-18T00:07:48.0581815Z inflating: build/bin/c10_error_test 2024-12-18T00:07:48.0641480Z inflating: build/bin/c10_exception_test 2024-12-18T00:07:48.0698124Z inflating: build/bin/c10_flags_test 2024-12-18T00:07:48.0756251Z inflating: build/bin/c10_generic_math_test 2024-12-18T00:07:48.0930561Z inflating: build/bin/c10_intrusive_ptr_test 2024-12-18T00:07:48.0986312Z inflating: build/bin/c10_irange_test 2024-12-18T00:07:48.1046806Z inflating: build/bin/c10_lazy_test 2024-12-18T00:07:48.1109068Z inflating: build/bin/c10_logging_test 2024-12-18T00:07:48.1193566Z inflating: build/bin/c10_optional_test 2024-12-18T00:07:48.1261242Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-12-18T00:07:48.1321579Z inflating: build/bin/c10_registry_test 2024-12-18T00:07:48.1484334Z inflating: build/bin/c10_small_vector_test 2024-12-18T00:07:48.1542238Z inflating: build/bin/c10_ssize_test 2024-12-18T00:07:48.1600774Z inflating: build/bin/c10_string_util_test 2024-12-18T00:07:48.1656830Z inflating: build/bin/c10_tempfile_test 2024-12-18T00:07:48.1722647Z inflating: build/bin/c10_string_view_test 2024-12-18T00:07:48.1784190Z inflating: build/bin/c10_typeid_test 2024-12-18T00:07:48.1837208Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-12-18T00:07:48.2299588Z inflating: build/bin/protoc-3.13.0.0 2024-12-18T00:07:48.2762729Z inflating: build/bin/protoc 2024-12-18T00:07:48.2820166Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-12-18T00:07:48.2879898Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-12-18T00:07:48.2939758Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-12-18T00:07:48.2997190Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-12-18T00:07:48.3052080Z inflating: build/bin/c10_cuda_CUDATest 2024-12-18T00:07:48.3109913Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-12-18T00:07:48.3170335Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-12-18T00:07:48.3227576Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-12-18T00:07:48.3636568Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-12-18T00:07:48.4055265Z inflating: build/bin/vec_test_all_types_AVX512 2024-12-18T00:07:48.4488628Z inflating: build/bin/vec_test_all_types_AVX2 2024-12-18T00:07:48.4547942Z inflating: build/bin/HashStoreTest 2024-12-18T00:07:48.4607404Z inflating: build/bin/FileStoreTest 2024-12-18T00:07:48.4668032Z inflating: build/bin/TCPStoreTest 2024-12-18T00:07:48.4726338Z inflating: build/bin/BackoffTest 2024-12-18T00:07:48.4740803Z inflating: build/bin/ProcessGroupMPITest 2024-12-18T00:07:48.4745214Z inflating: build/bin/torch_shm_manager 2024-12-18T00:07:48.4807099Z inflating: build/bin/test_edge_op_registration 2024-12-18T00:07:48.4807716Z inflating: build/bin/example_allreduce 2024-12-18T00:07:48.4866345Z inflating: build/bin/Dimname_test 2024-12-18T00:07:48.4948173Z inflating: build/bin/Dict_test 2024-12-18T00:07:48.5019474Z inflating: build/bin/MaybeOwned_test 2024-12-18T00:07:48.5083372Z inflating: build/bin/NamedTensor_test 2024-12-18T00:07:48.5148980Z inflating: build/bin/apply_utils_test 2024-12-18T00:07:48.5214764Z inflating: build/bin/atest 2024-12-18T00:07:48.5284630Z inflating: build/bin/basic 2024-12-18T00:07:48.5347089Z inflating: build/bin/broadcast_test 2024-12-18T00:07:48.5405123Z inflating: build/bin/cpu_allocator_test 2024-12-18T00:07:48.5469636Z inflating: build/bin/cpu_generator_test 2024-12-18T00:07:48.5528940Z inflating: build/bin/cpu_profiling_allocator_test 2024-12-18T00:07:48.5629376Z inflating: build/bin/cpu_rng_test 2024-12-18T00:07:48.5685917Z inflating: build/bin/dispatch_key_set_test 2024-12-18T00:07:48.5742520Z inflating: build/bin/dlconvertor_test 2024-12-18T00:07:48.5807568Z inflating: build/bin/extension_backend_test 2024-12-18T00:07:48.5868048Z inflating: build/bin/half_test 2024-12-18T00:07:48.5974489Z inflating: build/bin/ivalue_test 2024-12-18T00:07:48.6031414Z inflating: build/bin/lazy_tensor_test 2024-12-18T00:07:48.6091999Z inflating: build/bin/math_kernel_test 2024-12-18T00:07:48.6154373Z inflating: build/bin/memory_format_test 2024-12-18T00:07:48.6212352Z inflating: build/bin/memory_overlapping_test 2024-12-18T00:07:48.6270691Z inflating: build/bin/mobile_memory_cleanup 2024-12-18T00:07:48.6327604Z inflating: build/bin/operator_name_test 2024-12-18T00:07:48.6390405Z inflating: build/bin/native_test 2024-12-18T00:07:48.6449372Z inflating: build/bin/packedtensoraccessor_test 2024-12-18T00:07:48.6506576Z inflating: build/bin/operators_test 2024-12-18T00:07:48.6580445Z inflating: build/bin/pow_test 2024-12-18T00:07:48.6645161Z inflating: build/bin/quantized_test 2024-12-18T00:07:48.6700382Z inflating: build/bin/reduce_ops_test 2024-12-18T00:07:48.6758912Z inflating: build/bin/reportMemoryUsage_test 2024-12-18T00:07:48.6816242Z inflating: build/bin/StorageUtils_test 2024-12-18T00:07:48.6880441Z inflating: build/bin/scalar_tensor_test 2024-12-18T00:07:48.6946491Z inflating: build/bin/scalar_test 2024-12-18T00:07:48.7005635Z inflating: build/bin/stride_properties_test 2024-12-18T00:07:48.7091396Z inflating: build/bin/tensor_iterator_test 2024-12-18T00:07:48.7154033Z inflating: build/bin/test_parallel 2024-12-18T00:07:48.7215061Z inflating: build/bin/type_ptr_test 2024-12-18T00:07:48.7216784Z inflating: build/bin/thread_init_test 2024-12-18T00:07:48.7284181Z inflating: build/bin/type_test 2024-12-18T00:07:48.7343068Z inflating: build/bin/undefined_tensor_test 2024-12-18T00:07:48.7344559Z inflating: build/bin/verify_api_visibility 2024-12-18T00:07:48.7402803Z inflating: build/bin/wrapdim_test 2024-12-18T00:07:48.7481403Z inflating: build/bin/legacy_vmap_test 2024-12-18T00:07:48.7539554Z inflating: build/bin/weakref_test 2024-12-18T00:07:48.7597033Z inflating: build/bin/xla_tensor_test 2024-12-18T00:07:48.7661523Z inflating: build/bin/IListRef_test 2024-12-18T00:07:48.7776505Z inflating: build/bin/List_test 2024-12-18T00:07:48.7905693Z inflating: build/bin/kernel_function_legacy_test 2024-12-18T00:07:48.8011687Z inflating: build/bin/kernel_function_test 2024-12-18T00:07:48.8084795Z inflating: build/bin/KernelFunction_test 2024-12-18T00:07:48.8197148Z inflating: build/bin/kernel_lambda_test 2024-12-18T00:07:48.8331896Z inflating: build/bin/kernel_lambda_legacy_test 2024-12-18T00:07:48.8400902Z inflating: build/bin/kernel_stackbased_test 2024-12-18T00:07:48.8458151Z inflating: build/bin/CppSignature_test 2024-12-18T00:07:48.8562437Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-12-18T00:07:48.8616547Z inflating: build/bin/op_allowlist_test 2024-12-18T00:07:48.8927969Z inflating: build/bin/op_registration_test 2024-12-18T00:07:48.8989778Z inflating: build/bin/backend_fallback_test 2024-12-18T00:07:48.9060174Z inflating: build/bin/inline_container_test 2024-12-18T00:07:48.9119483Z inflating: build/bin/cuda_apply_test 2024-12-18T00:07:48.9177201Z inflating: build/bin/cuda_allocator_test 2024-12-18T00:07:48.9243101Z inflating: build/bin/cuda_atomic_ops_test 2024-12-18T00:07:48.9302292Z inflating: build/bin/cuda_caching_host_allocator_test 2024-12-18T00:07:48.9380253Z inflating: build/bin/cuda_complex_math_test 2024-12-18T00:07:48.9445366Z inflating: build/bin/cuda_complex_test 2024-12-18T00:07:48.9500211Z inflating: build/bin/cuda_device_test 2024-12-18T00:07:48.9566153Z inflating: build/bin/cuda_cub_test 2024-12-18T00:07:48.9621698Z inflating: build/bin/cuda_dlconvertor_test 2024-12-18T00:07:48.9679731Z inflating: build/bin/cuda_integer_divider_test 2024-12-18T00:07:48.9753339Z inflating: build/bin/cuda_distributions_test 2024-12-18T00:07:48.9815079Z inflating: build/bin/cuda_generator_test 2024-12-18T00:07:48.9873490Z inflating: build/bin/cuda_half_test 2024-12-18T00:07:48.9931134Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-12-18T00:07:48.9986244Z inflating: build/bin/cuda_optional_test 2024-12-18T00:07:49.0043036Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-12-18T00:07:49.0109004Z inflating: build/bin/cuda_stream_test 2024-12-18T00:07:49.0168773Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-12-18T00:07:49.0222507Z inflating: build/bin/cuda_cudnn_test 2024-12-18T00:07:49.0281110Z inflating: build/bin/cuda_vectorized_test 2024-12-18T00:07:49.0355468Z inflating: build/bin/ProcessGroupGlooTest 2024-12-18T00:07:49.0370608Z inflating: build/bin/tutorial_tensorexpr 2024-12-18T00:07:49.0433633Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-12-18T00:07:49.0502883Z inflating: build/bin/ProcessGroupNCCLTest 2024-12-18T00:07:49.0573231Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-12-18T00:07:49.0629965Z inflating: build/bin/ProcessGroupUCCTest 2024-12-18T00:07:49.0691246Z inflating: build/bin/test_dist_autograd 2024-12-18T00:07:49.0692403Z inflating: build/bin/parallel_benchmark 2024-12-18T00:07:49.0771020Z inflating: build/bin/test_cpp_rpc 2024-12-18T00:07:49.0844601Z inflating: build/bin/test_mobile_nnc 2024-12-18T00:07:49.0852681Z inflating: build/bin/aot_model_compiler_test 2024-12-18T00:07:49.1215206Z inflating: build/bin/test_lazy 2024-12-18T00:07:49.2079481Z inflating: build/bin/test_tensorexpr 2024-12-18T00:07:49.3298890Z inflating: build/bin/test_api 2024-12-18T00:07:49.3900470Z inflating: build/bin/test_jit 2024-12-18T00:07:49.3900922Z creating: .additional_ci_files/ 2024-12-18T00:07:49.3996240Z inflating: .additional_ci_files/test-times.json 2024-12-18T00:07:49.4366820Z inflating: .additional_ci_files/test-class-times.json 2024-12-18T00:07:49.4395363Z ##[group]Run rm artifacts.zip 2024-12-18T00:07:49.4395707Z rm artifacts.zip 2024-12-18T00:07:49.4401721Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:49.4402126Z env: 2024-12-18T00:07:49.4402364Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:49.4402721Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:49.4403109Z ##[endgroup] 2024-12-18T00:07:49.5039905Z ##[group]Run df -H 2024-12-18T00:07:49.5040328Z df -H 2024-12-18T00:07:49.5045859Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:49.5046260Z env: 2024-12-18T00:07:49.5046488Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:49.5046838Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:49.5047343Z ##[endgroup] 2024-12-18T00:07:49.5088804Z Filesystem Size Used Avail Use% Mounted on 2024-12-18T00:07:49.5089492Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-12-18T00:07:49.5089896Z tmpfs 101G 0 101G 0% /dev/shm 2024-12-18T00:07:49.5090434Z tmpfs 41G 697k 41G 1% /run 2024-12-18T00:07:49.5090815Z /dev/nvme0n1p1 161G 53G 109G 33% / 2024-12-18T00:07:49.5091330Z tmpfs 101G 13k 101G 1% /tmp 2024-12-18T00:07:49.5091731Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-12-18T00:07:49.5092143Z tmpfs 21G 0 21G 0% /run/user/0 2024-12-18T00:07:49.5141948Z Prepare all required actions 2024-12-18T00:07:49.5142836Z Getting action download info 2024-12-18T00:07:49.7078109Z ##[group]Run ./.github/actions/download-td-artifacts 2024-12-18T00:07:49.7078526Z with: 2024-12-18T00:07:49.7078800Z env: 2024-12-18T00:07:49.7079085Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:49.7079484Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:49.7080031Z ##[endgroup] 2024-12-18T00:07:49.7117999Z ##[group]Run seemethere/download-artifact-s3@v4 2024-12-18T00:07:49.7118388Z with: 2024-12-18T00:07:49.7118637Z name: td_results 2024-12-18T00:07:49.7118926Z s3-bucket: gha-artifacts 2024-12-18T00:07:49.7119221Z region: us-east-1 2024-12-18T00:07:49.7119485Z env: 2024-12-18T00:07:49.7119732Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:49.7120100Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:49.7120658Z ##[endgroup] 2024-12-18T00:07:50.1973743Z (node:57326) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-12-18T00:07:50.1974333Z 2024-12-18T00:07:50.1974569Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-12-18T00:07:50.1975193Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-12-18T00:07:50.1975826Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-12-18T00:07:50.2998946Z Found 1 objects with prefix pytorch/pytorch/12383255652/td_results/ 2024-12-18T00:07:50.2999855Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-12-18T00:07:50.3648214Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-12-18T00:07:50.3652913Z Artifact download has finished successfully 2024-12-18T00:07:50.3856124Z ##[group]Run mkdir -p .additional_ci_files 2024-12-18T00:07:50.3856521Z mkdir -p .additional_ci_files 2024-12-18T00:07:50.3857004Z mv td_results.json .additional_ci_files/td_results.json || true 2024-12-18T00:07:50.3863998Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:50.3864461Z env: 2024-12-18T00:07:50.3864718Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:50.3865117Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:50.3865540Z ##[endgroup] 2024-12-18T00:07:50.3956769Z ##[group]Run .github/scripts/parse_ref.py 2024-12-18T00:07:50.3957173Z .github/scripts/parse_ref.py 2024-12-18T00:07:50.3962926Z shell: /usr/bin/bash -e {0} 2024-12-18T00:07:50.3963237Z env: 2024-12-18T00:07:50.3963482Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:50.3963858Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:50.3964249Z ##[endgroup] 2024-12-18T00:07:50.4246873Z Prepare all required actions 2024-12-18T00:07:50.4286898Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-12-18T00:07:50.4287289Z with: 2024-12-18T00:07:50.4287703Z github-token: *** 2024-12-18T00:07:50.4287963Z env: 2024-12-18T00:07:50.4288206Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:50.4288713Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:50.4289121Z ##[endgroup] 2024-12-18T00:07:50.4310228Z ##[group]Run set -eux 2024-12-18T00:07:50.4310543Z set -eux 2024-12-18T00:07:50.4311055Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-12-18T00:07:50.4317174Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:50.4317576Z env: 2024-12-18T00:07:50.4317809Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:50.4318339Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:50.4318918Z GITHUB_TOKEN: *** 2024-12-18T00:07:50.4319183Z ##[endgroup] 2024-12-18T00:07:50.4342594Z + python3 .github/scripts/get_workflow_job_id.py 12383255652 i-0a62689d000b23974 2024-12-18T00:07:52.8078221Z setting job-id=34566462868 2024-12-18T00:07:52.8078956Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T00:07:52.8297686Z Prepare all required actions 2024-12-18T00:07:52.8298173Z Getting action download info 2024-12-18T00:07:52.9730840Z ##[group]Run ./.github/actions/filter-test-configs 2024-12-18T00:07:52.9731266Z with: 2024-12-18T00:07:52.9731721Z github-token: *** 2024-12-18T00:07:52.9733077Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}]} 2024-12-18T00:07:52.9734788Z job-name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T00:07:52.9735531Z env: 2024-12-18T00:07:52.9735785Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:52.9736178Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:52.9736594Z ##[endgroup] 2024-12-18T00:07:52.9816674Z ##[group]Run nick-fields/retry@v3.0.0 2024-12-18T00:07:52.9817064Z with: 2024-12-18T00:07:52.9817291Z shell: bash 2024-12-18T00:07:52.9817562Z timeout_minutes: 10 2024-12-18T00:07:52.9817848Z max_attempts: 5 2024-12-18T00:07:52.9818127Z retry_wait_seconds: 30 2024-12-18T00:07:52.9819359Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-12-18T00:07:52.9820338Z polling_interval_seconds: 1 2024-12-18T00:07:52.9820685Z warning_on_retry: true 2024-12-18T00:07:52.9821011Z continue_on_error: false 2024-12-18T00:07:52.9821331Z env: 2024-12-18T00:07:52.9821580Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:52.9822171Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:52.9822812Z GITHUB_TOKEN: *** 2024-12-18T00:07:52.9823089Z ##[endgroup] 2024-12-18T00:07:53.0864662Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-12-18T00:07:53.3455820Z Defaulting to user installation because normal site-packages is not writeable 2024-12-18T00:07:53.6050743Z Collecting requests==2.27.1 2024-12-18T00:07:53.6486032Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2024-12-18T00:07:53.9547246Z Collecting pyyaml==6.0.1 2024-12-18T00:07:53.9603569Z Downloading PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (738 kB) 2024-12-18T00:07:54.0611099Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-12-18T00:07:54.1755599Z Collecting certifi>=2017.4.17 2024-12-18T00:07:54.1803805Z Downloading certifi-2024.12.14-py3-none-any.whl (164 kB) 2024-12-18T00:07:54.2541217Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-12-18T00:07:54.6252529Z Collecting charset-normalizer~=2.0.0 2024-12-18T00:07:54.6318467Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-12-18T00:07:54.7804894Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2024-12-18T00:07:55.2781319Z Successfully installed certifi-2024.12.14 charset-normalizer-2.0.12 pyyaml-6.0.1 requests-2.27.1 2024-12-18T00:07:56.0691208Z Command completed after 1 attempt(s). 2024-12-18T00:07:56.0862165Z ##[group]Run set -x 2024-12-18T00:07:56.0862502Z set -x 2024-12-18T00:07:56.0862798Z  2024-12-18T00:07:56.0863255Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-12-18T00:07:56.0863845Z # in runner workspace 2024-12-18T00:07:56.0864322Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-12-18T00:07:56.0871245Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:56.0871791Z env: 2024-12-18T00:07:56.0872050Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:56.0872404Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:56.0872801Z ##[endgroup] 2024-12-18T00:07:56.0900600Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-12-18T00:07:56.1340594Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-12-18T00:07:56.1341078Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-12-18T00:07:56.1341490Z echo "Job name: ${JOB_NAME}" 2024-12-18T00:07:56.1341848Z  2024-12-18T00:07:56.1342308Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-12-18T00:07:56.1342886Z # in runner workspace 2024-12-18T00:07:56.1343390Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-12-18T00:07:56.1343967Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-12-18T00:07:56.1344386Z  --job-name "${JOB_NAME}" \ 2024-12-18T00:07:56.1345904Z  --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}]}" \ 2024-12-18T00:07:56.1347261Z  --selected-test-configs "" \ 2024-12-18T00:07:56.1347622Z  --pr-number "${PR_NUMBER}" \ 2024-12-18T00:07:56.1347962Z  --tag "${TAG}" \ 2024-12-18T00:07:56.1348268Z  --event-name "${EVENT_NAME}" \ 2024-12-18T00:07:56.1348626Z  --schedule "${SCHEDULE}" \ 2024-12-18T00:07:56.1348966Z  --branch "${HEAD_BRANCH}" 2024-12-18T00:07:56.1355142Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:56.1355578Z env: 2024-12-18T00:07:56.1355812Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:56.1356305Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:56.1356898Z GITHUB_TOKEN: *** 2024-12-18T00:07:56.1357434Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T00:07:56.1358029Z PR_NUMBER: 2024-12-18T00:07:56.1358270Z TAG: 2024-12-18T00:07:56.1358484Z EVENT_NAME: push 2024-12-18T00:07:56.1358743Z SCHEDULE: 2024-12-18T00:07:56.1359161Z HEAD_BRANCH: 2024-12-18T00:07:56.1359426Z ##[endgroup] 2024-12-18T00:07:56.1386918Z Workflow: pull 2024-12-18T00:07:56.1387563Z Job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T00:07:56.4122023Z ##[group]Run echo "Filtered matrix:" 2024-12-18T00:07:56.4122583Z echo "Filtered matrix:" 2024-12-18T00:07:56.4124021Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}]}" 2024-12-18T00:07:56.4125412Z  2024-12-18T00:07:56.4125680Z echo 2024-12-18T00:07:56.4126167Z echo "Is the current job unstable? False" 2024-12-18T00:07:56.4126563Z  2024-12-18T00:07:56.4126805Z echo 2024-12-18T00:07:56.4127094Z echo "Is keep-going label set? False" 2024-12-18T00:07:56.4127472Z  2024-12-18T00:07:56.4127714Z echo 2024-12-18T00:07:56.4127989Z echo "Renabled issues? " 2024-12-18T00:07:56.4133827Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:56.4134220Z env: 2024-12-18T00:07:56.4134458Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:56.4134811Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:56.4135191Z ##[endgroup] 2024-12-18T00:07:56.4159976Z Filtered matrix: 2024-12-18T00:07:56.4161659Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: linux.g4dn.12xlarge.nvidia.gpu}, {config: distributed, shard: 2, num_shards: 3, runner: linux.g4dn.12xlarge.nvidia.gpu}, {config: distributed, shard: 3, num_shards: 3, runner: linux.g4dn.12xlarge.nvidia.gpu}]} 2024-12-18T00:07:56.4162937Z 2024-12-18T00:07:56.4163081Z Is the current job unstable? False 2024-12-18T00:07:56.4163315Z 2024-12-18T00:07:56.4163454Z Is keep-going label set? False 2024-12-18T00:07:56.4163668Z 2024-12-18T00:07:56.4163775Z Renabled issues? 2024-12-18T00:07:56.4348150Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-12-18T00:07:56.4348690Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-12-18T00:07:56.4354097Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:07:56.4354505Z env: 2024-12-18T00:07:56.4354739Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:56.4355097Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:56.4355480Z JOB_TIMEOUT: 360 2024-12-18T00:07:56.4355715Z ##[endgroup] 2024-12-18T00:07:56.4498382Z ##[group]Run set -x 2024-12-18T00:07:56.4498861Z set -x 2024-12-18T00:07:56.4499296Z  2024-12-18T00:07:56.4499601Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-12-18T00:07:56.4500083Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-12-18T00:07:56.4500543Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-12-18T00:07:56.4500974Z  TEST_COMMAND=.ci/onnx/test.sh 2024-12-18T00:07:56.4501340Z else 2024-12-18T00:07:56.4501638Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-12-18T00:07:56.4502010Z fi 2024-12-18T00:07:56.4502244Z  2024-12-18T00:07:56.4502645Z # detached container should get cleaned up by teardown_ec2_linux 2024-12-18T00:07:56.4503283Z # TODO: Stop building test binaries as part of the build phase 2024-12-18T00:07:56.4503837Z # Used for GPU_FLAG since that doesn't play nice 2024-12-18T00:07:56.4504329Z # shellcheck disable=SC2086,SC2090 2024-12-18T00:07:56.4504736Z container_name=$(docker run \ 2024-12-18T00:07:56.4505108Z  ${GPU_FLAG:-} \ 2024-12-18T00:07:56.4505470Z  ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \ 2024-12-18T00:07:56.4505890Z  -e BUILD_ENVIRONMENT \ 2024-12-18T00:07:56.4506233Z  -e PR_NUMBER \ 2024-12-18T00:07:56.4506560Z  -e GITHUB_ACTIONS \ 2024-12-18T00:07:56.4506910Z  -e GITHUB_REPOSITORY \ 2024-12-18T00:07:56.4507271Z  -e GITHUB_WORKFLOW \ 2024-12-18T00:07:56.4507613Z  -e GITHUB_JOB \ 2024-12-18T00:07:56.4507926Z  -e GITHUB_RUN_ID \ 2024-12-18T00:07:56.4508258Z  -e GITHUB_RUN_NUMBER \ 2024-12-18T00:07:56.4508611Z  -e GITHUB_RUN_ATTEMPT \ 2024-12-18T00:07:56.4508955Z  -e JOB_ID \ 2024-12-18T00:07:56.4509260Z  -e JOB_NAME \ 2024-12-18T00:07:56.4509555Z  -e BASE_SHA \ 2024-12-18T00:07:56.4509861Z  -e BRANCH \ 2024-12-18T00:07:56.4510163Z  -e SHA1 \ 2024-12-18T00:07:56.4510463Z  -e AWS_DEFAULT_REGION \ 2024-12-18T00:07:56.4510817Z  -e IN_WHEEL_TEST \ 2024-12-18T00:07:56.4511240Z  -e SHARD_NUMBER \ 2024-12-18T00:07:56.4511708Z  -e TEST_CONFIG \ 2024-12-18T00:07:56.4512032Z  -e NUM_TEST_SHARDS \ 2024-12-18T00:07:56.4512377Z  -e REENABLED_ISSUES \ 2024-12-18T00:07:56.4512732Z  -e CONTINUE_THROUGH_ERROR \ 2024-12-18T00:07:56.4513098Z  -e VERBOSE_TEST_LOGS \ 2024-12-18T00:07:56.4513429Z  -e TEST_SHOWLOCALS \ 2024-12-18T00:07:56.4513764Z  -e NO_TEST_TIMEOUT \ 2024-12-18T00:07:56.4514087Z  -e NO_TD \ 2024-12-18T00:07:56.4514383Z  -e TD_DISTRIBUTED \ 2024-12-18T00:07:56.4514710Z  -e PR_LABELS \ 2024-12-18T00:07:56.4515038Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-12-18T00:07:56.4515423Z  -e SCCACHE_BUCKET \ 2024-12-18T00:07:56.4515762Z  -e SCCACHE_REGION \ 2024-12-18T00:07:56.4516095Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-12-18T00:07:56.4516446Z  -e XLA_CUDA \ 2024-12-18T00:07:56.4516894Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-12-18T00:07:56.4517293Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-12-18T00:07:56.4517711Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-12-18T00:07:56.4518132Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-12-18T00:07:56.4518526Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-12-18T00:07:56.4518902Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-12-18T00:07:56.4519259Z  -e DASHBOARD_TAG \ 2024-12-18T00:07:56.4519581Z  -e IS_A100_RUNNER \ 2024-12-18T00:07:56.4519914Z  -e ARTIFACTS_FILE_SUFFIX \ 2024-12-18T00:07:56.4520330Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-12-18T00:07:56.4520788Z  --security-opt seccomp=unconfined \ 2024-12-18T00:07:56.4521279Z  --cap-add=SYS_PTRACE \ 2024-12-18T00:07:56.4521602Z  --ipc=host \ 2024-12-18T00:07:56.4522245Z  --shm-size="${SHM_SIZE}" \ 2024-12-18T00:07:56.4522590Z  --tty \ 2024-12-18T00:07:56.4522950Z  --detach \ 2024-12-18T00:07:56.4523269Z  --name="${container_name}" \ 2024-12-18T00:07:56.4523621Z  --user jenkins \ 2024-12-18T00:07:56.4524031Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-12-18T00:07:56.4524492Z  -w /var/lib/jenkins/workspace \ 2024-12-18T00:07:56.4524866Z  "${DOCKER_IMAGE}" 2024-12-18T00:07:56.4525168Z ) 2024-12-18T00:07:56.4525492Z # Propagate download.pytorch.org IP to container 2024-12-18T00:07:56.4526263Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-12-18T00:07:56.4527088Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-12-18T00:07:56.4527900Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-12-18T00:07:56.4533771Z shell: /usr/bin/bash -e {0} 2024-12-18T00:07:56.4534088Z env: 2024-12-18T00:07:56.4534330Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:07:56.4534828Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:07:56.4535312Z BUILD_ENVIRONMENT: linux-focal-cuda11.8-py3.10-gcc9 2024-12-18T00:07:56.4535713Z PR_NUMBER: 2024-12-18T00:07:56.4535987Z GITHUB_REPOSITORY: pytorch/pytorch 2024-12-18T00:07:56.4536331Z GITHUB_WORKFLOW: pull 2024-12-18T00:07:56.4536597Z GITHUB_JOB: test 2024-12-18T00:07:56.4536862Z GITHUB_RUN_ID: 12383255652 2024-12-18T00:07:56.4537168Z GITHUB_RUN_NUMBER: 276594 2024-12-18T00:07:56.4537470Z GITHUB_RUN_ATTEMPT: 1 2024-12-18T00:07:56.4537743Z JOB_ID: 34566462868 2024-12-18T00:07:56.4538273Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T00:07:56.4539153Z BRANCH: release/2.6 2024-12-18T00:07:56.4539495Z SHA1: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:07:56.4539946Z BASE_SHA: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:07:56.4540359Z TEST_CONFIG: distributed 2024-12-18T00:07:56.4540788Z SHARD_NUMBER: 2 2024-12-18T00:07:56.4541072Z NUM_TEST_SHARDS: 3 2024-12-18T00:07:56.4541363Z REENABLED_ISSUES: 2024-12-18T00:07:56.4541663Z CONTINUE_THROUGH_ERROR: False 2024-12-18T00:07:56.4542000Z VERBOSE_TEST_LOGS: False 2024-12-18T00:07:56.4542306Z TEST_SHOWLOCALS: False 2024-12-18T00:07:56.4542618Z NO_TEST_TIMEOUT: False 2024-12-18T00:07:56.4542914Z NO_TD: False 2024-12-18T00:07:56.4543186Z TD_DISTRIBUTED: False 2024-12-18T00:07:56.4543551Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-12-18T00:07:56.4543964Z SCCACHE_REGION: us-east-1 2024-12-18T00:07:56.4544292Z SCCACHE_S3_KEY_PREFIX: pull 2024-12-18T00:07:56.4544615Z SHM_SIZE: 2g 2024-12-18T00:07:56.4545458Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:07:56.4546372Z XLA_CUDA: 2024-12-18T00:07:56.4546796Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-12-18T00:07:56.4547329Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-12-18T00:07:56.4547710Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-12-18T00:07:56.4548071Z DASHBOARD_TAG: 2024-12-18T00:07:56.4548563Z HUGGING_FACE_HUB_TOKEN: *** 2024-12-18T00:07:56.4549055Z SCRIBE_GRAPHQL_ACCESS_TOKEN: *** 2024-12-18T00:07:56.4549403Z IS_A100_RUNNER: 0 2024-12-18T00:07:56.4549913Z ARTIFACTS_FILE_SUFFIX: test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_34566462868 2024-12-18T00:07:56.4550518Z ##[endgroup] 2024-12-18T00:07:56.4573309Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-12-18T00:07:56.4573784Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *onnx* ]] 2024-12-18T00:07:56.4574198Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-12-18T00:07:56.4582633Z +++ nproc --ignore=2 2024-12-18T00:07:56.4989231Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=46 -e SCCACHE_BUCKET -e SCCACHE_REGION -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e IS_A100_RUNNER -e ARTIFACTS_FILE_SUFFIX --env-file=/tmp/github_env_12383255652 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:08:03.6604868Z + container_name=680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T00:08:03.6606765Z + grep download.pytorch.org /etc/hosts 2024-12-18T00:08:03.6608719Z + docker exec -i 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 sudo bash -c '/bin/cat >> /etc/hosts' 2024-12-18T00:08:03.7763915Z + echo DOCKER_CONTAINER_ID=680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T00:08:03.7768141Z ++ echo dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl 2024-12-18T00:08:03.7770571Z + docker exec -t 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 sh -c 'python3 -m pip install dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-12-18T00:08:04.1692341Z Processing ./dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl (from torch==2.6.0a0+git0cdf8b1) 2024-12-18T00:08:05.0548700Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.16.1) 2024-12-18T00:08:05.0551436Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (4.12.2) 2024-12-18T00:08:05.0554909Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (2.8.8) 2024-12-18T00:08:05.0558256Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.1.4) 2024-12-18T00:08:05.0561702Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (2024.10.0) 2024-12-18T00:08:05.0567448Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.13.1) 2024-12-18T00:08:05.0585955Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.3.0) 2024-12-18T00:08:05.0604743Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.3.0) 2024-12-18T00:08:05.0621414Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.22.4) 2024-12-18T00:08:05.1020999Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.0.2) 2024-12-18T00:08:05.4614391Z Installing collected packages: torch 2024-12-18T00:08:16.6430947Z Successfully installed torch-2.6.0a0+git0cdf8b1 2024-12-18T00:08:16.7171050Z + export TERM=vt100 2024-12-18T00:08:16.7171406Z + TERM=vt100 2024-12-18T00:08:16.7171702Z ++ dirname .ci/pytorch/test.sh 2024-12-18T00:08:16.7180132Z + source .ci/pytorch/common.sh 2024-12-18T00:08:16.7181593Z +++ dirname .ci/pytorch/common.sh 2024-12-18T00:08:16.7193870Z ++ source .ci/pytorch/common_utils.sh 2024-12-18T00:08:16.7194292Z +++ declare -f -t trap_add 2024-12-18T00:08:16.7201965Z ++ set -ex 2024-12-18T00:08:16.7202335Z ++ [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-12-18T00:08:16.7202758Z ++ BUILD_TEST_LIBTORCH=0 2024-12-18T00:08:16.7203119Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-12-18T00:08:16.7203544Z + [[ -d /var/lib/jenkins/workspace ]] 2024-12-18T00:08:16.7203937Z ++ stat -c %u /var/lib/jenkins/workspace 2024-12-18T00:08:16.7221054Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-12-18T00:08:16.7221467Z + trap_add cleanup_workspace EXIT 2024-12-18T00:08:16.7222034Z + trap_add_cmd=cleanup_workspace 2024-12-18T00:08:16.7222383Z + shift 2024-12-18T00:08:16.7222647Z + for trap_add_name in "$@" 2024-12-18T00:08:16.7226165Z +++ trap -p EXIT 2024-12-18T00:08:16.7228947Z ++ eval 'extract_trap_cmd ' 2024-12-18T00:08:16.7229261Z +++ extract_trap_cmd 2024-12-18T00:08:16.7229531Z +++ printf '%s\n' '' 2024-12-18T00:08:16.7229830Z ++ printf '%s\n' cleanup_workspace 2024-12-18T00:08:16.7231172Z + trap -- ' 2024-12-18T00:08:16.7231435Z cleanup_workspace' EXIT 2024-12-18T00:08:16.7231791Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-12-18T00:08:17.2151472Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-12-18T00:08:17.2173118Z + echo 'Environment variables:' 2024-12-18T00:08:17.2173479Z Environment variables: 2024-12-18T00:08:17.2173768Z + env 2024-12-18T00:08:17.2181392Z INSTALLED_DB=yes 2024-12-18T00:08:17.2181692Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-12-18T00:08:17.2182121Z NVIDIA_VISIBLE_DEVICES=all 2024-12-18T00:08:17.2182657Z NV_NVML_DEV_VERSION=11.8.86-1 2024-12-18T00:08:17.2183794Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-18T00:08:17.2184555Z CONTINUE_THROUGH_ERROR=False 2024-12-18T00:08:17.2184952Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-12-18T00:08:17.2185502Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-12-18T00:08:17.2185915Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-12-18T00:08:17.2186323Z HOSTNAME=680575e591d7 2024-12-18T00:08:17.2187034Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_d4d556fd-d5ac-4f41-b20f-bee1f4d209ca 2024-12-18T00:08:17.2187764Z GITHUB_ACTION=__self 2024-12-18T00:08:17.2188072Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-12-18T00:08:17.2189934Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-12-18T00:08:17.2191872Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-12-18T00:08:17.2192295Z NV_NVTX_VERSION=11.8.86-1 2024-12-18T00:08:17.2192598Z GITHUB_RUN_NUMBER=276594 2024-12-18T00:08:17.2192905Z TEST_CONFIG=distributed 2024-12-18T00:08:17.2193217Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-12-18T00:08:17.2193598Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-12-18T00:08:17.2193954Z IS_A100_RUNNER=0 2024-12-18T00:08:17.2194222Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-12-18T00:08:17.2194579Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-12-18T00:08:17.2195191Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2024-12-18T00:08:17.2195530Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-12-18T00:08:17.2196020Z GITHUB_TRIGGERING_ACTOR=malfet 2024-12-18T00:08:17.2196392Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-12-18T00:08:17.2196805Z GITHUB_REF_TYPE=branch 2024-12-18T00:08:17.2197106Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-12-18T00:08:17.2197430Z NCCL_VERSION=2.16.2-1 2024-12-18T00:08:17.2197765Z BASE_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:08:17.2198134Z XLA_CUDA= 2024-12-18T00:08:17.2198508Z HUGGING_FACE_HUB_TOKEN=*** 2024-12-18T00:08:17.2198900Z *** 2024-12-18T00:08:17.2199147Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-12-18T00:08:17.2199492Z GITHUB_REPOSITORY_ID=65600975 2024-12-18T00:08:17.2199810Z GITHUB_ACTIONS=true 2024-12-18T00:08:17.2200084Z NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:08:17.2200460Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-12-18T00:08:17.2200875Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-12-18T00:08:17.2201273Z SHA1=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:08:17.2201669Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-12-18T00:08:17.2202071Z GITHUB_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:08:17.2202688Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/heads/release/2.6 2024-12-18T00:08:17.2203260Z UCC_HOME=/usr 2024-12-18T00:08:17.2203540Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-12-18T00:08:17.2203890Z VERBOSE_TEST_LOGS=False 2024-12-18T00:08:17.2204177Z NVIDIA_PRODUCT_NAME=CUDA 2024-12-18T00:08:17.2204528Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-12-18T00:08:17.2204933Z GITHUB_REF=refs/heads/release/2.6 2024-12-18T00:08:17.2205276Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-12-18T00:08:17.2205586Z SHARD_NUMBER=2 2024-12-18T00:08:17.2216555Z GITHUB_REF_PROTECTED=true 2024-12-18T00:08:17.2216897Z HOME=/var/lib/jenkins 2024-12-18T00:08:17.2217232Z GITHUB_API_URL=https://api.github.com 2024-12-18T00:08:17.2217624Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-12-18T00:08:17.2218020Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-12-18T00:08:17.2218526Z SCCACHE_S3_KEY_PREFIX=pull 2024-12-18T00:08:17.2219040Z CUDA_VERSION=11.8.0 2024-12-18T00:08:17.2219377Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-12-18T00:08:17.2219776Z NUM_TEST_SHARDS=3 2024-12-18T00:08:17.2220045Z UCX_HOME=/usr 2024-12-18T00:08:17.2220611Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-12-18T00:08:17.2221526Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_d4d556fd-d5ac-4f41-b20f-bee1f4d209ca 2024-12-18T00:08:17.2222795Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T00:08:17.2223816Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_d4d556fd-d5ac-4f41-b20f-bee1f4d209ca 2024-12-18T00:08:17.2224764Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-12-18T00:08:17.2225364Z GITHUB_EVENT_NAME=push 2024-12-18T00:08:17.2225671Z DASHBOARD_TAG= 2024-12-18T00:08:17.2225947Z GITHUB_RUN_ID=12383255652 2024-12-18T00:08:17.2226323Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-12-18T00:08:17.2226754Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-12-18T00:08:17.2227578Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_d4d556fd-d5ac-4f41-b20f-bee1f4d209ca 2024-12-18T00:08:17.2228389Z GITHUB_ACTOR=malfet 2024-12-18T00:08:17.2228688Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-12-18T00:08:17.2229023Z PR_NUMBER= 2024-12-18T00:08:17.2229277Z GITHUB_RUN_ATTEMPT=1 2024-12-18T00:08:17.2229568Z ANACONDA_PYTHON_VERSION=3.10 2024-12-18T00:08:17.2229969Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-12-18T00:08:17.2230380Z TERM=vt100 2024-12-18T00:08:17.2230658Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-12-18T00:08:17.2231120Z INSTALLED_VISION=yes 2024-12-18T00:08:17.2231402Z BRANCH=release/2.6 2024-12-18T00:08:17.2231681Z SCCACHE_REGION=us-east-1 2024-12-18T00:08:17.2231995Z OPENSSL_ROOT_DIR=/opt/openssl 2024-12-18T00:08:17.2232343Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-12-18T00:08:17.2232843Z CUDA_PATH=/usr/local/cuda 2024-12-18T00:08:17.2233451Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-12-18T00:08:17.2234225Z GITHUB_SERVER_URL=https://github.com 2024-12-18T00:08:17.2234630Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-12-18T00:08:17.2235025Z REENABLED_ISSUES= 2024-12-18T00:08:17.2235269Z SHLVL=1 2024-12-18T00:08:17.2235500Z MAX_JOBS=46 2024-12-18T00:08:17.2235753Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-12-18T00:08:17.2236058Z NVARCH=x86_64 2024-12-18T00:08:17.2236313Z GITHUB_ACTOR_ID=2453524 2024-12-18T00:08:17.2236676Z GITHUB_WORKFLOW_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:08:17.2237111Z GITHUB_REF_NAME=release/2.6 2024-12-18T00:08:17.2237441Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-12-18T00:08:17.2238118Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-12-18T00:08:17.2238612Z GITHUB_JOB=test 2024-12-18T00:08:17.2238905Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-12-18T00:08:17.2239377Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-12-18T00:08:17.2239825Z NO_TEST_TIMEOUT=False 2024-12-18T00:08:17.2240117Z TD_DISTRIBUTED=False 2024-12-18T00:08:17.2240431Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-12-18T00:08:17.2240796Z GITHUB_REPOSITORY=pytorch/pytorch 2024-12-18T00:08:17.2241141Z NV_NVPROF_VERSION=11.8.87-1 2024-12-18T00:08:17.2241456Z GITHUB_RETENTION_DAYS=90 2024-12-18T00:08:17.2241760Z OPENSSL_DIR=/opt/openssl 2024-12-18T00:08:17.2242069Z GITHUB_ACTION_REPOSITORY= 2024-12-18T00:08:17.2242953Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T00:08:17.2243872Z GITHUB_BASE_REF= 2024-12-18T00:08:17.2244386Z ARTIFACTS_FILE_SUFFIX=test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_34566462868 2024-12-18T00:08:17.2244991Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-12-18T00:08:17.2245315Z CI=true 2024-12-18T00:08:17.2245566Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-12-18T00:08:17.2245925Z GITHUB_REPOSITORY_OWNER=pytorch 2024-12-18T00:08:17.2246249Z JOB_ID=34566462868 2024-12-18T00:08:17.2246524Z INSTALLED_PROTOBUF=yes 2024-12-18T00:08:17.2246916Z GITHUB_HEAD_REF= 2024-12-18T00:08:17.2247169Z GITHUB_ACTION_REF= 2024-12-18T00:08:17.2247504Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-12-18T00:08:17.2247912Z TEST_SHOWLOCALS=False 2024-12-18T00:08:17.2248202Z GITHUB_WORKFLOW=pull 2024-12-18T00:08:17.2248506Z DEBIAN_FRONTEND=noninteractive 2024-12-18T00:08:17.2249218Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_d4d556fd-d5ac-4f41-b20f-bee1f4d209ca 2024-12-18T00:08:17.2249952Z NO_TD=False 2024-12-18T00:08:17.2250213Z SKIP_SCCACHE_INITIALIZATION=1 2024-12-18T00:08:17.2250527Z _=/usr/bin/env 2024-12-18T00:08:17.2250904Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-12-18T00:08:17.2426417Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-12-18T00:08:17.2427798Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T00:08:17.2429051Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-12-18T00:08:17.2430264Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-12-18T00:08:17.2431307Z + BUILD_DIR=build 2024-12-18T00:08:17.2431596Z + BUILD_RENAMED_DIR=build_renamed 2024-12-18T00:08:17.2431943Z + BUILD_BIN_DIR=build/bin 2024-12-18T00:08:17.2432240Z + SHARD_NUMBER=2 2024-12-18T00:08:17.2432505Z + NUM_TEST_SHARDS=3 2024-12-18T00:08:17.2432766Z + export VALGRIND=ON 2024-12-18T00:08:17.2433045Z + VALGRIND=ON 2024-12-18T00:08:17.2433360Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *clang9* ]] 2024-12-18T00:08:17.2433814Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *xpu* ]] 2024-12-18T00:08:17.2434205Z + [[ 0 == \1 ]] 2024-12-18T00:08:17.2434445Z + [[ False == \1 ]] 2024-12-18T00:08:17.2434943Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *bazel* ]] 2024-12-18T00:08:17.2435371Z ++ realpath build/custom_test_artifacts 2024-12-18T00:08:17.2442875Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-12-18T00:08:17.2443879Z + [[ -n '' ]] 2024-12-18T00:08:17.2444361Z + echo 'Environment variables' 2024-12-18T00:08:17.2444736Z Environment variables 2024-12-18T00:08:17.2445011Z + env 2024-12-18T00:08:17.2451799Z INSTALLED_DB=yes 2024-12-18T00:08:17.2452093Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-12-18T00:08:17.2452457Z NVIDIA_VISIBLE_DEVICES=all 2024-12-18T00:08:17.2452940Z NV_NVML_DEV_VERSION=11.8.86-1 2024-12-18T00:08:17.2453396Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-18T00:08:17.2453900Z CONTINUE_THROUGH_ERROR=False 2024-12-18T00:08:17.2454287Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-12-18T00:08:17.2454715Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-12-18T00:08:17.2455357Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-12-18T00:08:17.2456024Z HOSTNAME=680575e591d7 2024-12-18T00:08:17.2457206Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_d4d556fd-d5ac-4f41-b20f-bee1f4d209ca 2024-12-18T00:08:17.2458634Z GITHUB_ACTION=__self 2024-12-18T00:08:17.2459377Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-12-18T00:08:17.2462983Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-12-18T00:08:17.2466828Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-12-18T00:08:17.2467557Z NV_NVTX_VERSION=11.8.86-1 2024-12-18T00:08:17.2468096Z GITHUB_RUN_NUMBER=276594 2024-12-18T00:08:17.2468650Z TEST_CONFIG=distributed 2024-12-18T00:08:17.2468999Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-12-18T00:08:17.2469389Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-12-18T00:08:17.2469742Z IS_A100_RUNNER=0 2024-12-18T00:08:17.2470034Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-12-18T00:08:17.2470682Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-12-18T00:08:17.2471266Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2024-12-18T00:08:17.2471606Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-12-18T00:08:17.2471939Z GITHUB_TRIGGERING_ACTOR=malfet 2024-12-18T00:08:17.2472307Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-12-18T00:08:17.2472713Z GITHUB_REF_TYPE=branch 2024-12-18T00:08:17.2473011Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-12-18T00:08:17.2473324Z NCCL_VERSION=2.16.2-1 2024-12-18T00:08:17.2473641Z BASE_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:08:17.2474015Z XLA_CUDA= 2024-12-18T00:08:17.2474381Z HUGGING_FACE_HUB_TOKEN=*** 2024-12-18T00:08:17.2474802Z *** 2024-12-18T00:08:17.2475054Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-12-18T00:08:17.2475400Z GITHUB_REPOSITORY_ID=65600975 2024-12-18T00:08:17.2475710Z GITHUB_ACTIONS=true 2024-12-18T00:08:17.2476000Z NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:08:17.2476375Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-12-18T00:08:17.2476792Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-12-18T00:08:17.2477194Z SHA1=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:08:17.2477577Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-12-18T00:08:17.2477985Z GITHUB_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:08:17.2478599Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/heads/release/2.6 2024-12-18T00:08:17.2479164Z UCC_HOME=/usr 2024-12-18T00:08:17.2479437Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-12-18T00:08:17.2479772Z VERBOSE_TEST_LOGS=False 2024-12-18T00:08:17.2480065Z NVIDIA_PRODUCT_NAME=CUDA 2024-12-18T00:08:17.2480415Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-12-18T00:08:17.2480822Z GITHUB_REF=refs/heads/release/2.6 2024-12-18T00:08:17.2481165Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-12-18T00:08:17.2481576Z SHARD_NUMBER=2 2024-12-18T00:08:17.2481851Z GITHUB_REF_PROTECTED=true 2024-12-18T00:08:17.2482159Z HOME=/var/lib/jenkins 2024-12-18T00:08:17.2482483Z GITHUB_API_URL=https://api.github.com 2024-12-18T00:08:17.2482871Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-12-18T00:08:17.2483256Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-12-18T00:08:17.2483661Z SCCACHE_S3_KEY_PREFIX=pull 2024-12-18T00:08:17.2483967Z CUDA_VERSION=11.8.0 2024-12-18T00:08:17.2484283Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-12-18T00:08:17.2484664Z NUM_TEST_SHARDS=3 2024-12-18T00:08:17.2484956Z UCX_HOME=/usr 2024-12-18T00:08:17.2485335Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-12-18T00:08:17.2486208Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_d4d556fd-d5ac-4f41-b20f-bee1f4d209ca 2024-12-18T00:08:17.2487229Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T00:08:17.2488222Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_d4d556fd-d5ac-4f41-b20f-bee1f4d209ca 2024-12-18T00:08:17.2489154Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-12-18T00:08:17.2489735Z GITHUB_EVENT_NAME=push 2024-12-18T00:08:17.2490011Z DASHBOARD_TAG= 2024-12-18T00:08:17.2490276Z GITHUB_RUN_ID=12383255652 2024-12-18T00:08:17.2490624Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-12-18T00:08:17.2491039Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-12-18T00:08:17.2491835Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_d4d556fd-d5ac-4f41-b20f-bee1f4d209ca 2024-12-18T00:08:17.2492611Z GITHUB_ACTOR=malfet 2024-12-18T00:08:17.2492883Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-12-18T00:08:17.2493202Z PR_NUMBER= 2024-12-18T00:08:17.2493451Z GITHUB_RUN_ATTEMPT=1 2024-12-18T00:08:17.2493728Z VALGRIND=ON 2024-12-18T00:08:17.2493999Z ANACONDA_PYTHON_VERSION=3.10 2024-12-18T00:08:17.2494389Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-12-18T00:08:17.2494779Z TERM=vt100 2024-12-18T00:08:17.2495152Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-12-18T00:08:17.2495619Z INSTALLED_VISION=yes 2024-12-18T00:08:17.2495989Z BRANCH=release/2.6 2024-12-18T00:08:17.2496271Z SCCACHE_REGION=us-east-1 2024-12-18T00:08:17.2496573Z OPENSSL_ROOT_DIR=/opt/openssl 2024-12-18T00:08:17.2496923Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-12-18T00:08:17.2497296Z CUDA_PATH=/usr/local/cuda 2024-12-18T00:08:17.2497895Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-12-18T00:08:17.2498665Z GITHUB_SERVER_URL=https://github.com 2024-12-18T00:08:17.2499254Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-12-18T00:08:17.2499671Z REENABLED_ISSUES= 2024-12-18T00:08:17.2499946Z SHLVL=1 2024-12-18T00:08:17.2500187Z MAX_JOBS=46 2024-12-18T00:08:17.2500451Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-12-18T00:08:17.2500866Z NVARCH=x86_64 2024-12-18T00:08:17.2501220Z GITHUB_ACTOR_ID=2453524 2024-12-18T00:08:17.2501624Z GITHUB_WORKFLOW_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:08:17.2502088Z GITHUB_REF_NAME=release/2.6 2024-12-18T00:08:17.2502437Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-12-18T00:08:17.2502993Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-12-18T00:08:17.2503496Z GITHUB_JOB=test 2024-12-18T00:08:17.2503806Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-12-18T00:08:17.2504292Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-12-18T00:08:17.2504747Z NO_TEST_TIMEOUT=False 2024-12-18T00:08:17.2505028Z TD_DISTRIBUTED=False 2024-12-18T00:08:17.2505345Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-12-18T00:08:17.2505732Z GITHUB_REPOSITORY=pytorch/pytorch 2024-12-18T00:08:17.2506091Z NV_NVPROF_VERSION=11.8.87-1 2024-12-18T00:08:17.2506410Z GITHUB_RETENTION_DAYS=90 2024-12-18T00:08:17.2506710Z OPENSSL_DIR=/opt/openssl 2024-12-18T00:08:17.2507027Z GITHUB_ACTION_REPOSITORY= 2024-12-18T00:08:17.2508032Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T00:08:17.2508988Z GITHUB_BASE_REF= 2024-12-18T00:08:17.2509511Z ARTIFACTS_FILE_SUFFIX=test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_34566462868 2024-12-18T00:08:17.2510119Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-12-18T00:08:17.2510559Z CI=true 2024-12-18T00:08:17.2510816Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-12-18T00:08:17.2511173Z GITHUB_REPOSITORY_OWNER=pytorch 2024-12-18T00:08:17.2511491Z JOB_ID=34566462868 2024-12-18T00:08:17.2511751Z INSTALLED_PROTOBUF=yes 2024-12-18T00:08:17.2512036Z GITHUB_HEAD_REF= 2024-12-18T00:08:17.2512301Z GITHUB_ACTION_REF= 2024-12-18T00:08:17.2512632Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-12-18T00:08:17.2513034Z TEST_SHOWLOCALS=False 2024-12-18T00:08:17.2513310Z GITHUB_WORKFLOW=pull 2024-12-18T00:08:17.2513610Z DEBIAN_FRONTEND=noninteractive 2024-12-18T00:08:17.2514337Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_d4d556fd-d5ac-4f41-b20f-bee1f4d209ca 2024-12-18T00:08:17.2515071Z NO_TD=False 2024-12-18T00:08:17.2515334Z SKIP_SCCACHE_INITIALIZATION=1 2024-12-18T00:08:17.2515639Z _=/usr/bin/env 2024-12-18T00:08:17.2515899Z + echo 'Testing pytorch' 2024-12-18T00:08:17.2516197Z Testing pytorch 2024-12-18T00:08:17.2516545Z + export LANG=C.UTF-8 2024-12-18T00:08:17.2516812Z + LANG=C.UTF-8 2024-12-18T00:08:17.2517067Z + PR_NUMBER= 2024-12-18T00:08:17.2517339Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-12-18T00:08:17.2517715Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-12-18T00:08:17.2518136Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-12-18T00:08:17.2518522Z + [[ distributed == \s\l\o\w ]] 2024-12-18T00:08:17.2518928Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *slow-gradcheck* ]] 2024-12-18T00:08:17.2519446Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-12-18T00:08:17.2519881Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-12-18T00:08:17.2520280Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-12-18T00:08:17.2520628Z + [[ distributed == *crossref* ]] 2024-12-18T00:08:17.2521006Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-12-18T00:08:17.2521514Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *xpu* ]] 2024-12-18T00:08:17.2522165Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-12-18T00:08:17.2522764Z + pip_install --user ninja==1.10.2 2024-12-18T00:08:17.2523218Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2024-12-18T00:08:17.2523779Z + python3 -m pip install --progress-bar off --user ninja==1.10.2 2024-12-18T00:08:18.9475299Z Collecting ninja==1.10.2 2024-12-18T00:08:18.9693824Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-12-18T00:08:18.9806947Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-12-18T00:08:19.3336513Z Installing collected packages: ninja 2024-12-18T00:08:19.3414643Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-12-18T00:08:19.3415765Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-12-18T00:08:19.3460745Z Successfully installed ninja-1.10.2 2024-12-18T00:08:19.4127859Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T00:08:19.4129695Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T00:08:19.4130828Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-12-18T00:08:19.4131245Z + install_tlparse 2024-12-18T00:08:19.4131548Z + pip_install --user tlparse==0.3.25 2024-12-18T00:08:19.4132262Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2024-12-18T00:08:19.4132825Z + python3 -m pip install --progress-bar off --user tlparse==0.3.25 2024-12-18T00:08:19.8044766Z Collecting tlparse==0.3.25 2024-12-18T00:08:19.8233059Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-12-18T00:08:19.8349878Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-12-18T00:08:20.2249818Z Installing collected packages: tlparse 2024-12-18T00:08:20.2621736Z Successfully installed tlparse-0.3.25 2024-12-18T00:08:20.3289232Z ++ python -m site --user-base 2024-12-18T00:08:20.3540667Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T00:08:20.3541984Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *asan* ]] 2024-12-18T00:08:20.3542504Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-debug* ]] 2024-12-18T00:08:20.3542985Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-12-18T00:08:20.3543656Z + echo 'We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass' 2024-12-18T00:08:20.3544482Z We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass 2024-12-18T00:08:20.3545046Z + cd test 2024-12-18T00:08:20.3545442Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-12-18T00:08:22.0844746Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-12-18T00:08:22.0845582Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-12-18T00:08:22.0848262Z + DYNAMO_BENCHMARK_FLAGS=() 2024-12-18T00:08:22.0849621Z + [[ distributed == *pr_time_benchmarks* ]] 2024-12-18T00:08:22.0850323Z + [[ distributed == *dynamo_eager* ]] 2024-12-18T00:08:22.0851009Z + [[ distributed == *aot_eager* ]] 2024-12-18T00:08:22.0851663Z + [[ distributed == *aot_inductor* ]] 2024-12-18T00:08:22.0852359Z + [[ distributed == *inductor* ]] 2024-12-18T00:08:22.0852992Z + [[ distributed == *dynamic* ]] 2024-12-18T00:08:22.0853607Z + [[ distributed == *cpu* ]] 2024-12-18T00:08:22.0854237Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-12-18T00:08:22.0891536Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-12-18T00:08:22.0892055Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-bazel-* ]] 2024-12-18T00:08:22.0892796Z + cd test 2024-12-18T00:08:22.0893725Z + python -c 'import torch; print(torch.__config__.show())' 2024-12-18T00:08:24.0893768Z PyTorch built with: 2024-12-18T00:08:24.0894150Z - GCC 9.4 2024-12-18T00:08:24.0894409Z - C++ Version: 201703 2024-12-18T00:08:24.0895059Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-12-18T00:08:24.0895900Z - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-12-18T00:08:24.0896419Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-12-18T00:08:24.0896854Z - LAPACK is enabled (usually provided by MKL) 2024-12-18T00:08:24.0897242Z - NNPACK is enabled 2024-12-18T00:08:24.0897534Z - CPU capability usage: AVX512 2024-12-18T00:08:24.0897871Z - CUDA Runtime 11.8 2024-12-18T00:08:24.0898290Z - NVCC architecture flags: -gencode;arch=compute_75,code=sm_75 2024-12-18T00:08:24.0898863Z - CuDNN 90.1 2024-12-18T00:08:24.0899300Z - Magma 2.6.1 2024-12-18T00:08:24.0905667Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7, CUDA_VERSION=11.8, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.6.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-12-18T00:08:24.0912129Z 2024-12-18T00:08:24.5315720Z + cd test 2024-12-18T00:08:24.5316178Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-12-18T00:08:26.0055623Z ATen/Parallel: 2024-12-18T00:08:26.0056022Z at::get_num_threads() : 24 2024-12-18T00:08:26.0056381Z at::get_num_interop_threads() : 24 2024-12-18T00:08:26.0056774Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-12-18T00:08:26.0057124Z omp_get_max_threads() : 24 2024-12-18T00:08:26.0057797Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-12-18T00:08:26.0058612Z mkl_get_max_threads() : 24 2024-12-18T00:08:26.0059247Z Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-12-18T00:08:26.0059767Z std::thread::hardware_concurrency() : 48 2024-12-18T00:08:26.0060143Z Environment variables: 2024-12-18T00:08:26.0060454Z OMP_NUM_THREADS : [not set] 2024-12-18T00:08:26.0060779Z MKL_NUM_THREADS : [not set] 2024-12-18T00:08:26.0061112Z ATen parallel backend: OpenMP 2024-12-18T00:08:26.0061330Z 2024-12-18T00:08:26.2483231Z + [[ distributed == *numpy_2* ]] 2024-12-18T00:08:26.2483741Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-12-18T00:08:26.2484185Z + [[ distributed == *backward* ]] 2024-12-18T00:08:26.2484522Z + [[ distributed == *xla* ]] 2024-12-18T00:08:26.2484896Z + [[ distributed == *executorch* ]] 2024-12-18T00:08:26.2485271Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-12-18T00:08:26.2485709Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-12-18T00:08:26.2486554Z + [[ distributed == distributed ]] 2024-12-18T00:08:26.2486880Z + test_distributed 2024-12-18T00:08:26.2487185Z + echo 'Testing distributed python tests' 2024-12-18T00:08:26.2487570Z Testing distributed python tests 2024-12-18T00:08:26.2488043Z + python test/run_test.py --distributed-tests --shard 2 3 --verbose 2024-12-18T00:08:26.3543085Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T00:08:26.3544122Z import pkg_resources 2024-12-18T00:08:31.4465920Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T00:08:31.4981451Z Ignoring disabled issues: [''] 2024-12-18T00:08:31.5150915Z Found test times from artifacts 2024-12-18T00:08:31.5798962Z Found test times from artifacts 2024-12-18T00:08:31.5819821Z Running all tests 2024-12-18T00:08:31.5932234Z Running parallel tests on 2 processes 2024-12-18T00:08:31.5933574Z Name: tests to run (est. time: 115.46min) 2024-12-18T00:08:31.5933956Z Serial tests (69): 2024-12-18T00:08:31.5934326Z distributed/_tools/test_runtime_estimator 1/1 2024-12-18T00:08:31.5934747Z distributed/test_device_mesh 1/1 2024-12-18T00:08:31.5935156Z distributed/_tensor/test_experimental_ops 1/1 2024-12-18T00:08:31.5935585Z distributed/_tensor/test_tensor_ops 1/1 2024-12-18T00:08:31.5936074Z distributed/_composable/fsdp/test_fully_shard_extensions 1/1 2024-12-18T00:08:31.5936546Z distributed/fsdp/test_wrap 1/1 2024-12-18T00:08:31.5936995Z distributed/_composable/fsdp/test_fully_shard_autograd 1/1 2024-12-18T00:08:31.5937761Z distributed/fsdp/test_fsdp_clip_grad_norm 1/1 2024-12-18T00:08:31.5938180Z distributed/fsdp/test_fsdp_misc 1/1 2024-12-18T00:08:31.5938662Z distributed/test_c10d_common 1/1 2024-12-18T00:08:31.5939207Z distributed/_tools/test_mod_tracker 1/1 2024-12-18T00:08:31.5939672Z distributed/pipelining/test_schedule_multiproc 1/1 2024-12-18T00:08:31.5940194Z distributed/tensor/parallel/test_micro_pipeline_tp 1/1 2024-12-18T00:08:31.5940681Z distributed/checkpoint/test_nested_dict 1/1 2024-12-18T00:08:31.5941118Z distributed/pipelining/test_transformer 1/1 2024-12-18T00:08:31.5941531Z distributed/pipelining/test_backward 1/1 2024-12-18T00:08:31.5941975Z distributed/_shard/sharded_tensor/test_logger 1/1 2024-12-18T00:08:31.5942418Z distributed/pipelining/test_pipe 1/1 2024-12-18T00:08:31.5942825Z distributed/_tools/test_sac_estimator 1/1 2024-12-18T00:08:31.5943233Z distributed/_tools/test_sac_ilp 1/1 2024-12-18T00:08:31.5943623Z distributed/test_c10d_functional_native 1/1 2024-12-18T00:08:31.5944070Z distributed/checkpoint/test_save_load_api 1/1 2024-12-18T00:08:31.5944521Z distributed/checkpoint/test_dedup_tensors 1/1 2024-12-18T00:08:31.5945003Z distributed/_composable/test_contract 1/1 2024-12-18T00:08:31.5945475Z distributed/pipelining/test_microbatch 1/1 2024-12-18T00:08:31.5945904Z distributed/optim/test_named_optimizer 1/1 2024-12-18T00:08:31.5946332Z distributed/checkpoint/e2e/test_fsdp_ep 1/1 2024-12-18T00:08:31.5946836Z distributed/_tensor/experimental/test_register_sharding 1/1 2024-12-18T00:08:31.5947319Z distributed/_tensor/test_attention 1/1 2024-12-18T00:08:31.5947707Z distributed/test_fake_pg 1/1 2024-12-18T00:08:31.5948110Z distributed/checkpoint/test_fsdp_optim_state 1/1 2024-12-18T00:08:31.5948570Z distributed/_composable/test_checkpoint 1/1 2024-12-18T00:08:31.5949035Z distributed/checkpoint/test_dtensor_resharding 1/1 2024-12-18T00:08:31.5949475Z distributed/test_distributed_spawn 2/9 2024-12-18T00:08:31.5949885Z distributed/test_distributed_spawn 5/9 2024-12-18T00:08:31.5950397Z distributed/test_distributed_spawn 8/9 2024-12-18T00:08:31.5950786Z distributed/checkpoint/test_fsspec 1/1 2024-12-18T00:08:31.5951197Z distributed/tensor/parallel/test_tp_style 1/1 2024-12-18T00:08:31.5951805Z distributed/checkpoint/e2e/test_e2e_save_and_load 1/1 2024-12-18T00:08:31.5952277Z distributed/tensor/parallel/test_tp_examples 1/1 2024-12-18T00:08:31.5952769Z distributed/_composable/fsdp/test_fully_shard_state 1/1 2024-12-18T00:08:31.5953230Z distributed/test_control_collectives 1/1 2024-12-18T00:08:31.5953645Z distributed/_tensor/debug/test_comm_mode 1/1 2024-12-18T00:08:31.5954070Z distributed/_composable/test_replicate 1/1 2024-12-18T00:08:31.5954522Z distributed/optim/test_apply_optimizer_in_backward 1/1 2024-12-18T00:08:31.5954991Z distributed/fsdp/test_fsdp_state_dict 2/2 2024-12-18T00:08:31.5955378Z distributed/test_c10d_nccl 3/3 2024-12-18T00:08:31.5955802Z distributed/_composable/fsdp/test_fully_shard_comm 1/1 2024-12-18T00:08:31.5956263Z distributed/nn/jit/test_instantiator 1/1 2024-12-18T00:08:31.5956685Z distributed/fsdp/test_fsdp_backward_prefetch 1/1 2024-12-18T00:08:31.5957123Z distributed/fsdp/test_fsdp_pure_fp16 1/1 2024-12-18T00:08:31.5957534Z distributed/test_c10d_object_collectives 1/1 2024-12-18T00:08:31.5957935Z distributed/test_c10d_pypg 1/1 2024-12-18T00:08:31.5958333Z distributed/fsdp/test_fsdp_freezing_weights 1/1 2024-12-18T00:08:31.5958821Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1 2024-12-18T00:08:31.5959294Z distributed/fsdp/test_fsdp_exec_order 1/1 2024-12-18T00:08:31.5959697Z distributed/fsdp/test_fsdp_grad_acc 1/1 2024-12-18T00:08:31.5960113Z distributed/fsdp/test_fsdp_ignored_modules 1/1 2024-12-18T00:08:31.5960575Z distributed/fsdp/test_fsdp_multiple_wrapping 1/1 2024-12-18T00:08:31.5961104Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 2024-12-18T00:08:31.5961812Z distributed/_composable/test_composability/test_2d_composability 1/1 2024-12-18T00:08:31.5962356Z distributed/fsdp/test_fsdp_comm_hooks 1/1 2024-12-18T00:08:31.5962756Z distributed/fsdp/test_fsdp_core 2/2 2024-12-18T00:08:31.5963141Z distributed/_shard/test_sharder 1/1 2024-12-18T00:08:31.5963516Z distributed/fsdp/test_fsdp_comm 1/1 2024-12-18T00:08:31.5963920Z distributed/fsdp/test_distributed_checkpoint 1/1 2024-12-18T00:08:31.5964378Z distributed/fsdp/test_fsdp_use_orig_params 1/1 2024-12-18T00:08:31.5964784Z distributed/test_c10d_gloo 1/1 2024-12-18T00:08:31.5965149Z distributed/elastic/timer/api_test 1/1 2024-12-18T00:08:31.5965517Z Parallel tests (0): 2024-12-18T00:08:31.5965805Z Name: excluded (est. time: 0.0min) 2024-12-18T00:08:31.5966139Z Serial tests (0): 2024-12-18T00:08:31.5966417Z Parallel tests (0): 2024-12-18T00:08:31.5999538Z Running distributed/_tools/test_runtime_estimator 1/1 ... [2024-12-18 00:08:31.599328] 2024-12-18T00:08:31.6000189Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:08:31.6001483Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_runtime_estimator.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:08:31.599660] 2024-12-18T00:08:47.5020427Z 2024-12-18T00:08:47.5022083Z distributed/_tools/test_runtime_estimator 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_runtime_estimator_1.1_85fe516bc8b708a2_.log 2024-12-18T00:08:47.5024286Z Running 2 items in this shard: test/distributed/_tools/test_runtime_estimator.py::TestRuntimeEstimator::test_conv_model_runtime, test/distributed/_tools/test_runtime_estimator.py::TestRuntimeEstimator::test_transformer_runtime 2024-12-18T00:08:47.5025474Z 2024-12-18T00:08:47.5025744Z Running distributed/test_device_mesh 1/1 ... [2024-12-18 00:08:47.502106] 2024-12-18T00:08:47.5026300Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:08:47.5027525Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_device_mesh.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:08:47.502519] 2024-12-18T00:13:26.3402501Z 2024-12-18T00:13:26.3403578Z distributed/test_device_mesh 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_device_mesh_1.1_20328142d33f817f_.log 2024-12-18T00:13:26.3424311Z Running 47 items in this shard: test/distributed/test_device_mesh.py::DeviceMeshTestGlooBackend::test_device_mesh_reuse_default_group, test/distributed/test_device_mesh.py::DeviceMeshTest::test_2d_mesh_eager_init_subgroup, test/distributed/test_device_mesh.py::DeviceMeshTest::test_2d_mesh_non_eager_init_subgroup, test/distributed/test_device_mesh.py::DeviceMeshTest::test_assert_invalid_mesh_tensor, test/distributed/test_device_mesh.py::DeviceMeshTest::test_device_mesh_2d, test/distributed/test_device_mesh.py::DeviceMeshTest::test_device_mesh_init_backend, test/distributed/test_device_mesh.py::DeviceMeshTest::test_fake_pg_device_mesh, test/distributed/test_device_mesh.py::DeviceMeshTest::test_from_group_with_global_pg, test/distributed/test_device_mesh.py::DeviceMeshTest::test_from_group_with_invalid_mesh, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_group_and_get_all_groups, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_local_rank, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_local_rank_raises_exception, test/distributed/test_device_mesh.py::DeviceMeshTest::test_init_process_group, test/distributed/test_device_mesh.py::DeviceMeshTest::test_raises_invalid_device_type, test/distributed/test_device_mesh.py::DeviceMeshTest::test_set_mesh_dim_group_options, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_hash, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_nd, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_parent_child_hash, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_from_group_with_mesh_shape, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_get_local_rank_3d, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_init_device_mesh, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_raises_duplicate_mesh_dim_names, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_raises_mesh_shape_mesh_dim_names_mismatch, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_cache_and_reuse_submesh_slice_result, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_flatten_mesh_3d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_flatten_mesh_4d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_1d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_2d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_3d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_3d_noncontiguous_slicing, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_invalid_mesh_dim_name, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_no_mesh_dim_found, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_reconstruct_mesh_with_flatten_dim, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_all_submeshes, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_mesh_dim_by_name, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_root_mesh, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_root_mesh_dim_exist, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_root_mesh_dim_not_exist, test/distributed/test_device_mesh.py::TestMeshEnv::test_mesh_slice_fake_tensor_mode, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_all_gather_uneven, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_1d, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_nd, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_reduce_scatter_contiguous, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_reduce_scatter_uneven, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_1d, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_nd, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_uneven 2024-12-18T00:13:26.3443963Z 2024-12-18T00:13:26.3444281Z Running distributed/_tensor/test_experimental_ops 1/1 ... [2024-12-18 00:13:26.340026] 2024-12-18T00:13:26.3444882Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:13:26.3446181Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_experimental_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:13:26.340392] 2024-12-18T00:13:47.8102311Z 2024-12-18T00:13:47.8103759Z distributed/_tensor/test_experimental_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_experimental_ops_1.1_824e98994873664b_.log 2024-12-18T00:13:47.8106039Z Running 3 items in this shard: test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_bernoulli, test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_nll, test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_slice 2024-12-18T00:13:47.8107476Z 2024-12-18T00:13:47.8108562Z Running distributed/_tensor/test_tensor_ops 1/1 ... [2024-12-18 00:13:47.810440] 2024-12-18T00:13:47.8109126Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:13:47.8110681Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_tensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:13:47.810818] 2024-12-18T00:18:14.7364476Z 2024-12-18T00:18:14.7365678Z distributed/_tensor/test_tensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_tensor_ops_1.1_655f3bc36620a502_.log 2024-12-18T00:18:14.7376511Z Running 25 items in this shard: test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_aten_contiguous, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_clone, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_contiguous, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_detach, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_dtensor_dtype_conversion, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_empty_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_equal, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_fill_inplace, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_fill_inplace_partial_sum, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_full_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_gather, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_index, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_inplace_op, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_new_empty_strided, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_new_full, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_ones_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_ones_like_partial_sum, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_op_out_variant, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_scatter, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_slice, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_stack, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_where_type_promotion, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zero_inplace, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zeros_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zeros_like_partial_sum 2024-12-18T00:18:14.7387299Z 2024-12-18T00:18:14.7387728Z Running distributed/_composable/fsdp/test_fully_shard_extensions 1/1 ... [2024-12-18 00:18:14.736571] 2024-12-18T00:18:14.7388399Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:18:14.7389804Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_extensions.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:18:14.736962] 2024-12-18T00:18:37.7555399Z 2024-12-18T00:18:37.7556773Z distributed/_composable/fsdp/test_fully_shard_extensions 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_extensions_1.1_085a69f5359d19d6_.log 2024-12-18T00:18:37.7562220Z Running 5 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_extensions.py::TestFullyShardAllGatherExtensionsMultiProcess::test_all_gather_extensions_train_parity, test/distributed/_composable/fsdp/test_fully_shard_extensions.py::TestFullyShardAllGatherExtensionsMultiThread::test_all_gather_extension_hsdp_mesh, test/distributed/_composable/fsdp/test_fully_shard_extensions.py::TestFullyShardAllGatherExtensionsMultiThread::test_all_gather_extension_outer_size_stride, test/distributed/_composable/fsdp/test_fully_shard_extensions.py::TestFullyShardAllGatherExtensionsMultiThread::test_all_gather_extensions_end_to_end, test/distributed/_composable/fsdp/test_fully_shard_extensions.py::TestFullyShardAllGatherExtensionsMultiThread::test_all_gather_extensions_monkey_patch 2024-12-18T00:18:37.7566308Z 2024-12-18T00:18:37.7566579Z Running distributed/fsdp/test_wrap 1/1 ... [2024-12-18 00:18:37.755590] 2024-12-18T00:18:37.7567108Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:18:37.7568333Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_wrap.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:18:37.755953] 2024-12-18T00:21:27.6139883Z 2024-12-18T00:21:27.6140996Z distributed/fsdp/test_wrap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_wrap_1.1_1cbb7aec53fa7ba0_.log 2024-12-18T00:21:27.6168748Z Running 51 items in this shard: test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_bn_always_wrapped_individually, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_wrap_batchnorm_individually_use_or_policy_False, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_wrap_batchnorm_individually_use_or_policy_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_always_wrap, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_always_wrap_with_ignored_modules_wrap_method0, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_always_wrap_with_ignored_modules_wrap_method1, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_api, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_exclude_wrap, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_exclude_wrap_include_children, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_force_leaf, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_force_leaf_custom, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode0_cpu_offload0_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode0_cpu_offload0_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode0_cpu_offload1_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode0_cpu_offload1_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode1_cpu_offload0_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode1_cpu_offload0_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode1_cpu_offload1_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode1_cpu_offload1_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_with_ignored_modules_wrap_method0, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_with_ignored_modules_wrap_method1, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_custom_policy, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_frozen_params, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_module_wrap_policy, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_module_wrap_policy_callable, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_transformer_auto_wrap_policy, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_disabled_outside_context, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_override_defaults, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_wrap_method0, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_wrap_method1, test/distributed/fsdp/test_wrap.py::TestWrapUtils::test_validate_frozen_params 2024-12-18T00:21:27.6196101Z 2024-12-18T00:21:27.6196478Z Running distributed/_composable/fsdp/test_fully_shard_autograd 1/1 ... [2024-12-18 00:21:27.614293] 2024-12-18T00:21:27.6197101Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:21:27.6198414Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_autograd.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:21:27.614659] 2024-12-18T00:22:03.5713216Z 2024-12-18T00:22:03.5714615Z distributed/_composable/fsdp/test_fully_shard_autograd 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_autograd_1.1_7b8f662490ea681d_.log 2024-12-18T00:22:03.5719026Z Running 5 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_autograd.py::TestFullyShardAutograd::test_nontensor_activations, test/distributed/_composable/fsdp/test_fully_shard_autograd.py::TestFullyShardAutograd::test_unused_forward_module, test/distributed/_composable/fsdp/test_fully_shard_autograd.py::TestFullyShardAutograd::test_unused_forward_output, test/distributed/_composable/fsdp/test_fully_shard_autograd.py::TestFullyShardPostAccGradHookMultiThread::test_post_acc_grad_hook_runs, test/distributed/_composable/fsdp/test_fully_shard_autograd.py::TestFullyShardPostAccGradHookMultiProcess::test_post_acc_grad_hook_optim_parity 2024-12-18T00:22:03.5722678Z 2024-12-18T00:22:03.5723354Z Running distributed/fsdp/test_fsdp_clip_grad_norm 1/1 ... [2024-12-18 00:22:03.571428] 2024-12-18T00:22:03.5723947Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:22:03.5725283Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_clip_grad_norm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:22:03.571797] 2024-12-18T00:23:01.8841895Z 2024-12-18T00:23:01.8843139Z distributed/fsdp/test_fsdp_clip_grad_norm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_clip_grad_norm_1.1_f28d8deddf23c59c_.log 2024-12-18T00:23:01.8846087Z Running 4 items in this shard: test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_ddp_parity_cuda, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_low_precision_grads_cuda, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_no_gradients_cuda, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_non_root_cuda 2024-12-18T00:23:01.8848170Z 2024-12-18T00:23:01.8848482Z Running distributed/fsdp/test_fsdp_misc 1/1 ... [2024-12-18 00:23:01.884268] 2024-12-18T00:23:01.8849032Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:23:01.8850282Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_misc.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:23:01.884634] 2024-12-18T00:24:25.9133319Z 2024-12-18T00:24:25.9134472Z distributed/fsdp/test_fsdp_misc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_misc_1.1_c9f4bc7a3a7dc932_.log 2024-12-18T00:24:25.9150572Z Running 28 items in this shard: test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_cpu_init_with_sync_module_states, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_cpu_init_stays_on_cpu, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_cpu_training, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_device_id_use_index_False, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_device_id_use_index_True, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_not_all_outputs_used_in_loss, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_optim_overlap_no_use_orig_params_error, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_optimizer_overlap, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_zero2_eval_with_prefetch, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_cpu_gpu_module, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_device_id_auto_wrap, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_device_id_cpu_offload, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_device_id_no_move_ignored_params_and_bufs, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_ignored_module_meta, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_namedtuple, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_same_model_across_ranks, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_unsupported_module_cls, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_homogeneous_attributes, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_module_device_mismatches_device_id, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_multigpu_module, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_no_params, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscWorldSize1::test_training_device_mismatch_errors, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscWorldSize1::test_unsafe_setattr, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscWorldSize1::test_world_size_1_sharding_strategy_warning 2024-12-18T00:24:25.9165147Z 2024-12-18T00:24:25.9165432Z Running distributed/test_c10d_common 1/1 ... [2024-12-18 00:24:25.913370] 2024-12-18T00:24:25.9165964Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:24:25.9167255Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_common.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:24:25.913734] 2024-12-18T00:27:49.0714233Z 2024-12-18T00:27:49.0715331Z distributed/test_c10d_common 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_common_1.1_e947e4b06e84b5f7_.log 2024-12-18T00:27:49.0726984Z Running 23 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype, test/distributed/test_c10d_common.py::CommTest::test_debug_level, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle, test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRank, test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRankOverridesFallback, test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnv, test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnvWithFallback 2024-12-18T00:27:49.0738761Z Running 1 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier 2024-12-18T00:27:49.0739995Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype 2024-12-18T00:27:49.0741331Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype 2024-12-18T00:27:49.0742512Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype 2024-12-18T00:27:49.0743713Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype 2024-12-18T00:27:49.0744749Z Running 1 items in this shard: test/distributed/test_c10d_common.py::CommTest::test_debug_level 2024-12-18T00:27:49.0745777Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr 2024-12-18T00:27:49.0746953Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config 2024-12-18T00:27:49.0748073Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives 2024-12-18T00:27:49.0749210Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name 2024-12-18T00:27:49.0750508Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends 2024-12-18T00:27:49.0751894Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available 2024-12-18T00:27:49.0752974Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv 2024-12-18T00:27:49.0754197Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2024-12-18T00:27:49.0755622Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2024-12-18T00:27:49.0756824Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop 2024-12-18T00:27:49.0759766Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable 2024-12-18T00:27:49.0760788Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal 2024-12-18T00:27:49.0761860Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle 2024-12-18T00:27:49.0762761Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRank 2024-12-18T00:27:49.0763753Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRankOverridesFallback 2024-12-18T00:27:49.0764736Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnv 2024-12-18T00:27:49.0765666Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnvWithFallback 2024-12-18T00:27:49.0766225Z 2024-12-18T00:27:49.0766520Z Running distributed/_tools/test_mod_tracker 1/1 ... [2024-12-18 00:27:49.071965] 2024-12-18T00:27:49.0767064Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:27:49.0768667Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_mod_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:27:49.072378] 2024-12-18T00:27:53.3981358Z 2024-12-18T00:27:53.3982625Z distributed/_tools/test_mod_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_mod_tracker_1.1_44cf98526b1f1078_.log 2024-12-18T00:27:53.3985550Z Running 4 items in this shard: test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_ac, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_bw_detection, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_module_hierarchy, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_user_hooks 2024-12-18T00:27:53.3987262Z 2024-12-18T00:27:53.3987644Z Running distributed/pipelining/test_schedule_multiproc 1/1 ... [2024-12-18 00:27:53.398087] 2024-12-18T00:27:53.3988268Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:27:53.3990588Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_schedule_multiproc.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:27:53.398478] 2024-12-18T00:28:09.0532801Z 2024-12-18T00:28:09.0534157Z distributed/pipelining/test_schedule_multiproc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_schedule_multiproc_1.1_0bc390351f80e4ff_.log 2024-12-18T00:28:09.0552985Z Running 27 items in this shard: test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_forward_only_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass0_shape_inference_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass0_shape_inference_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass1_shape_inference_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass1_shape_inference_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass2_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass2_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass0_ModelClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass1_ModelClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class0_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class0_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class1_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class1_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class2_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class2_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_pipeline_schedule_runtime_custom_sched_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_native_zero_bubble_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_native_zero_bubble_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_weight_update_mlp_e2e_ScheduleClass0 2024-12-18T00:28:09.0587224Z Running 27 items in this shard: test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_forward_only_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass0_shape_inference_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass0_shape_inference_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass1_shape_inference_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass1_shape_inference_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass2_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass2_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass0_ModelClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass1_ModelClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class0_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class0_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class1_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class1_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class2_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class2_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_pipeline_schedule_runtime_custom_sched_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_native_zero_bubble_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_native_zero_bubble_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_weight_update_mlp_e2e_ScheduleClass0 2024-12-18T00:28:09.0604380Z 2024-12-18T00:28:09.0604770Z Running distributed/tensor/parallel/test_micro_pipeline_tp 1/1 ... [2024-12-18 00:28:09.053376] 2024-12-18T00:28:09.0605402Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:28:09.0606844Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_micro_pipeline_tp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:28:09.053746] 2024-12-18T00:28:26.9586111Z 2024-12-18T00:28:26.9587611Z distributed/tensor/parallel/test_micro_pipeline_tp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_micro_pipeline_tp_1.1_29ea31d878dfd070_.log 2024-12-18T00:28:26.9607758Z Running 29 items in this shard: test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_all_gather_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_reduce_scatter_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_get_unexposed_collectives 2024-12-18T00:28:26.9627281Z 2024-12-18T00:28:26.9627624Z Running distributed/checkpoint/test_nested_dict 1/1 ... [2024-12-18 00:28:26.958903] 2024-12-18T00:28:26.9628235Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:28:26.9629566Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_nested_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:28:26.959288] 2024-12-18T00:28:31.2853183Z 2024-12-18T00:28:31.2854387Z distributed/checkpoint/test_nested_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_nested_dict_1.1_9ddf00a6a608549f_.log 2024-12-18T00:28:31.2856318Z Running 2 items in this shard: test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_flattening_round_trip, test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_mapping 2024-12-18T00:28:31.2857350Z 2024-12-18T00:28:31.2857689Z Running distributed/pipelining/test_transformer 1/1 ... [2024-12-18 00:28:31.285263] 2024-12-18T00:28:31.2858278Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:28:31.2859908Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_transformer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:28:31.285647] 2024-12-18T00:28:36.1124418Z 2024-12-18T00:28:36.1125703Z distributed/pipelining/test_transformer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_transformer_1.1_a8a30047e9c1a9b5_.log 2024-12-18T00:28:36.1127695Z Running 1 items in this shard: test/distributed/pipelining/test_transformer.py::TransformerTests::test_ir 2024-12-18T00:28:36.1128268Z 2024-12-18T00:28:36.1128612Z Running distributed/pipelining/test_backward 1/1 ... [2024-12-18 00:28:36.112526] 2024-12-18T00:28:36.1129305Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:28:36.1131016Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_backward.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:28:36.112916] 2024-12-18T00:28:40.5392437Z 2024-12-18T00:28:40.5393867Z distributed/pipelining/test_backward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_backward_1.1_9682a21479527d48_.log 2024-12-18T00:28:40.5397036Z Running 4 items in this shard: test/distributed/pipelining/test_backward.py::StageBackwardTests::test_stage_backward, test/distributed/pipelining/test_backward.py::StageBackwardTests::test_stage_backward_input, test/distributed/pipelining/test_backward.py::StageBackwardTests::test_stage_backward_weight, test/distributed/pipelining/test_backward.py::StageBackwardTests::test_stage_backward_weight_multiple_iters 2024-12-18T00:28:40.5399168Z 2024-12-18T00:28:40.5399507Z Running distributed/_shard/sharded_tensor/test_logger 1/1 ... [2024-12-18 00:28:40.539093] 2024-12-18T00:28:40.5400109Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:28:40.5401873Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_logger.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:28:40.539460] 2024-12-18T00:28:44.8154535Z 2024-12-18T00:28:44.8155846Z distributed/_shard/sharded_tensor/test_logger 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.test_logger_1.1_d94a14034fee1f41_.log 2024-12-18T00:28:44.8157571Z Running 1 items in this shard: test/distributed/_shard/sharded_tensor/test_logger.py::ShardingSpecLoggerTest::test_get_or_create_logger 2024-12-18T00:28:44.8158320Z 2024-12-18T00:28:44.8158689Z Running distributed/pipelining/test_pipe 1/1 ... [2024-12-18 00:28:44.815338] 2024-12-18T00:28:44.8159230Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:28:44.8160493Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_pipe.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:28:44.815706] 2024-12-18T00:28:49.7424944Z 2024-12-18T00:28:49.7426176Z distributed/pipelining/test_pipe 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_pipe_1.1_1a85a1dad4f76019_.log 2024-12-18T00:28:49.7428504Z Running 3 items in this shard: test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass0, test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass1, test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass2 2024-12-18T00:28:49.7430032Z 2024-12-18T00:28:49.7430361Z Running distributed/_tools/test_sac_estimator 1/1 ... [2024-12-18 00:28:49.742791] 2024-12-18T00:28:49.7430923Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:28:49.7433495Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_sac_estimator.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:28:49.743157] 2024-12-18T00:28:57.4760540Z 2024-12-18T00:28:57.4761723Z distributed/_tools/test_sac_estimator 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_sac_estimator_1.1_efac988a30c89dc4_.log 2024-12-18T00:28:57.4764099Z Running 2 items in this shard: test/distributed/_tools/test_sac_estimator.py::TestSACEstimator::test_simple_model_sac_estimation, test/distributed/_tools/test_sac_estimator.py::TestSACEstimator::test_transformer_sac_estimation 2024-12-18T00:28:57.4765348Z 2024-12-18T00:28:57.4765742Z Running distributed/_tools/test_sac_ilp 1/1 ... [2024-12-18 00:28:57.475969] 2024-12-18T00:28:57.4766319Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:28:57.4770249Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_sac_ilp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:28:57.476371] 2024-12-18T00:29:09.4700380Z 2024-12-18T00:29:09.4701691Z distributed/_tools/test_sac_ilp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_sac_ilp_1.1_83de39b7eb08edaa_.log 2024-12-18T00:29:09.4704342Z Running 4 items in this shard: test/distributed/_tools/test_sac_ilp.py::TestSACILP::test_sac_ilp_case1, test/distributed/_tools/test_sac_ilp.py::TestSACILP::test_sac_ilp_case2, test/distributed/_tools/test_sac_ilp.py::TestSACILP::test_sac_ilp_case3, test/distributed/_tools/test_sac_ilp.py::TestOptimalCheckpointingPolicy::test_get_optimial_checkpointing_policy_per_module 2024-12-18T00:29:09.4706241Z 2024-12-18T00:29:09.4706565Z Running distributed/test_c10d_functional_native 1/1 ... [2024-12-18 00:29:09.469899] 2024-12-18T00:29:09.4707151Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:29:09.4708742Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_functional_native.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:29:09.470375] 2024-12-18T00:31:53.1359915Z 2024-12-18T00:31:53.1361202Z distributed/test_c10d_functional_native 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_functional_native_1.1_dcafd296dbbfb2ff_.log 2024-12-18T00:31:53.1374341Z Running 28 items in this shard: test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_to_all_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_broadcast, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_inductor_dtypeview_memory_leak, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_py_work, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_threading, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_unwaited, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_wait_tensor, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_non_contig_input, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_to_all_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_broadcast, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_inplace_op_on_view, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reuse_buffer_after_inplace_collective, test/distributed/test_c10d_functional_native.py::CompileTest::test_ranks_and_tag, test/distributed/test_c10d_functional_native.py::CompileTest::test_wait_tensor 2024-12-18T00:31:53.1389227Z 2024-12-18T00:31:53.1389573Z Running distributed/checkpoint/test_save_load_api 1/1 ... [2024-12-18 00:31:53.136339] 2024-12-18T00:31:53.1390177Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:31:53.1391606Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_save_load_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:31:53.136930] 2024-12-18T00:32:03.1770225Z 2024-12-18T00:32:03.1772717Z distributed/checkpoint/test_save_load_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_save_load_api_1.1_34ae5c10d327c468_.log 2024-12-18T00:32:03.1774227Z Running 1 items in this shard: test/distributed/checkpoint/test_save_load_api.py::TestSaveAndLoadAPI::test_auto_detect 2024-12-18T00:32:03.1774834Z 2024-12-18T00:32:03.1779656Z Running distributed/checkpoint/test_dedup_tensors 1/1 ... [2024-12-18 00:32:03.177387] 2024-12-18T00:32:03.1780310Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:32:03.1781666Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_dedup_tensors.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:32:03.177760] 2024-12-18T00:32:07.4528850Z 2024-12-18T00:32:07.4530137Z distributed/checkpoint/test_dedup_tensors 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dedup_tensors_1.1_60b0df5cf9ccfd18_.log 2024-12-18T00:32:07.4531743Z Running 1 items in this shard: test/distributed/checkpoint/test_dedup_tensors.py::TestDedupTensor::test_dedup_shards 2024-12-18T00:32:07.4532390Z 2024-12-18T00:32:07.4533159Z Running distributed/_composable/test_contract 1/1 ... [2024-12-18 00:32:07.453188] 2024-12-18T00:32:07.4533781Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:32:07.4537584Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_contract.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:32:07.453581] 2024-12-18T00:32:11.7798189Z 2024-12-18T00:32:11.7799651Z distributed/_composable/test_contract 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_contract_1.1_dcc08c236508da95_.log 2024-12-18T00:32:11.7802801Z Running 5 items in this shard: test/distributed/_composable/test_contract.py::TestContract::test_add_hooks, test/distributed/_composable/test_contract.py::TestContract::test_modify_fqn, test/distributed/_composable/test_contract.py::TestContract::test_multi_module_api, test/distributed/_composable/test_contract.py::TestContract::test_registry, test/distributed/_composable/test_contract.py::TestContract::test_state 2024-12-18T00:32:11.7804803Z 2024-12-18T00:32:11.7805146Z Running distributed/pipelining/test_microbatch 1/1 ... [2024-12-18 00:32:11.779741] 2024-12-18T00:32:11.7805735Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:32:11.7807372Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_microbatch.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:32:11.780129] 2024-12-18T00:32:16.4067265Z 2024-12-18T00:32:16.4068536Z distributed/pipelining/test_microbatch 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_microbatch_1.1_3c7add31fd4d5aaf_.log 2024-12-18T00:32:16.4070439Z Running 2 items in this shard: test/distributed/pipelining/test_microbatch.py::MicrobatchTests::test_chunk_spec, test/distributed/pipelining/test_microbatch.py::MicrobatchTests::test_split_and_merge 2024-12-18T00:32:16.4071496Z 2024-12-18T00:32:16.4075670Z Running distributed/optim/test_named_optimizer 1/1 ... [2024-12-18 00:32:16.406914] 2024-12-18T00:32:16.4076261Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:32:16.4077569Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/optim/test_named_optimizer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:32:16.407280] 2024-12-18T00:32:19.3821441Z 2024-12-18T00:32:19.3822972Z distributed/optim/test_named_optimizer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_named_optimizer_1.1_c2176295cfe0f188_.log 2024-12-18T00:32:19.3824030Z 2024-12-18T00:32:19.3824392Z Running distributed/checkpoint/e2e/test_fsdp_ep 1/1 ... [2024-12-18 00:32:19.382157] 2024-12-18T00:32:19.3825007Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:32:19.3827982Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/e2e/test_fsdp_ep.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:32:19.382567] 2024-12-18T00:32:28.6216594Z 2024-12-18T00:32:28.6217826Z distributed/checkpoint/e2e/test_fsdp_ep 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.e2e.test_fsdp_ep_1.1_4e2a9011772a3b00_.log 2024-12-18T00:32:28.6219642Z Running 1 items in this shard: test/distributed/checkpoint/e2e/test_fsdp_ep.py::TestFSDPWithEP::test_e2e 2024-12-18T00:32:28.6220218Z 2024-12-18T00:32:28.6220633Z Running distributed/_tensor/experimental/test_register_sharding 1/1 ... [2024-12-18 00:32:28.621539] 2024-12-18T00:32:28.6221309Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:32:28.6223514Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/experimental/test_register_sharding.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:32:28.621909] 2024-12-18T00:32:46.5303394Z 2024-12-18T00:32:46.5304793Z distributed/_tensor/experimental/test_register_sharding 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.experimental.test_register_sharding_1.1_14320eed10b309d6_.log 2024-12-18T00:32:46.5307155Z Running 2 items in this shard: test/distributed/_tensor/experimental/test_register_sharding.py::TestRegisterSharding::test_argmax, test/distributed/_tensor/experimental/test_register_sharding.py::TestRegisterSharding::test_softmax_fwd 2024-12-18T00:32:46.5308351Z 2024-12-18T00:32:46.5310560Z Running distributed/_tensor/test_attention 1/1 ... [2024-12-18 00:32:46.530658] 2024-12-18T00:32:46.5311239Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:32:46.5312519Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_attention.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:32:46.531019] 2024-12-18T00:34:20.3404184Z 2024-12-18T00:34:20.3405351Z distributed/_tensor/test_attention 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_attention_1.1_55b3b1d3a8d03733_.log 2024-12-18T00:34:20.3422490Z Running 23 items in this shard: test/distributed/_tensor/test_attention.py::RingAttentionTest::test_is_causal_behavior, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_custom_transformer_rotater0, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_custom_transformer_rotater1, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_native_transformer_is_causal_False_rotater0, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_native_transformer_is_causal_False_rotater1, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_native_transformer_is_causal_True_rotater0, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_native_transformer_is_causal_True_rotater1, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa_is_causal_False_compiled_False_backend0_load_balance_False_rotater0, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa_is_causal_False_compiled_False_backend0_load_balance_False_rotater1, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa_is_causal_False_compiled_False_backend0_load_balance_True_rotater0, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa_is_causal_False_compiled_False_backend0_load_balance_True_rotater1, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa_is_causal_False_compiled_True_backend0_load_balance_False_rotater0, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa_is_causal_False_compiled_True_backend0_load_balance_False_rotater1, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa_is_causal_False_compiled_True_backend0_load_balance_True_rotater0, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa_is_causal_False_compiled_True_backend0_load_balance_True_rotater1, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa_is_causal_True_compiled_False_backend0_load_balance_False_rotater0, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa_is_causal_True_compiled_False_backend0_load_balance_False_rotater1, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa_is_causal_True_compiled_False_backend0_load_balance_True_rotater0, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa_is_causal_True_compiled_False_backend0_load_balance_True_rotater1, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa_is_causal_True_compiled_True_backend0_load_balance_False_rotater0, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa_is_causal_True_compiled_True_backend0_load_balance_False_rotater1, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa_is_causal_True_compiled_True_backend0_load_balance_True_rotater0, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa_is_causal_True_compiled_True_backend0_load_balance_True_rotater1 2024-12-18T00:34:20.3438518Z 2024-12-18T00:34:20.3438765Z Running distributed/test_fake_pg 1/1 ... [2024-12-18 00:34:20.340438] 2024-12-18T00:34:20.3439260Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:34:20.3440443Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_fake_pg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:34:20.340808] 2024-12-18T00:34:25.5186625Z 2024-12-18T00:34:25.5187709Z distributed/test_fake_pg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_fake_pg_1.1_e84fb7c933bbe00d_.log 2024-12-18T00:34:25.5192371Z Running 13 items in this shard: test/distributed/test_fake_pg.py::TestFakePG::test_all_reduce, test/distributed/test_fake_pg.py::TestFakePG::test_allgather, test/distributed/test_fake_pg.py::TestFakePG::test_alltoall, test/distributed/test_fake_pg.py::TestFakePG::test_alltoall_base, test/distributed/test_fake_pg.py::TestFakePG::test_broadcast, test/distributed/test_fake_pg.py::TestFakePG::test_construct_fsdp, test/distributed/test_fake_pg.py::TestFakePG::test_fake_pg_tracing, test/distributed/test_fake_pg.py::TestFakePG::test_fsdp_fake_e2e, test/distributed/test_fake_pg.py::TestFakePG::test_fsdp_tp_fake_e2e, test/distributed/test_fake_pg.py::TestFakePG::test_recv, test/distributed/test_fake_pg.py::TestFakePG::test_reduce_scatter, test/distributed/test_fake_pg.py::TestFakePG::test_scatter, test/distributed/test_fake_pg.py::TestFakePG::test_send 2024-12-18T00:34:25.5196280Z 2024-12-18T00:34:25.5196649Z Running distributed/checkpoint/test_fsdp_optim_state 1/1 ... [2024-12-18 00:34:25.518835] 2024-12-18T00:34:25.5197240Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:34:25.5198555Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_optim_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:34:25.519218] 2024-12-18T00:34:43.4308174Z 2024-12-18T00:34:43.4310122Z distributed/checkpoint/test_fsdp_optim_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsdp_optim_state_1.1_e0c2fd9a3de4f808_.log 2024-12-18T00:34:43.4312771Z Running 2 items in this shard: test/distributed/checkpoint/test_fsdp_optim_state.py::FsdpOptimStateCheckpoint::test_load_sharded_optimizer_state_dict_pass_planner_False, test/distributed/checkpoint/test_fsdp_optim_state.py::FsdpOptimStateCheckpoint::test_load_sharded_optimizer_state_dict_pass_planner_True 2024-12-18T00:34:43.4314337Z 2024-12-18T00:34:43.4314652Z Running distributed/_composable/test_checkpoint 1/1 ... [2024-12-18 00:34:43.431347] 2024-12-18T00:34:43.4315227Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:34:43.4318863Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:34:43.431711] 2024-12-18T00:34:48.2081184Z 2024-12-18T00:34:48.2082595Z distributed/_composable/test_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_checkpoint_1.1_6638656fd917ce3f_.log 2024-12-18T00:34:48.2086216Z Running 6 items in this shard: test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_checkpoint_kwargs, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_clears_state_on_error_in_forward, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_multi_args, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_random_cpu, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_tensor_only_cpu, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_tensor_only_gpu 2024-12-18T00:34:48.2089090Z 2024-12-18T00:34:48.2089457Z Running distributed/checkpoint/test_dtensor_resharding 1/1 ... [2024-12-18 00:34:48.208042] 2024-12-18T00:34:48.2090082Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:34:48.2091425Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_dtensor_resharding.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:34:48.208435] 2024-12-18T00:35:44.9298633Z 2024-12-18T00:35:44.9300170Z distributed/checkpoint/test_dtensor_resharding 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dtensor_resharding_1.1_2adb3d8109de5f87_.log 2024-12-18T00:35:44.9304832Z Running 5 items in this shard: test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardPlacementChange::test_1d_to_1d_reshard_placement_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardPlacementChange::test_2d_to_2d_reshard_placement_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_1d_to_2d_reshard_mesh_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_2d_to_1d_reshard_mesh_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_dtensor_checkpoint_resharding_with_empty_shard 2024-12-18T00:35:44.9308330Z 2024-12-18T00:35:44.9308631Z Running distributed/test_distributed_spawn 2/9 ... [2024-12-18 00:35:44.930076] 2024-12-18T00:35:44.9312435Z /usr/bin/mpiexec 2024-12-18T00:35:44.9313417Z MPI not available -- MPI backend tests will be skipped 2024-12-18T00:35:44.9314857Z Running distributed tests for the test backend with env init_method 2024-12-18T00:35:44.9317201Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:35:44.9321339Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:35:44.931952] 2024-12-18T00:35:49.1428926Z 2024-12-18T00:35:49.1430145Z distributed/test_distributed_spawn 2/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.9_9abe803ee66f3e4b_.log 2024-12-18T00:35:49.1431235Z Running 0 items in this shard: 2024-12-18T00:35:49.1431459Z 2024-12-18T00:35:49.1439887Z Running distributed tests for the test backend with file init_method 2024-12-18T00:35:49.1440447Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:35:49.1444316Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:35:49.144244] 2024-12-18T00:35:53.3418391Z 2024-12-18T00:35:53.3419880Z distributed/test_distributed_spawn 2/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.9_c53285b5fb75255a_.log 2024-12-18T00:35:53.3420946Z Running 0 items in this shard: 2024-12-18T00:35:53.3421168Z 2024-12-18T00:35:53.3424803Z Running distributed tests for the nccl backend with env init_method 2024-12-18T00:35:53.3426643Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:35:53.3430935Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:35:53.342879] 2024-12-18T00:40:09.1487783Z 2024-12-18T00:40:09.1488973Z distributed/test_distributed_spawn 2/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.9_30a30dc6053d7711_.log 2024-12-18T00:40:09.1504929Z Running 28 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2024-12-18T00:40:09.1520522Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2024-12-18T00:40:09.1522008Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding 2024-12-18T00:40:09.1523662Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group 2024-12-18T00:40:09.1526197Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty 2024-12-18T00:40:09.1527536Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2024-12-18T00:40:09.1528803Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2024-12-18T00:40:09.1530002Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2024-12-18T00:40:09.1531241Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2024-12-18T00:40:09.1532603Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex 2024-12-18T00:40:09.1533993Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group 2024-12-18T00:40:09.1535220Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda 2024-12-18T00:40:09.1536381Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward 2024-12-18T00:40:09.1537670Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future 2024-12-18T00:40:09.1539204Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2024-12-18T00:40:09.1540405Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized 2024-12-18T00:40:09.1541590Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce 2024-12-18T00:40:09.1542837Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2024-12-18T00:40:09.1544031Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone 2024-12-18T00:40:09.1545235Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2024-12-18T00:40:09.1546430Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group 2024-12-18T00:40:09.1547719Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2024-12-18T00:40:09.1549098Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks 2024-12-18T00:40:09.1550450Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2024-12-18T00:40:09.1552025Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size 2024-12-18T00:40:09.1553393Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed 2024-12-18T00:40:09.1554647Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module 2024-12-18T00:40:09.1555843Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager 2024-12-18T00:40:09.1556970Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2024-12-18T00:40:09.1557585Z 2024-12-18T00:40:09.1557847Z Running distributed tests for the nccl backend with file init_method 2024-12-18T00:40:09.1558344Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:40:09.1559617Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:40:09.150289] 2024-12-18T00:44:24.7684261Z 2024-12-18T00:44:24.7685430Z distributed/test_distributed_spawn 2/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.9_6da35615c1074dea_.log 2024-12-18T00:44:24.7701329Z Running 28 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2024-12-18T00:44:24.7717143Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2024-12-18T00:44:24.7718438Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding 2024-12-18T00:44:24.7719673Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group 2024-12-18T00:44:24.7720911Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty 2024-12-18T00:44:24.7722575Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2024-12-18T00:44:24.7723846Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2024-12-18T00:44:24.7725029Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2024-12-18T00:44:24.7726271Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2024-12-18T00:44:24.7727744Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex 2024-12-18T00:44:24.7729006Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group 2024-12-18T00:44:24.7730129Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda 2024-12-18T00:44:24.7731325Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward 2024-12-18T00:44:24.7732656Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future 2024-12-18T00:44:24.7734089Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2024-12-18T00:44:24.7735255Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized 2024-12-18T00:44:24.7736406Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce 2024-12-18T00:44:24.7737618Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2024-12-18T00:44:24.7738847Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone 2024-12-18T00:44:24.7740222Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2024-12-18T00:44:24.7741519Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group 2024-12-18T00:44:24.7742726Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2024-12-18T00:44:24.7744109Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks 2024-12-18T00:44:24.7745471Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2024-12-18T00:44:24.7746868Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size 2024-12-18T00:44:24.7748320Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed 2024-12-18T00:44:24.7749651Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module 2024-12-18T00:44:24.7750918Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager 2024-12-18T00:44:24.7752284Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2024-12-18T00:44:24.7752895Z 2024-12-18T00:44:24.7753151Z Running distributed tests for the gloo backend with env init_method 2024-12-18T00:44:24.7753640Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:44:24.7754923Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:44:24.769894] 2024-12-18T00:48:46.0519097Z 2024-12-18T00:48:46.0520281Z distributed/test_distributed_spawn 2/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.9_45751ecb2d5e33e0_.log 2024-12-18T00:48:46.0537056Z Running 28 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2024-12-18T00:48:46.0553242Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2024-12-18T00:48:46.0554509Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding 2024-12-18T00:48:46.0555704Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group 2024-12-18T00:48:46.0556904Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty 2024-12-18T00:48:46.0558147Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2024-12-18T00:48:46.0559332Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2024-12-18T00:48:46.0560432Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2024-12-18T00:48:46.0561673Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2024-12-18T00:48:46.0562945Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex 2024-12-18T00:48:46.0564119Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group 2024-12-18T00:48:46.0565176Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda 2024-12-18T00:48:46.0566290Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward 2024-12-18T00:48:46.0567539Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future 2024-12-18T00:48:46.0568782Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2024-12-18T00:48:46.0569915Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized 2024-12-18T00:48:46.0571042Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce 2024-12-18T00:48:46.0572214Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2024-12-18T00:48:46.0573342Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone 2024-12-18T00:48:46.0574532Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2024-12-18T00:48:46.0575638Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group 2024-12-18T00:48:46.0576772Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2024-12-18T00:48:46.0578066Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks 2024-12-18T00:48:46.0579590Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2024-12-18T00:48:46.0581000Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size 2024-12-18T00:48:46.0582454Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed 2024-12-18T00:48:46.0583787Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module 2024-12-18T00:48:46.0585064Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager 2024-12-18T00:48:46.0586248Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2024-12-18T00:48:46.0586909Z 2024-12-18T00:48:46.0587175Z Running distributed tests for the gloo backend with file init_method 2024-12-18T00:48:46.0587698Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:48:46.0589066Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:48:46.053455] 2024-12-18T00:53:06.9273253Z 2024-12-18T00:53:06.9274378Z distributed/test_distributed_spawn 2/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.9_4c89dbc6c1a5f9d9_.log 2024-12-18T00:53:06.9290185Z Running 28 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2024-12-18T00:53:06.9305835Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm 2024-12-18T00:53:06.9307171Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding 2024-12-18T00:53:06.9308439Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group 2024-12-18T00:53:06.9309714Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty 2024-12-18T00:53:06.9311041Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2024-12-18T00:53:06.9312448Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2024-12-18T00:53:06.9313669Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex 2024-12-18T00:53:06.9314879Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2024-12-18T00:53:06.9316198Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex 2024-12-18T00:53:06.9317415Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group 2024-12-18T00:53:06.9318501Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda 2024-12-18T00:53:06.9319650Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward 2024-12-18T00:53:06.9320938Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future 2024-12-18T00:53:06.9322628Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2024-12-18T00:53:06.9323837Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized 2024-12-18T00:53:06.9325033Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce 2024-12-18T00:53:06.9326277Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence 2024-12-18T00:53:06.9327593Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone 2024-12-18T00:53:06.9328801Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval 2024-12-18T00:53:06.9329982Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group 2024-12-18T00:53:06.9331178Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2024-12-18T00:53:06.9332548Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks 2024-12-18T00:53:06.9334008Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast 2024-12-18T00:53:06.9335446Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size 2024-12-18T00:53:06.9336809Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed 2024-12-18T00:53:06.9338061Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module 2024-12-18T00:53:06.9339507Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager 2024-12-18T00:53:06.9340690Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source 2024-12-18T00:53:06.9341356Z 2024-12-18T00:53:06.9341658Z Running distributed/test_distributed_spawn 5/9 ... [2024-12-18 00:53:06.928804] 2024-12-18T00:53:06.9342203Z /usr/bin/mpiexec 2024-12-18T00:53:06.9342563Z MPI not available -- MPI backend tests will be skipped 2024-12-18T00:53:06.9343140Z Running distributed tests for the test backend with env init_method 2024-12-18T00:53:06.9343655Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:53:06.9345004Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:53:06.930722] 2024-12-18T00:53:11.1425950Z 2024-12-18T00:53:11.1427088Z distributed/test_distributed_spawn 5/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.9_302bb113143f2f98_.log 2024-12-18T00:53:11.1428166Z Running 0 items in this shard: 2024-12-18T00:53:11.1428393Z 2024-12-18T00:53:11.1435194Z Running distributed tests for the test backend with file init_method 2024-12-18T00:53:11.1437217Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:53:11.1441264Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:53:11.143936] 2024-12-18T00:53:15.3494392Z 2024-12-18T00:53:15.3495539Z distributed/test_distributed_spawn 5/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.9_87b762fd44fa80f5_.log 2024-12-18T00:53:15.3496610Z Running 0 items in this shard: 2024-12-18T00:53:15.3496831Z 2024-12-18T00:53:15.3499048Z Running distributed tests for the nccl backend with env init_method 2024-12-18T00:53:15.3501043Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:53:15.3505471Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:53:15.350326] 2024-12-18T00:58:41.3232475Z 2024-12-18T00:58:41.3235384Z distributed/test_distributed_spawn 5/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.9_d2d3c4a56c261414_.log 2024-12-18T00:58:41.3254935Z Running 37 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2024-12-18T00:58:41.3274982Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel 2024-12-18T00:58:41.3276328Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook 2024-12-18T00:58:41.3277691Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2024-12-18T00:58:41.3278942Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple 2024-12-18T00:58:41.3280117Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex 2024-12-18T00:58:41.3281276Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex 2024-12-18T00:58:41.3282418Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda 2024-12-18T00:58:41.3283581Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum 2024-12-18T00:58:41.3284763Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2024-12-18T00:58:41.3285906Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda 2024-12-18T00:58:41.3287090Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split 2024-12-18T00:58:41.3288398Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group 2024-12-18T00:58:41.3289650Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl 2024-12-18T00:58:41.3290817Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err 2024-12-18T00:58:41.3292050Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging 2024-12-18T00:58:41.3293256Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2024-12-18T00:58:41.3294483Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD 2024-12-18T00:58:41.3295618Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2024-12-18T00:58:41.3296877Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2024-12-18T00:58:41.3298220Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2024-12-18T00:58:41.3299786Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2024-12-18T00:58:41.3301116Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2024-12-18T00:58:41.3302312Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2024-12-18T00:58:41.3303518Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2024-12-18T00:58:41.3304803Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda 2024-12-18T00:58:41.3305950Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup 2024-12-18T00:58:41.3307085Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2024-12-18T00:58:41.3308241Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group 2024-12-18T00:58:41.3309532Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync 2024-12-18T00:58:41.3310839Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend 2024-12-18T00:58:41.3312336Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd 2024-12-18T00:58:41.3313643Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2024-12-18T00:58:41.3314740Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2024-12-18T00:58:41.3315843Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda 2024-12-18T00:58:41.3316962Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2024-12-18T00:58:41.3318162Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler 2024-12-18T00:58:41.3319437Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2024-12-18T00:58:41.3320122Z 2024-12-18T00:58:41.3320376Z Running distributed tests for the nccl backend with file init_method 2024-12-18T00:58:41.3320884Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:58:41.3322581Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:58:41.325174] 2024-12-18T01:04:07.1175084Z 2024-12-18T01:04:07.1176262Z distributed/test_distributed_spawn 5/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.9_d6d5f31063ca4623_.log 2024-12-18T01:04:07.1196542Z Running 37 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2024-12-18T01:04:07.1215721Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel 2024-12-18T01:04:07.1217065Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook 2024-12-18T01:04:07.1218444Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2024-12-18T01:04:07.1219983Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple 2024-12-18T01:04:07.1221200Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex 2024-12-18T01:04:07.1222617Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex 2024-12-18T01:04:07.1223791Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda 2024-12-18T01:04:07.1224988Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum 2024-12-18T01:04:07.1226213Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2024-12-18T01:04:07.1227396Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda 2024-12-18T01:04:07.1228756Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split 2024-12-18T01:04:07.1230100Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group 2024-12-18T01:04:07.1231387Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl 2024-12-18T01:04:07.1232589Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err 2024-12-18T01:04:07.1233940Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging 2024-12-18T01:04:07.1235113Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2024-12-18T01:04:07.1236310Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD 2024-12-18T01:04:07.1237406Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2024-12-18T01:04:07.1238637Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2024-12-18T01:04:07.1239941Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2024-12-18T01:04:07.1241209Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2024-12-18T01:04:07.1242441Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2024-12-18T01:04:07.1243563Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2024-12-18T01:04:07.1244696Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2024-12-18T01:04:07.1245891Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda 2024-12-18T01:04:07.1246972Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup 2024-12-18T01:04:07.1248342Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2024-12-18T01:04:07.1249623Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group 2024-12-18T01:04:07.1250877Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync 2024-12-18T01:04:07.1252048Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend 2024-12-18T01:04:07.1253332Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd 2024-12-18T01:04:07.1254636Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2024-12-18T01:04:07.1255734Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2024-12-18T01:04:07.1256831Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda 2024-12-18T01:04:07.1257959Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2024-12-18T01:04:07.1259480Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler 2024-12-18T01:04:07.1260777Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2024-12-18T01:04:07.1261492Z 2024-12-18T01:04:07.1261755Z Running distributed tests for the gloo backend with env init_method 2024-12-18T01:04:07.1262280Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:04:07.1263654Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:04:07.119082] 2024-12-18T01:09:45.3810958Z 2024-12-18T01:09:45.3812249Z distributed/test_distributed_spawn 5/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.9_0735238399ddf8cc_.log 2024-12-18T01:09:45.3832816Z Running 37 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2024-12-18T01:09:45.3852380Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel 2024-12-18T01:09:45.3853733Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook 2024-12-18T01:09:45.3855116Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2024-12-18T01:09:45.3856372Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple 2024-12-18T01:09:45.3857549Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex 2024-12-18T01:09:45.3858788Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex 2024-12-18T01:09:45.3860139Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda 2024-12-18T01:09:45.3861335Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum 2024-12-18T01:09:45.3862561Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2024-12-18T01:09:45.3863751Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda 2024-12-18T01:09:45.3865065Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split 2024-12-18T01:09:45.3866399Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group 2024-12-18T01:09:45.3867684Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl 2024-12-18T01:09:45.3868899Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err 2024-12-18T01:09:45.3870115Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging 2024-12-18T01:09:45.3871444Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2024-12-18T01:09:45.3872672Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD 2024-12-18T01:09:45.3873802Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2024-12-18T01:09:45.3875068Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2024-12-18T01:09:45.3876413Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2024-12-18T01:09:45.3877781Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2024-12-18T01:09:45.3879065Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2024-12-18T01:09:45.3880229Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2024-12-18T01:09:45.3881387Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2024-12-18T01:09:45.3882525Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda 2024-12-18T01:09:45.3883635Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup 2024-12-18T01:09:45.3884758Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2024-12-18T01:09:45.3885878Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group 2024-12-18T01:09:45.3887129Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync 2024-12-18T01:09:45.3888299Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend 2024-12-18T01:09:45.3889523Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd 2024-12-18T01:09:45.3890827Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2024-12-18T01:09:45.3891926Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2024-12-18T01:09:45.3893031Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda 2024-12-18T01:09:45.3894162Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2024-12-18T01:09:45.3895418Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler 2024-12-18T01:09:45.3896681Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2024-12-18T01:09:45.3897352Z 2024-12-18T01:09:45.3897623Z Running distributed tests for the gloo backend with file init_method 2024-12-18T01:09:45.3898119Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:09:45.3899701Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:09:45.382794] 2024-12-18T01:15:24.0902137Z 2024-12-18T01:15:24.0903455Z distributed/test_distributed_spawn 5/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.9_5682993814e1d560_.log 2024-12-18T01:15:24.0924189Z Running 37 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2024-12-18T01:15:24.0944481Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel 2024-12-18T01:15:24.0945982Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook 2024-12-18T01:15:24.0947361Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group 2024-12-18T01:15:24.0948607Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple 2024-12-18T01:15:24.0949780Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex 2024-12-18T01:15:24.0951094Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex 2024-12-18T01:15:24.0952206Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda 2024-12-18T01:15:24.0953341Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum 2024-12-18T01:15:24.0954494Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2024-12-18T01:15:24.0955610Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda 2024-12-18T01:15:24.0956774Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split 2024-12-18T01:15:24.0958028Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group 2024-12-18T01:15:24.0959242Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl 2024-12-18T01:15:24.0960371Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err 2024-12-18T01:15:24.0961505Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging 2024-12-18T01:15:24.0962678Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization 2024-12-18T01:15:24.0963866Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD 2024-12-18T01:15:24.0964966Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference 2024-12-18T01:15:24.0966198Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2024-12-18T01:15:24.0967510Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler 2024-12-18T01:15:24.0968837Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception 2024-12-18T01:15:24.0970081Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2024-12-18T01:15:24.0971202Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2024-12-18T01:15:24.0972353Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2024-12-18T01:15:24.0973458Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda 2024-12-18T01:15:24.0974542Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup 2024-12-18T01:15:24.0975627Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend 2024-12-18T01:15:24.0976714Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group 2024-12-18T01:15:24.0977936Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync 2024-12-18T01:15:24.0979343Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend 2024-12-18T01:15:24.0980616Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd 2024-12-18T01:15:24.0982059Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2024-12-18T01:15:24.0983199Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2024-12-18T01:15:24.0984344Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda 2024-12-18T01:15:24.0985515Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2024-12-18T01:15:24.0986766Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler 2024-12-18T01:15:24.0988061Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler 2024-12-18T01:15:24.0988770Z 2024-12-18T01:15:24.0989071Z Running distributed/test_distributed_spawn 8/9 ... [2024-12-18 01:15:24.091940] 2024-12-18T01:15:24.0989618Z /usr/bin/mpiexec 2024-12-18T01:15:24.0989989Z MPI not available -- MPI backend tests will be skipped 2024-12-18T01:15:24.0990567Z Running distributed tests for the test backend with env init_method 2024-12-18T01:15:24.0991097Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:15:24.0992421Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:15:24.093955] 2024-12-18T01:15:28.2937834Z 2024-12-18T01:15:28.2939180Z distributed/test_distributed_spawn 8/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.9_af74545114a96014_.log 2024-12-18T01:15:28.2940473Z Running 0 items in this shard: 2024-12-18T01:15:28.2940718Z 2024-12-18T01:15:28.2945128Z Running distributed tests for the test backend with file init_method 2024-12-18T01:15:28.2946864Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:15:28.2951093Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:15:28.294917] 2024-12-18T01:15:32.5029993Z 2024-12-18T01:15:32.5031182Z distributed/test_distributed_spawn 8/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.9_d7699238841b4071_.log 2024-12-18T01:15:32.5032251Z Running 0 items in this shard: 2024-12-18T01:15:32.5032474Z 2024-12-18T01:15:32.5039020Z Running distributed tests for the nccl backend with env init_method 2024-12-18T01:15:32.5040882Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:15:32.5045069Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:15:32.504318] 2024-12-18T01:19:13.6892454Z 2024-12-18T01:19:13.6893918Z distributed/test_distributed_spawn 8/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.9_95d273721db6a18d_.log 2024-12-18T01:19:13.6909366Z Running 26 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2024-12-18T01:19:13.6924809Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process 2024-12-18T01:19:13.6926363Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half 2024-12-18T01:19:13.6927728Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product 2024-12-18T01:19:13.6928889Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all 2024-12-18T01:19:13.6930002Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2024-12-18T01:19:13.6931249Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2024-12-18T01:19:13.6932646Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2024-12-18T01:19:13.6934216Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda 2024-12-18T01:19:13.6935503Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2024-12-18T01:19:13.6936637Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group 2024-12-18T01:19:13.6937986Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params 2024-12-18T01:19:13.6939479Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer 2024-12-18T01:19:13.6940881Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none 2024-12-18T01:19:13.6942496Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2024-12-18T01:19:13.6943961Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler 2024-12-18T01:19:13.6945247Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception 2024-12-18T01:19:13.6946467Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2024-12-18T01:19:13.6947745Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2024-12-18T01:19:13.6949067Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration 2024-12-18T01:19:13.6950366Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2024-12-18T01:19:13.6951828Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2024-12-18T01:19:13.6952931Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum 2024-12-18T01:19:13.6954171Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group 2024-12-18T01:19:13.6955288Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum 2024-12-18T01:19:13.6956530Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2024-12-18T01:19:13.6957659Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2024-12-18T01:19:13.6958264Z 2024-12-18T01:19:13.6958539Z Running distributed tests for the nccl backend with file init_method 2024-12-18T01:19:13.6959134Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:19:13.6960428Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:19:13.690578] 2024-12-18T01:22:54.7385436Z 2024-12-18T01:22:54.7386914Z distributed/test_distributed_spawn 8/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.9_c4199e6eb08a7a72_.log 2024-12-18T01:22:54.7403501Z Running 26 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2024-12-18T01:22:54.7418408Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process 2024-12-18T01:22:54.7420368Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half 2024-12-18T01:22:54.7421730Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product 2024-12-18T01:22:54.7423128Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all 2024-12-18T01:22:54.7424237Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2024-12-18T01:22:54.7425485Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2024-12-18T01:22:54.7426883Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2024-12-18T01:22:54.7428333Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda 2024-12-18T01:22:54.7429669Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2024-12-18T01:22:54.7430845Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group 2024-12-18T01:22:54.7432128Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params 2024-12-18T01:22:54.7433631Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer 2024-12-18T01:22:54.7435066Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none 2024-12-18T01:22:54.7436573Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2024-12-18T01:22:54.7437962Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler 2024-12-18T01:22:54.7439170Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception 2024-12-18T01:22:54.7440311Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2024-12-18T01:22:54.7441511Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2024-12-18T01:22:54.7442748Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration 2024-12-18T01:22:54.7443981Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2024-12-18T01:22:54.7445182Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2024-12-18T01:22:54.7446272Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum 2024-12-18T01:22:54.7447332Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group 2024-12-18T01:22:54.7448418Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum 2024-12-18T01:22:54.7449553Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2024-12-18T01:22:54.7450646Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2024-12-18T01:22:54.7451319Z 2024-12-18T01:22:54.7451577Z Running distributed tests for the gloo backend with env init_method 2024-12-18T01:22:54.7452066Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:22:54.7453350Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:22:54.740273] 2024-12-18T01:26:53.3767533Z 2024-12-18T01:26:53.3768713Z distributed/test_distributed_spawn 8/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.9_fc9d2d2f31142769_.log 2024-12-18T01:26:53.3783798Z Running 26 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2024-12-18T01:26:53.3798774Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process 2024-12-18T01:26:53.3800273Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half 2024-12-18T01:26:53.3801715Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product 2024-12-18T01:26:53.3802868Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all 2024-12-18T01:26:53.3803945Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2024-12-18T01:26:53.3805165Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2024-12-18T01:26:53.3806519Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2024-12-18T01:26:53.3807943Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda 2024-12-18T01:26:53.3809237Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2024-12-18T01:26:53.3810361Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group 2024-12-18T01:26:53.3811599Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params 2024-12-18T01:26:53.3812867Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer 2024-12-18T01:26:53.3814221Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none 2024-12-18T01:26:53.3815842Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2024-12-18T01:26:53.3817272Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler 2024-12-18T01:26:53.3818610Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception 2024-12-18T01:26:53.3819993Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2024-12-18T01:26:53.3821252Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2024-12-18T01:26:53.3822803Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration 2024-12-18T01:26:53.3824129Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2024-12-18T01:26:53.3825408Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2024-12-18T01:26:53.3826586Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum 2024-12-18T01:26:53.3827718Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group 2024-12-18T01:26:53.3828874Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum 2024-12-18T01:26:53.3830073Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2024-12-18T01:26:53.3831227Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2024-12-18T01:26:53.3831862Z 2024-12-18T01:26:53.3832119Z Running distributed tests for the gloo backend with file init_method 2024-12-18T01:26:53.3832765Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:26:53.3834207Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=9', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:26:53.378000] 2024-12-18T01:30:52.3220389Z 2024-12-18T01:30:52.3222303Z distributed/test_distributed_spawn 8/9 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.9_b2097b31d6561d68_.log 2024-12-18T01:30:52.3250530Z Running 26 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2024-12-18T01:30:52.3266785Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process 2024-12-18T01:30:52.3268335Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half 2024-12-18T01:30:52.3269652Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product 2024-12-18T01:30:52.3270759Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all 2024-12-18T01:30:52.3272030Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2024-12-18T01:30:52.3273302Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2024-12-18T01:30:52.3274538Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex 2024-12-18T01:30:52.3275818Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda 2024-12-18T01:30:52.3277003Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2024-12-18T01:30:52.3278029Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group 2024-12-18T01:30:52.3279171Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params 2024-12-18T01:30:52.3280327Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer 2024-12-18T01:30:52.3281754Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none 2024-12-18T01:30:52.3283267Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2024-12-18T01:30:52.3284713Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler 2024-12-18T01:30:52.3285919Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception 2024-12-18T01:30:52.3287166Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2024-12-18T01:30:52.3288380Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout 2024-12-18T01:30:52.3289554Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration 2024-12-18T01:30:52.3290717Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2024-12-18T01:30:52.3291850Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max 2024-12-18T01:30:52.3292892Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum 2024-12-18T01:30:52.3293899Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group 2024-12-18T01:30:52.3294926Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum 2024-12-18T01:30:52.3295991Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp 2024-12-18T01:30:52.3297010Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged 2024-12-18T01:30:52.3297580Z 2024-12-18T01:30:52.3297850Z Running distributed/checkpoint/test_fsspec 1/1 ... [2024-12-18 01:30:52.324278] 2024-12-18T01:30:52.3298367Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:30:52.3299840Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_fsspec.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:30:52.324812] 2024-12-18T01:31:08.3810856Z 2024-12-18T01:31:08.3812101Z distributed/checkpoint/test_fsspec 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsspec_1.1_1967e82aa101dc3f_.log 2024-12-18T01:31:08.3813865Z Running 2 items in this shard: test/distributed/checkpoint/test_fsspec.py::TestFSSpec::test_fsspec, test/distributed/checkpoint/test_fsspec.py::TestFSSpec::test_overwrite 2024-12-18T01:31:08.3814753Z 2024-12-18T01:31:08.3818870Z Running distributed/tensor/parallel/test_tp_style 1/1 ... [2024-12-18 01:31:08.381276] 2024-12-18T01:31:08.3819646Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:31:08.3821030Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_style.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:31:08.381725] 2024-12-18T01:32:14.3190816Z 2024-12-18T01:32:14.3192238Z distributed/tensor/parallel/test_tp_style 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_style_1.1_d193ded39d05106a_.log 2024-12-18T01:32:14.3198542Z Running 9 items in this shard: test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_colwise_parallel_embedding, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_colwise_parallel_style, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_input, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_input_multiple_inputs, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_kwargs_input, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_output, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_rowwise_parallel_embedding, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_rowwise_parallel_style, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_sequence_parallel_style 2024-12-18T01:32:14.3203633Z 2024-12-18T01:32:14.3204024Z Running distributed/checkpoint/e2e/test_e2e_save_and_load 1/1 ... [2024-12-18 01:32:14.318920] 2024-12-18T01:32:14.3204743Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:32:14.3206039Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/e2e/test_e2e_save_and_load.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:32:14.319293] 2024-12-18T01:34:17.8074273Z 2024-12-18T01:34:17.8075562Z distributed/checkpoint/e2e/test_e2e_save_and_load 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.e2e.test_e2e_save_and_load_1.1_bda595a177166fc0_.log 2024-12-18T01:34:17.8084696Z Running 15 items in this shard: test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_different_ordered_state_dict_keys, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_async_cached_cache_staged_state_dict_False, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_async_cached_cache_staged_state_dict_True, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_compile_False_model_type0, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_compile_False_model_type1, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_compile_False_model_type2, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_compile_True_model_type0, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_compile_True_model_type1, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_compile_True_model_type2, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_no_dist, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_overwrite, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_partial_load, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_stateful_and_non_stateful_loads, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestNoCPU::test_no_cpu, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestInitStateDict::test_init_state_dict 2024-12-18T01:34:17.8093352Z 2024-12-18T01:34:17.8093715Z Running distributed/tensor/parallel/test_tp_examples 1/1 ... [2024-12-18 01:34:17.807538] 2024-12-18T01:34:17.8094334Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:34:17.8095657Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_examples.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:34:17.807911] 2024-12-18T01:36:17.8852870Z 2024-12-18T01:36:17.8854151Z distributed/tensor/parallel/test_tp_examples 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_examples_1.1_161991ee1f835690_.log 2024-12-18T01:36:17.8868144Z Running 16 items in this shard: test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_loss_parallel, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_inference, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_training_is_seq_parallel_False_recompute_activation_False, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_training_is_seq_parallel_True_recompute_activation_False, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_float64_thaw_all, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_all, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_layers_0_attention_wv__layers_0_feed_forward_w1__layers_1_feed_forward_w2__layers_1_ffn_norm__output__tok_embeddings, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_layers_1_ffn_norm__norm__output__tok_embeddings, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_norm__output, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_norm__output__tok_embeddings, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_output__tok_embeddings, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_False_float32, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_False_float64, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_True_float32, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_True_float64, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_weight_tying 2024-12-18T01:36:17.8880575Z 2024-12-18T01:36:17.8880955Z Running distributed/_composable/fsdp/test_fully_shard_state 1/1 ... [2024-12-18 01:36:17.885311] 2024-12-18T01:36:17.8881588Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:36:17.8882923Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:36:17.885673] 2024-12-18T01:36:22.5622101Z 2024-12-18T01:36:22.5623533Z distributed/_composable/fsdp/test_fully_shard_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_state_1.1_3fded1c04c318bf8_.log 2024-12-18T01:36:22.5627623Z Running 5 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_cls, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_deepcopy, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_reapply, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_state, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_unsupported_module_cls 2024-12-18T01:36:22.5630419Z 2024-12-18T01:36:22.5630728Z Running distributed/test_control_collectives 1/1 ... [2024-12-18 01:36:22.562279] 2024-12-18T01:36:22.5631294Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:36:22.5632856Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_control_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:36:22.562666] 2024-12-18T01:36:26.9381850Z 2024-12-18T01:36:26.9383204Z distributed/test_control_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_control_collectives_1.1_a6881c3ddab1d34c_.log 2024-12-18T01:36:26.9389791Z Running 13 items in this shard: test/distributed/test_control_collectives.py::TestCollectives::test_all_gather_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_all_sum, test/distributed/test_control_collectives.py::TestCollectives::test_all_sum_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_barrier, test/distributed/test_control_collectives.py::TestCollectives::test_barrier_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_broadcast, test/distributed/test_control_collectives.py::TestCollectives::test_broadcast_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_gather, test/distributed/test_control_collectives.py::TestCollectives::test_gather_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_scatter, test/distributed/test_control_collectives.py::TestCollectives::test_scatter_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_simple_user_func, test/distributed/test_control_collectives.py::TestCollectives::test_unique 2024-12-18T01:36:26.9395324Z 2024-12-18T01:36:26.9395637Z Running distributed/_tensor/debug/test_comm_mode 1/1 ... [2024-12-18 01:36:26.938316] 2024-12-18T01:36:26.9396223Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:36:26.9397513Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/debug/test_comm_mode.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:36:26.938686] 2024-12-18T01:36:31.8155631Z 2024-12-18T01:36:31.8157069Z distributed/_tensor/debug/test_comm_mode 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.debug.test_comm_mode_1.1_b90474a629a5b2cc_.log 2024-12-18T01:36:31.8160382Z Running 4 items in this shard: test/distributed/_tensor/debug/test_comm_mode.py::TestCommMode::test_comm_mode, test/distributed/_tensor/debug/test_comm_mode.py::TestCommMode::test_comm_mode_coalesced, test/distributed/_tensor/debug/test_comm_mode.py::TestCommMode::test_comm_mode_with_c10d, test/distributed/_tensor/debug/test_comm_mode.py::TestCommMode::test_comm_mode_with_dtensor 2024-12-18T01:36:31.8162376Z 2024-12-18T01:36:31.8162711Z Running distributed/_composable/test_replicate 1/1 ... [2024-12-18 01:36:31.815290] 2024-12-18T01:36:31.8163291Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:36:31.8164651Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_replicate.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:36:31.815681] 2024-12-18T01:37:58.7012096Z 2024-12-18T01:37:58.7013467Z distributed/_composable/test_replicate 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_replicate_1.1_326cf4636ea1e768_.log 2024-12-18T01:37:58.7023794Z Running 17 items in this shard: test/distributed/_composable/test_replicate.py::ReplicateStateDictTest::test_replicate_non_root_multiple_save_load, test/distributed/_composable/test_replicate.py::ReplicateStateDictTest::test_replicate_single_module_save_load, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_device_id, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_ignore_module, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_move_args_kwargs_to_device, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_multi_module, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_single_module, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_with_kwargs, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_wrong_device_id_type, test/distributed/_composable/test_replicate.py::ReplicateFullyShardInit::test_replicate_device_id, test/distributed/_composable/test_replicate.py::ReplicateFullyShardInit::test_replicate_fully_shard_init, test/distributed/_composable/test_replicate.py::ReplicateFullyShardInit::test_replicate_ignore_module, test/distributed/_composable/test_replicate.py::ReplicateFullyShardInit::test_replicate_move_args_kwargs_to_device, test/distributed/_composable/test_replicate.py::ReplicateFullyShardInit::test_replicate_multi_module, test/distributed/_composable/test_replicate.py::ReplicateFullyShardInit::test_replicate_single_module, test/distributed/_composable/test_replicate.py::ReplicateFullyShardInit::test_replicate_with_kwargs, test/distributed/_composable/test_replicate.py::ReplicateFullyShardInit::test_replicate_wrong_device_id_type 2024-12-18T01:37:58.7032764Z 2024-12-18T01:37:58.7033167Z Running distributed/optim/test_apply_optimizer_in_backward 1/1 ... [2024-12-18 01:37:58.701136] 2024-12-18T01:37:58.7033926Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:37:58.7035258Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/optim/test_apply_optimizer_in_backward.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:37:58.701508] 2024-12-18T01:38:01.6657454Z 2024-12-18T01:38:01.6658918Z distributed/optim/test_apply_optimizer_in_backward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_apply_optimizer_in_backward_1.1_70a212ff41a5eac3_.log 2024-12-18T01:38:01.6660167Z 2024-12-18T01:38:01.6660544Z Running distributed/fsdp/test_fsdp_state_dict 2/2 ... [2024-12-18 01:38:01.665833] 2024-12-18T01:38:01.6661124Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:38:01.6664153Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_state_dict.py', '--shard-id=2', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:38:01.666228] 2024-12-18T01:47:44.1204720Z 2024-12-18T01:47:44.1205903Z distributed/fsdp/test_fsdp_state_dict 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_state_dict_2.2_15f9c0c5290fadb7_.log 2024-12-18T01:47:44.1296697Z Running 101 items in this shard: test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_keys_state_dict_type_sharded_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_local_state_dict_with_empty_ranks, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_sharded_load_multi_backend_pg, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_shared_module_and_shared_parameter, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_rank0_offload_save_load_flow_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_save_load_flow_state_dict_type_local_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_save_load_flow_state_dict_type_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_skip_module_state_dict_type_local_state_dict_double_nest_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_skip_module_state_dict_type_sharded_state_dict_double_nest_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_skip_module_state_dict_type_state_dict_double_nest_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_shared_parameters_state_dict_type_local_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_shared_parameters_state_dict_type_sharded_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_world_size_one, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_wrong_state_dict_config 2024-12-18T01:47:44.1385880Z 2024-12-18T01:47:44.1386147Z Running distributed/test_c10d_nccl 3/3 ... [2024-12-18 01:47:44.121761] 2024-12-18T01:47:44.1386679Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:47:44.1388003Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=3', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:47:44.122168] 2024-12-18T01:58:58.3724106Z 2024-12-18T01:58:58.3725214Z distributed/test_c10d_nccl 3/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_3.3_1a50706c964e8d56_.log 2024-12-18T01:58:58.3757774Z Running 64 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLInitTest::test_init_wo_backend_str, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_mixed_empty_pgs, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_eager_init_subgroup, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extra_cuda_context, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_get_uid, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float32, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nccl_dist_backend_error, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_with_eager_init, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_restart_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_subgroup_p2p_eager_init_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_channels_last_contig, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output_unused_param, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_packed_sequence, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_failure_recovery, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_info, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_ids_not_allowed, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_blocking_wait_with_barrier, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_abort, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_timeout, test/distributed/test_c10d_nccl.py::NcclUserBufferRegistrationTest::test_nccl_user_buffer_registration, test/distributed/test_c10d_nccl.py::CommTest::test_intra_node_comm_all_reduce, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids_function_argument, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_detail, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_info, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced_float8_errors, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_default_pg_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_complex, test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_mismatch, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e4m3fn, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_duplicated_pg, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_sanity_check, test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_False_async_op_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_barrier_profiling, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_True 2024-12-18T01:58:58.3789245Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLInitTest::test_init_wo_backend_str 2024-12-18T01:58:58.3790411Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_mixed_empty_pgs 2024-12-18T01:58:58.3791833Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_eager_init_subgroup 2024-12-18T01:58:58.3792885Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extra_cuda_context 2024-12-18T01:58:58.3793849Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_get_uid 2024-12-18T01:58:58.3794866Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float32 2024-12-18T01:58:58.3795903Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nccl_dist_backend_error 2024-12-18T01:58:58.3796991Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_with_eager_init 2024-12-18T01:58:58.3798134Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_restart_pg 2024-12-18T01:58:58.3799128Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend_nccl 2024-12-18T01:58:58.3800211Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_subgroup_p2p_eager_init_False 2024-12-18T01:58:58.3801344Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value_grad_is_view 2024-12-18T01:58:58.3802472Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl 2024-12-18T01:58:58.3803494Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_channels_last_contig 2024-12-18T01:58:58.3804534Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output_unused_param 2024-12-18T01:58:58.3805895Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True 2024-12-18T01:58:58.3807292Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False 2024-12-18T01:58:58.3808705Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True 2024-12-18T01:58:58.3810043Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True 2024-12-18T01:58:58.3811307Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing 2024-12-18T01:58:58.3812601Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True 2024-12-18T01:58:58.3813944Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True 2024-12-18T01:58:58.3815204Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl 2024-12-18T01:58:58.3816320Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_packed_sequence 2024-12-18T01:58:58.3817352Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_failure_recovery 2024-12-18T01:58:58.3818537Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_info 2024-12-18T01:58:58.3820103Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_is_view 2024-12-18T01:58:58.3821423Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward_grad_is_view 2024-12-18T01:58:58.3823021Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_ids_not_allowed 2024-12-18T01:58:58.3824286Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input 2024-12-18T01:58:58.3825639Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input 2024-12-18T01:58:58.3826855Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_blocking_wait_with_barrier 2024-12-18T01:58:58.3827989Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_abort 2024-12-18T01:58:58.3829039Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_timeout 2024-12-18T01:58:58.3830170Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclUserBufferRegistrationTest::test_nccl_user_buffer_registration 2024-12-18T01:58:58.3831296Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_intra_node_comm_all_reduce 2024-12-18T01:58:58.3832324Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids_function_argument 2024-12-18T01:58:58.3833408Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_detail 2024-12-18T01:58:58.3834508Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_info 2024-12-18T01:58:58.3835479Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced_float8_errors 2024-12-18T01:58:58.3836445Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_default_pg_nccl 2024-12-18T01:58:58.3837322Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_complex 2024-12-18T01:58:58.3838227Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_mismatch 2024-12-18T01:58:58.3839233Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single 2024-12-18T01:58:58.3840494Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e4m3fn 2024-12-18T01:58:58.3841763Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced 2024-12-18T01:58:58.3843060Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2024-12-18T01:58:58.3844338Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1_group_rank_False 2024-12-18T01:58:58.3845443Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup_group_rank_True 2024-12-18T01:58:58.3846436Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_duplicated_pg 2024-12-18T01:58:58.3847411Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_sanity_check 2024-12-18T01:58:58.3848351Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup_group_rank_True 2024-12-18T01:58:58.3849301Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup_group_rank_False 2024-12-18T01:58:58.3850334Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_False_async_op_True 2024-12-18T01:58:58.3851306Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_barrier_profiling 2024-12-18T01:58:58.3852326Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_True 2024-12-18T01:58:58.3853484Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_False 2024-12-18T01:58:58.3854669Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_True 2024-12-18T01:58:58.3855770Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_False 2024-12-18T01:58:58.3856919Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_False 2024-12-18T01:58:58.3858083Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_True 2024-12-18T01:58:58.3859519Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_True 2024-12-18T01:58:58.3860737Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_False 2024-12-18T01:58:58.3861869Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_True 2024-12-18T01:58:58.3862493Z 2024-12-18T01:58:58.3862873Z Running distributed/_composable/fsdp/test_fully_shard_comm 1/1 ... [2024-12-18 01:58:58.374034] 2024-12-18T01:58:58.3863509Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:58:58.3864928Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_comm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:58:58.374408] 2024-12-18T02:01:03.7768208Z 2024-12-18T02:01:03.7770508Z distributed/_composable/fsdp/test_fully_shard_comm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_comm_1.1_7cea8c95aad49f2f_.log 2024-12-18T02:01:03.7805503Z Running 15 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_all_gather_fp32, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_reduce_scatter_fp16, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_reduce_scatter_fp32, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_fully_shard_communication_count, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_manual_reshard_with_reshard_after_forward_false, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_set_reduce_scatter_divide_factor, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_backward_misprefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_fully_shard_backward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_fully_shard_multi_module_backward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_fully_shard_multi_module_unused_module, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_set_modules_to_backward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_set_modules_to_forward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardUnshardMultiProcess::test_unshard_async, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardUnshardMultiThread::test_unshard_no_param_group, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardUnshardMultiThread::test_unshard_without_lazy_init 2024-12-18T02:01:03.7815367Z 2024-12-18T02:01:03.7815804Z Running distributed/nn/jit/test_instantiator 1/1 ... [2024-12-18 02:01:03.779249] 2024-12-18T02:01:03.7818242Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:01:03.7819791Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/nn/jit/test_instantiator.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:01:03.779799] 2024-12-18T02:01:08.1061152Z 2024-12-18T02:01:08.1062746Z distributed/nn/jit/test_instantiator 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.nn.jit.test_instantiator_1.1_e3de813d6dce6007_.log 2024-12-18T02:01:08.1065800Z Running 3 items in this shard: test/distributed/nn/jit/test_instantiator.py::TestInstantiator::test_get_arg_return_types_from_interface, test/distributed/nn/jit/test_instantiator.py::TestInstantiator::test_instantiate_non_scripted_remote_module_template, test/distributed/nn/jit/test_instantiator.py::TestInstantiator::test_instantiate_scripted_remote_module_template 2024-12-18T02:01:08.1067796Z 2024-12-18T02:01:08.1068784Z Running distributed/fsdp/test_fsdp_backward_prefetch 1/1 ... [2024-12-18 02:01:08.106220] 2024-12-18T02:01:08.1069398Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:01:08.1070716Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_backward_prefetch.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:01:08.106654] 2024-12-18T02:01:19.5001457Z 2024-12-18T02:01:19.5003857Z distributed/fsdp/test_fsdp_backward_prefetch 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_backward_prefetch_1.1_1bd477d2a4bcab10_.log 2024-12-18T02:01:19.5006345Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_backward_prefetch.py::TestBackwardPrefetch::test_backward_prefetch 2024-12-18T02:01:19.5007417Z 2024-12-18T02:01:19.5008072Z Running distributed/fsdp/test_fsdp_pure_fp16 1/1 ... [2024-12-18 02:01:19.500125] 2024-12-18T02:01:19.5009095Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:01:19.5013413Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_pure_fp16.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:01:19.500665] 2024-12-18T02:01:45.1293394Z 2024-12-18T02:01:45.1294692Z distributed/fsdp/test_fsdp_pure_fp16 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_pure_fp16_1.1_db41536cef1a8abe_.log 2024-12-18T02:01:45.1296600Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_pure_fp16.py::TestPureFP16CUDA::test_fp16_dtypes_cuda, test/distributed/fsdp/test_fsdp_pure_fp16.py::TestPureFP16CUDA::test_pure_fp16_training_cuda 2024-12-18T02:01:45.1297681Z 2024-12-18T02:01:45.1297996Z Running distributed/test_c10d_object_collectives 1/1 ... [2024-12-18 02:01:45.129277] 2024-12-18T02:01:45.1298572Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:01:45.1300167Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_object_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:01:45.129656] 2024-12-18T02:02:50.9186910Z 2024-12-18T02:02:50.9188159Z distributed/test_c10d_object_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_object_collectives_1.1_d8c25a97bcc0c130_.log 2024-12-18T02:02:50.9194880Z Running 9 items in this shard: test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_all_gather_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_broadcast_object_list, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_gather_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_scatter_object_list, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_send_recv_object_list, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_subpg_all_gather_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_subpg_broadcast_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_subpg_gather_object, test/distributed/test_c10d_object_collectives.py::TestObjectCollectives::test_subpg_scatter_object 2024-12-18T02:02:50.9199749Z 2024-12-18T02:02:50.9200003Z Running distributed/test_c10d_pypg 1/1 ... [2024-12-18 02:02:50.918742] 2024-12-18T02:02:50.9200503Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:02:50.9201691Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_pypg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:02:50.919114] 2024-12-18T02:02:57.7502712Z 2024-12-18T02:02:57.7503828Z distributed/test_c10d_pypg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_pypg_1.1_4f4bb7400b0f840c_.log 2024-12-18T02:02:57.7526650Z Running 43 items in this shard: test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_dataclass_output, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_dataclass_output_unused_param, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_invoke_work_object, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_with_pypg, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_with_pypg_with_grad_views, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_invalid_powerSGD_state, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_sync_batch_norm_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_dataclass_output, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_dataclass_output_unused_param, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_invoke_work_object, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_with_pypg, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_with_pypg_with_grad_views, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_invalid_powerSGD_state, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_sync_batch_norm_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_pypg.py::TestPyProcessGroup::test_attr_overrides 2024-12-18T02:02:57.7548705Z 2024-12-18T02:02:57.7549031Z Running distributed/fsdp/test_fsdp_freezing_weights 1/1 ... [2024-12-18 02:02:57.750518] 2024-12-18T02:02:57.7549615Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:02:57.7550899Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_freezing_weights.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:02:57.750896] 2024-12-18T02:06:40.0870096Z 2024-12-18T02:06:40.0871355Z distributed/fsdp/test_fsdp_freezing_weights 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_freezing_weights_1.1_7ce3ffff2ed59e82_.log 2024-12-18T02:06:40.0909994Z Running 32 items in this shard: test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_True 2024-12-18T02:06:40.0946950Z 2024-12-18T02:06:40.0947357Z Running distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1 ... [2024-12-18 02:06:40.087281] 2024-12-18T02:06:40.0948023Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:06:40.0949419Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:06:40.087654] 2024-12-18T02:07:20.2041753Z 2024-12-18T02:07:20.2043291Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks_1.1_5c9f7d5f923de9e9_.log 2024-12-18T02:07:20.2049063Z Running 6 items in this shard: test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_allreduce_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_fp16compress_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_noop_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_quantize_per_channel_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_quantize_per_tensor_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_is_last_hook 2024-12-18T02:07:20.2053241Z 2024-12-18T02:07:20.2053558Z Running distributed/fsdp/test_fsdp_exec_order 1/1 ... [2024-12-18 02:07:20.204230] 2024-12-18T02:07:20.2054121Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:07:20.2055394Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_exec_order.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:07:20.204607] 2024-12-18T02:08:46.5845417Z 2024-12-18T02:08:46.5846630Z distributed/fsdp/test_fsdp_exec_order 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_exec_order_1.1_97e5115fb10e78d2_.log 2024-12-18T02:08:46.5852702Z Running 8 items in this shard: test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrderCUDA::test_invalid_first_iter_order_sharding_strategy0_cuda, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrderCUDA::test_invalid_first_iter_order_sharding_strategy1_cuda, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrderCUDA::test_invalid_later_iter_order_sharding_strategy0_iters_before_path_change_1_cuda, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrderCUDA::test_invalid_later_iter_order_sharding_strategy0_iters_before_path_change_3_cuda, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrderCUDA::test_invalid_later_iter_order_sharding_strategy1_iters_before_path_change_1_cuda, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrderCUDA::test_invalid_later_iter_order_sharding_strategy1_iters_before_path_change_3_cuda, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrderCUDA::test_train_eval_sharding_strategy0_cuda, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrderCUDA::test_train_eval_sharding_strategy1_cuda 2024-12-18T02:08:46.5858205Z 2024-12-18T02:08:46.5858498Z Running distributed/fsdp/test_fsdp_grad_acc 1/1 ... [2024-12-18 02:08:46.584815] 2024-12-18T02:08:46.5859337Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:08:46.5860969Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_grad_acc.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:08:46.585183] 2024-12-18T02:09:35.5240650Z 2024-12-18T02:09:35.5242082Z distributed/fsdp/test_fsdp_grad_acc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_grad_acc_1.1_a7183b063ff1fe6c_.log 2024-12-18T02:09:35.5246084Z Running 6 items in this shard: test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs0_use_orig_params_False, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs0_use_orig_params_True, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs1_use_orig_params_False, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs1_use_orig_params_True, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_cpu_offload_use_orig_params_False, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_cpu_offload_use_orig_params_True 2024-12-18T02:09:35.5249143Z 2024-12-18T02:09:35.5249480Z Running distributed/fsdp/test_fsdp_ignored_modules 1/1 ... [2024-12-18 02:09:35.524198] 2024-12-18T02:09:35.5250071Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:09:35.5251381Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_ignored_modules.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:09:35.524579] 2024-12-18T02:10:26.0715925Z 2024-12-18T02:10:26.0717173Z distributed/fsdp/test_fsdp_ignored_modules 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_ignored_modules_1.1_ee345141a32e10d3_.log 2024-12-18T02:10:26.0723793Z Running 8 items in this shard: test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_diff_ignored_modules_across_ranks, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_invalid, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_nested, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_not_under_wrapped_root_ignore_modules_False, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_not_under_wrapped_root_ignore_modules_True, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_transformer, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_states_auto_wrap, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_states_check 2024-12-18T02:10:26.0729021Z 2024-12-18T02:10:26.0729488Z Running distributed/fsdp/test_fsdp_multiple_wrapping 1/1 ... [2024-12-18 02:10:26.071703] 2024-12-18T02:10:26.0730078Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:10:26.0731362Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_wrapping.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:10:26.072090] 2024-12-18T02:10:41.1757161Z 2024-12-18T02:10:41.1758477Z distributed/fsdp/test_fsdp_multiple_wrapping 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_multiple_wrapping_1.1_a8564f6030a24b15_.log 2024-12-18T02:10:41.1760157Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_multiple_wrapping.py::TestMultipleWrappingCUDA::test_multiple_wrapping_cuda 2024-12-18T02:10:41.1760884Z 2024-12-18T02:10:41.1761662Z Running distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 ... [2024-12-18 02:10:41.175662] 2024-12-18T02:10:41.1762435Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:10:41.1763825Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_mixed_precision.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:10:41.176027] 2024-12-18T02:11:16.2818169Z 2024-12-18T02:11:16.2820046Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_25f44a3f728dc1fe_.log 2024-12-18T02:11:16.2826407Z Running 7 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_compute_dtype, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_grad_acc_with_reduce_dtype, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_reduce_dtype, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_float16_on_one_submodule, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_norm_modules_bf16, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_norm_modules_fp16, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_submodules_with_external_inputs 2024-12-18T02:11:16.2831369Z 2024-12-18T02:11:16.2831833Z Running distributed/_composable/test_composability/test_2d_composability 1/1 ... [2024-12-18 02:11:16.281612] 2024-12-18T02:11:16.2832518Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:11:16.2833938Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_composability/test_2d_composability.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:11:16.282017] 2024-12-18T02:14:08.8966473Z 2024-12-18T02:14:08.8967945Z distributed/_composable/test_composability/test_2d_composability 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_composability.test_2d_composability_1.1_77c0c2692f2612dc_.log 2024-12-18T02:14:08.8983882Z Running 20 items in this shard: test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DTraining::test_2d_uneven_shard_raise_error, test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DTraining::test_tp_with_fsdp_offloading, test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DTraining::test_train_parity_2d_mlp, test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DTraining::test_train_parity_2d_transformer, test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DTraining::test_train_parity_2d_transformer_checkpoint_resume, test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DStateDict::test_fully_shard_tp_2d_set_full_state_dict, test/distributed/_composable/test_composability/test_2d_composability.py::Test2dFSDP1ParallelIntegration::test_2d_ddp_integration_functionality, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelTraining::test_2d_e2e_training_default, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelTraining::test_2d_e2e_training_not_use_orig_params, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelTraining::test_2d_e2e_training_use_orig_params, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelTraining::test_2d_fsdp_state_enable_extension, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelTraining::test_raise_invalid_tp_composition, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_load_state_dict_is_even_sharded_model_False, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_load_state_dict_is_even_sharded_model_True, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_optim_state_dict_is_even_sharded_model_False, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_optim_state_dict_is_even_sharded_model_True, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_state_dict_is_even_sharded_model_False, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_state_dict_is_even_sharded_model_True, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_fsdp1_tp_2d_set_full_state_dict, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_fsdp_2d_extension 2024-12-18T02:14:08.8997713Z 2024-12-18T02:14:08.8998019Z Running distributed/fsdp/test_fsdp_comm_hooks 1/1 ... [2024-12-18 02:14:08.896731] 2024-12-18T02:14:08.8998568Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:14:08.8999806Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm_hooks.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:14:08.897098] 2024-12-18T02:17:22.5670500Z 2024-12-18T02:17:22.5671715Z distributed/fsdp/test_fsdp_comm_hooks 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_comm_hooks_1.1_aa579ddb4e97d7b8_.log 2024-12-18T02:17:22.5690616Z Running 28 items in this shard: test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_False_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_True_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_behavior_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_behavior_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_behavior_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_False_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_True_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_hybrid_strategy, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_non_root_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_non_root_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_non_root_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_submodules_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_submodules_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_submodules_sharding_strategy2 2024-12-18T02:17:22.5709553Z 2024-12-18T02:17:22.5709844Z Running distributed/fsdp/test_fsdp_core 2/2 ... [2024-12-18 02:17:22.567253] 2024-12-18T02:17:22.5710365Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:17:22.5711580Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_core.py', '--shard-id=2', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:17:22.567633] 2024-12-18T02:25:43.1742019Z 2024-12-18T02:25:43.1743307Z distributed/fsdp/test_fsdp_core 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_core_2.2_a81ab11c8b7f6336_.log 2024-12-18T02:25:43.1760063Z Running 27 items in this shard: test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_pre_backward_hook_registration_after_state_dict_cuda, test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_pre_backward_hook_registration_cuda_first_False_cuda, test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_register_functions_called_cuda_first_True_mixed_precision_True_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_true_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestNoGradCUDA::test_transformer_no_grad_mixed_precision_False_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParamInitCUDA::test_param_change_after_init_mixed_precision_False_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParamInitCUDA::test_param_change_after_init_mixed_precision_True_cuda, test/distributed/fsdp/test_fsdp_core.py::TestAutogradCUDA::test_unshard_params_as_tensors_cuda 2024-12-18T02:25:43.1775611Z 2024-12-18T02:25:43.1775886Z Running distributed/_shard/test_sharder 1/1 ... [2024-12-18 02:25:43.175209] 2024-12-18T02:25:43.1776392Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:25:43.1777603Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/test_sharder.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:25:43.175582] 2024-12-18T02:26:00.9873204Z 2024-12-18T02:26:00.9875053Z distributed/_shard/test_sharder 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.test_sharder_1.1_525d4c73ec376388_.log 2024-12-18T02:26:00.9878019Z Running 2 items in this shard: test/distributed/_shard/test_sharder.py::TestCustomSharder::test_custom_sharder, test/distributed/_shard/test_sharder.py::TestCustomSharder::test_custom_sharder_errors 2024-12-18T02:26:00.9879612Z 2024-12-18T02:26:00.9880060Z Running distributed/fsdp/test_fsdp_comm 1/1 ... [2024-12-18 02:26:00.987540] 2024-12-18T02:26:00.9880999Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:26:00.9887699Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:26:00.988126] 2024-12-18T02:27:43.2140075Z 2024-12-18T02:27:43.2141604Z distributed/fsdp/test_fsdp_comm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_comm_1.1_4cee44d5f80cf3f7_.log 2024-12-18T02:27:43.2150903Z Running 10 items in this shard: test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_False_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_True_cuda 2024-12-18T02:27:43.2157762Z 2024-12-18T02:27:43.2158135Z Running distributed/fsdp/test_distributed_checkpoint 1/1 ... [2024-12-18 02:27:43.214075] 2024-12-18T02:27:43.2158730Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:27:43.2160056Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_distributed_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:27:43.214578] 2024-12-18T02:33:02.8714054Z 2024-12-18T02:33:02.8715319Z PRINTING LOG FILE of distributed/fsdp/test_distributed_checkpoint 1/1 (test/test-reports/distributed.fsdp.test_distributed_checkpoint_1.1_a59c6f4c0a14e29b_.log) 2024-12-18T02:33:02.8716937Z Test results will be stored in test-reports/python-pytest/distributed.fsdp.test_distributed_checkpoint/distributed.fsdp.test_distributed_checkpoint-91d946e08ae42faa.xml 2024-12-18T02:33:02.8717997Z ============================= test session starts ============================== 2024-12-18T02:33:02.8718754Z platform linux -- Python 3.10.15, pytest-7.3.2, pluggy-1.5.0 -- /opt/conda/envs/py_3.10/bin/python 2024-12-18T02:33:02.8719408Z cachedir: .pytest_cache 2024-12-18T02:33:02.8720120Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2024-12-18T02:33:02.8721254Z rootdir: /var/lib/jenkins/workspace 2024-12-18T02:33:02.8721603Z configfile: pytest.ini 2024-12-18T02:33:02.8722680Z plugins: hypothesis-5.35.1, cpp-2.3.0, flakefinder-1.1.0, rerunfailures-14.0, subtests-0.13.1, xdist-3.3.1, xdoctest-1.1.0, typeguard-4.3.0 2024-12-18T02:33:02.8723511Z collecting ... collected 2 items 2024-12-18T02:33:02.8723937Z stepcurrent: Cannot find last run test, not skipping 2024-12-18T02:33:02.8725746Z Running 2 items in this shard: test/distributed/fsdp/test_distributed_checkpoint.py::TestDistributedCheckpointCUDA::test_distributed_checkpoint_state_dict_type0_cuda, test/distributed/fsdp/test_distributed_checkpoint.py::TestDistributedCheckpointCUDA::test_distributed_checkpoint_state_dict_type1_cuda 2024-12-18T02:33:02.8727370Z 2024-12-18T02:33:02.8729779Z distributed/fsdp/test_distributed_checkpoint.py::TestDistributedCheckpointCUDA::test_distributed_checkpoint_state_dict_type0_cuda /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:845: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2024-12-18T02:33:02.8732399Z warnings.warn( 2024-12-18T02:33:02.8736057Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:845: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2024-12-18T02:33:02.8737978Z warnings.warn( 2024-12-18T02:33:02.8740872Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html . 2024-12-18T02:33:02.8743600Z warnings.warn( 2024-12-18T02:33:02.8746267Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html . 2024-12-18T02:33:02.8748985Z warnings.warn( 2024-12-18T02:33:02.8750937Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:863: UserWarning: `_get_pg_default_device` will be deprecated, it only stays for backward-compatiblity reason. If you need to find a device for object collectives, please use `_get_object_coll_device`. If you need to query the device types supported by group, please use `_device_capability(group)`. 2024-12-18T02:33:02.8752920Z warnings.warn( 2024-12-18T02:33:02.8754702Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:863: UserWarning: `_get_pg_default_device` will be deprecated, it only stays for backward-compatiblity reason. If you need to find a device for object collectives, please use `_get_object_coll_device`. If you need to query the device types supported by group, please use `_device_capability(group)`. 2024-12-18T02:33:02.8756547Z warnings.warn( 2024-12-18T02:33:02.8757591Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_state_dict_utils.py:732: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8758610Z local_shape = tensor.shape 2024-12-18T02:33:02.8759587Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_state_dict_utils.py:749: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8760769Z tensor.shape, 2024-12-18T02:33:02.8761703Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_state_dict_utils.py:732: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8762837Z local_shape = tensor.shape 2024-12-18T02:33:02.8763812Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_state_dict_utils.py:751: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8764804Z tensor.dtype, 2024-12-18T02:33:02.8765725Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_state_dict_utils.py:749: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8766716Z tensor.shape, 2024-12-18T02:33:02.8767620Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_state_dict_utils.py:751: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8768610Z tensor.dtype, 2024-12-18T02:33:02.8769651Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/checkpoint/utils.py:411: UserWarning: The argument order of save has been changed. Please check the document to avoid future breakages. 2024-12-18T02:33:02.8770767Z warnings.warn( 2024-12-18T02:33:02.8771863Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/checkpoint/utils.py:411: UserWarning: The argument order of save has been changed. Please check the document to avoid future breakages. 2024-12-18T02:33:02.8772982Z warnings.warn( 2024-12-18T02:33:02.8774138Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/checkpoint/utils.py:411: UserWarning: The argument order of load has been changed. Please check the document to avoid future breakages. 2024-12-18T02:33:02.8775180Z warnings.warn( 2024-12-18T02:33:02.8776343Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/checkpoint/utils.py:411: UserWarning: The argument order of load has been changed. Please check the document to avoid future breakages. 2024-12-18T02:33:02.8777454Z warnings.warn( 2024-12-18T02:33:02.8778435Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/checkpoint/planner_helpers.py:316: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8779775Z device = getattr(value, "device", None) 2024-12-18T02:33:02.8780915Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/checkpoint/planner_helpers.py:316: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8782059Z device = getattr(value, "device", None) 2024-12-18T02:33:02.8783175Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/checkpoint/default_planner.py:362: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8784295Z and md.size != obj.size() 2024-12-18T02:33:02.8785376Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/checkpoint/default_planner.py:362: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8786493Z and md.size != obj.size() 2024-12-18T02:33:02.8787518Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_state_dict_utils.py:635: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8788597Z param_numel = param.size().numel() 2024-12-18T02:33:02.8789716Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_state_dict_utils.py:635: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8790790Z param_numel = param.size().numel() 2024-12-18T02:33:02.8792017Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_state_dict_utils.py:636: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8792966Z dim_0_size = param.size()[0] 2024-12-18T02:33:02.8793891Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_state_dict_utils.py:636: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8794845Z dim_0_size = param.size()[0] 2024-12-18T02:33:02.8795771Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_state_dict_utils.py:662: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8796807Z tensor = tensor.narrow(0, 0, param_numel).reshape(param.size()) 2024-12-18T02:33:02.8797854Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_state_dict_utils.py:662: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8798896Z tensor = tensor.narrow(0, 0, param_numel).reshape(param.size()) 2024-12-18T02:33:02.8799320Z dist init r=0, world=2 2024-12-18T02:33:02.8799616Z Using temp directory: /tmp/tmpjlf7dcyo 2024-12-18T02:33:02.8799948Z dist init r=1, world=2 2024-12-18T02:33:02.8800204Z PASSED [7.4835s] [ 50%] 2024-12-18T02:33:02.8802766Z distributed/fsdp/test_distributed_checkpoint.py::TestDistributedCheckpointCUDA::test_distributed_checkpoint_state_dict_type1_cuda /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:845: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2024-12-18T02:33:02.8805255Z warnings.warn( 2024-12-18T02:33:02.8807022Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:845: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2024-12-18T02:33:02.8808851Z warnings.warn( 2024-12-18T02:33:02.8811362Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html . 2024-12-18T02:33:02.8814249Z warnings.warn( 2024-12-18T02:33:02.8816724Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html . 2024-12-18T02:33:02.8819514Z warnings.warn( 2024-12-18T02:33:02.8820240Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] Process 0 timed out with traceback: 2024-12-18T02:33:02.8821321Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] 2024-12-18T02:33:02.8822649Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] Current thread 0x00007fa029360700 (most recent call first): 2024-12-18T02:33:02.8824427Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 689 in _event_listener 2024-12-18T02:33:02.8826250Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 953 in run 2024-12-18T02:33:02.8827855Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 1016 in _bootstrap_inner 2024-12-18T02:33:02.8829493Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 973 in _bootstrap 2024-12-18T02:33:02.8830719Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] 2024-12-18T02:33:02.8831886Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] Thread 0x00007fa166c754c0 (most recent call first): 2024-12-18T02:33:02.8833549Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2959 in _tensor_to_object 2024-12-18T02:33:02.8835686Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3052 in all_gather_object 2024-12-18T02:33:02.8837487Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 81 in wrapper 2024-12-18T02:33:02.8839345Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/_shard/sharded_tensor/api.py", line 742 in _init_from_local_shards 2024-12-18T02:33:02.8841345Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/_shard/sharded_tensor/__init__.py", line 407 in init_from_local_shards 2024-12-18T02:33:02.8843348Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_state_dict_utils.py", line 442 in _local_post_state_dict_hook 2024-12-18T02:33:02.8845296Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_state_dict_utils.py", line 724 in _post_state_dict_hook 2024-12-18T02:33:02.8847130Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116 in decorate_context 2024-12-18T02:33:02.8848828Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2222 in state_dict 2024-12-18T02:33:02.8850518Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2216 in state_dict 2024-12-18T02:33:02.8852206Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2216 in state_dict 2024-12-18T02:33:02.8853954Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2216 in state_dict 2024-12-18T02:33:02.8855828Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2216 in state_dict 2024-12-18T02:33:02.8857765Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_distributed_checkpoint.py", line 67 in test_distributed_checkpoint 2024-12-18T02:33:02.8860082Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/checkpoint_utils.py", line 44 in wrapper 2024-12-18T02:33:02.8862203Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 199 in wrapper 2024-12-18T02:33:02.8864313Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_device_type.py", line 460 in instantiated_test 2024-12-18T02:33:02.8866455Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3108 in wrapper 2024-12-18T02:33:02.8868457Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3108 in wrapper 2024-12-18T02:33:02.8870479Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 599 in wrapper 2024-12-18T02:33:02.8872621Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 726 in run_test 2024-12-18T02:33:02.8874405Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1233 in _run 2024-12-18T02:33:02.8876004Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/process.py", line 108 in run 2024-12-18T02:33:02.8877520Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/process.py", line 314 in _bootstrap 2024-12-18T02:33:02.8879037Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/spawn.py", line 129 in _main 2024-12-18T02:33:02.8880543Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/spawn.py", line 116 in spawn_main 2024-12-18T02:33:02.8881829Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "", line 1 in 2024-12-18T02:33:02.8882727Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] 2024-12-18T02:33:02.8883834Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] Process 1 timed out with traceback: 2024-12-18T02:33:02.8884842Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] 2024-12-18T02:33:02.8885882Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] Current thread 0x00007f7d19360700 (most recent call first): 2024-12-18T02:33:02.8887523Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 689 in _event_listener 2024-12-18T02:33:02.8889257Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 953 in run 2024-12-18T02:33:02.8890761Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 1016 in _bootstrap_inner 2024-12-18T02:33:02.8892291Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 973 in _bootstrap 2024-12-18T02:33:02.8893434Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] 2024-12-18T02:33:02.8894440Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] Thread 0x00007f7e55cc14c0 (most recent call first): 2024-12-18T02:33:02.8896051Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/cuda/__init__.py", line 967 in synchronize 2024-12-18T02:33:02.8897867Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_state_dict_utils.py", line 150 in _common_pre_state_dict_hook 2024-12-18T02:33:02.8900180Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_state_dict_utils.py", line 524 in _sharded_pre_state_dict_hook 2024-12-18T02:33:02.8902382Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_state_dict_utils.py", line 787 in _pre_state_dict_hook 2024-12-18T02:33:02.8904441Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116 in decorate_context 2024-12-18T02:33:02.8906355Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2212 in state_dict 2024-12-18T02:33:02.8908265Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2216 in state_dict 2024-12-18T02:33:02.8910165Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2216 in state_dict 2024-12-18T02:33:02.8912285Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_distributed_checkpoint.py", line 67 in test_distributed_checkpoint 2024-12-18T02:33:02.8914390Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/checkpoint_utils.py", line 44 in wrapper 2024-12-18T02:33:02.8916497Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 199 in wrapper 2024-12-18T02:33:02.8918531Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_device_type.py", line 460 in instantiated_test 2024-12-18T02:33:02.8920545Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3108 in wrapper 2024-12-18T02:33:02.8922629Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3108 in wrapper 2024-12-18T02:33:02.8924595Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 599 in wrapper 2024-12-18T02:33:02.8926581Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 726 in run_test 2024-12-18T02:33:02.8928527Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 1233 in _run 2024-12-18T02:33:02.8930390Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/process.py", line 108 in run 2024-12-18T02:33:02.8932045Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/process.py", line 314 in _bootstrap 2024-12-18T02:33:02.8933690Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/spawn.py", line 129 in _main 2024-12-18T02:33:02.8935330Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/spawn.py", line 116 in spawn_main 2024-12-18T02:33:02.8936739Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] File "", line 1 in 2024-12-18T02:33:02.8937727Z E1218 02:32:54.653000 225689 site-packages/torch/testing/_internal/common_distributed.py:783] 2024-12-18T02:33:02.8938342Z ('RERUN', {'yellow': True}) [300.0375s] [100%] 2024-12-18T02:33:02.8941210Z distributed/fsdp/test_distributed_checkpoint.py::TestDistributedCheckpointCUDA::test_distributed_checkpoint_state_dict_type1_cuda /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:845: UserWarning: FSDP got the argument `device_id` cuda on rank 0, which does not have an explicit index. FSDP will use the current device 0. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2024-12-18T02:33:02.8943825Z warnings.warn( 2024-12-18T02:33:02.8945705Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:845: UserWarning: FSDP got the argument `device_id` cuda on rank 1, which does not have an explicit index. FSDP will use the current device 1. If this is incorrect, please explicitly call `torch.cuda.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument. 2024-12-18T02:33:02.8947648Z warnings.warn( 2024-12-18T02:33:02.8950408Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html . 2024-12-18T02:33:02.8953129Z warnings.warn( 2024-12-18T02:33:02.8955475Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html . 2024-12-18T02:33:02.8957879Z warnings.warn( 2024-12-18T02:33:02.8958748Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_state_dict_utils.py:732: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8959710Z local_shape = tensor.shape 2024-12-18T02:33:02.8968447Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_state_dict_utils.py:732: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8969585Z local_shape = tensor.shape 2024-12-18T02:33:02.8970707Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_state_dict_utils.py:749: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8971715Z tensor.shape, 2024-12-18T02:33:02.8972639Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_state_dict_utils.py:751: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8973632Z tensor.dtype, 2024-12-18T02:33:02.8974553Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_state_dict_utils.py:749: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8975547Z tensor.shape, 2024-12-18T02:33:02.8976468Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/fsdp/_state_dict_utils.py:751: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8977466Z tensor.dtype, 2024-12-18T02:33:02.8978511Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/checkpoint/utils.py:411: UserWarning: The argument order of save has been changed. Please check the document to avoid future breakages. 2024-12-18T02:33:02.8980002Z warnings.warn( 2024-12-18T02:33:02.8981098Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/checkpoint/utils.py:411: UserWarning: The argument order of save has been changed. Please check the document to avoid future breakages. 2024-12-18T02:33:02.8982286Z warnings.warn( 2024-12-18T02:33:02.8983399Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/checkpoint/utils.py:411: UserWarning: The argument order of load has been changed. Please check the document to avoid future breakages. 2024-12-18T02:33:02.8984582Z warnings.warn( 2024-12-18T02:33:02.8985691Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/checkpoint/utils.py:411: UserWarning: The argument order of load has been changed. Please check the document to avoid future breakages. 2024-12-18T02:33:02.8986873Z warnings.warn( 2024-12-18T02:33:02.8987905Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/checkpoint/planner_helpers.py:316: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8989138Z device = getattr(value, "device", None) 2024-12-18T02:33:02.8990277Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/checkpoint/planner_helpers.py:316: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8991513Z device = getattr(value, "device", None) 2024-12-18T02:33:02.8992591Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/checkpoint/default_planner.py:362: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8993652Z and md.size != obj.size() 2024-12-18T02:33:02.8994674Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/checkpoint/default_planner.py:362: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor. 2024-12-18T02:33:02.8995720Z and md.size != obj.size() 2024-12-18T02:33:02.8996026Z dist init r=1, world=2 2024-12-18T02:33:02.8996310Z dist init r=0, world=2 2024-12-18T02:33:02.8996629Z Using temp directory: /tmp/tmpze3p80ba 2024-12-18T02:33:02.8996987Z PASSED [7.4177s] [100%] 2024-12-18T02:33:02.8997162Z 2024-12-18T02:33:02.8997307Z ==================================== RERUNS ==================================== 2024-12-18T02:33:02.8997933Z _ TestDistributedCheckpointCUDA.test_distributed_checkpoint_state_dict_type1_cuda _ 2024-12-18T02:33:02.8998634Z Traceback (most recent call last): 2024-12-18T02:33:02.8999349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 597, in wrapper 2024-12-18T02:33:02.9000070Z self._join_processes(fn) 2024-12-18T02:33:02.9000792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 837, in _join_processes 2024-12-18T02:33:02.9001652Z self._check_return_codes(elapsed_time) 2024-12-18T02:33:02.9002439Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 891, in _check_return_codes 2024-12-18T02:33:02.9003219Z raise RuntimeError( 2024-12-18T02:33:02.9003658Z RuntimeError: Process 0 terminated or timed out after 300.0336151123047 seconds 2024-12-18T02:33:02.9004508Z ----------------------------- Captured stdout call ----------------------------- 2024-12-18T02:33:02.9005038Z Timing out after 300 seconds and killing subprocesses. 2024-12-18T02:33:02.9006222Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.fsdp.test_distributed_checkpoint/distributed.fsdp.test_distributed_checkpoint-91d946e08ae42faa.xml - 2024-12-18T02:33:02.9007413Z ==================== 2 passed, 1 rerun in 315.17s (0:05:15) ==================== 2024-12-18T02:33:02.9007733Z 2024-12-18T02:33:02.9008444Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_distributed_checkpoint 1/1 (test/test-reports/distributed.fsdp.test_distributed_checkpoint_1.1_a59c6f4c0a14e29b_.log) 2024-12-18T02:33:02.9009292Z 2024-12-18T02:33:02.9009596Z Running distributed/fsdp/test_fsdp_use_orig_params 1/1 ... [2024-12-18 02:33:02.871794] 2024-12-18T02:33:02.9010167Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:33:02.9011420Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_use_orig_params.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:33:02.872197] 2024-12-18T02:38:06.3230637Z 2024-12-18T02:38:06.3231888Z distributed/fsdp/test_fsdp_use_orig_params 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_use_orig_params_1.1_d7c4eb0d244e68e4_.log 2024-12-18T02:38:06.3250822Z Running 25 items in this shard: test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_cpu_offload_sharding_strategy_str_full_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_cpu_offload_sharding_strategy_str_no_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_cpu_offload_sharding_strategy_str_shard_grad_op, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_sharding_strategy_str_full_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_sharding_strategy_str_no_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_sharding_strategy_str_shard_grad_op, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_trainability, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_fsdp_compile, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_multiple_optimizers, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_multiple_forward_offload_params_False, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_multiple_forward_offload_params_True, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_summon_between_two_forwards_offload_params_False, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_summon_between_two_forwards_offload_params_True, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsParamAccess::test_access_params_after_forward, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_grad_writeback, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_no_reshard_and_mixed_precision, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_param_writeback, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_writeback_between_fwd_and_bwd_for_no_reshard_raises, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_writeback_shape_mismatch, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsFQNs::test_named_parameters_in_forward, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsNoSync::test_no_sync_correctness, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsNoSync::test_no_sync_mixed_precision, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsInit::test_non_uniform_requires_grad, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestMultiTensorApply::test_multi_tensor_apply_size0_tensors_cpu, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestMultiTensorApply::test_multi_tensor_apply_size0_tensors_cuda 2024-12-18T02:38:06.3267849Z 2024-12-18T02:38:06.3268118Z Running distributed/test_c10d_gloo 1/1 ... [2024-12-18 02:38:06.323630] 2024-12-18T02:38:06.3268652Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:38:06.3269974Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_gloo.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:38:06.324005] 2024-12-18T02:56:17.0945128Z 2024-12-18T02:56:17.0946253Z distributed/test_c10d_gloo 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_gloo_1.1_37d1f0fa984c921f_.log 2024-12-18T02:56:17.1004138Z Running 121 items in this shard: test/distributed/test_c10d_gloo.py::RendezvousEnvTest::test_logging_init, test/distributed/test_c10d_gloo.py::TimeoutTest::test_default_store_timeout_gloo, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_async, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_inference_mode, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_into_tensor_coalesced, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_noncontiguous_input, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_async, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_barrier_implies_wait, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_empty_tensors, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_noncontiguous_input, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_multi_device_constructor, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_send_recv_all_to_all, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_set_gloo_pg_timeout, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_cuda_dispatched, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output_unused_param, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_cpu, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_gloo, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_register_just_once, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_sparse_gradients, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_init, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_return_type, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_find_unused_parameters_when_unused_parameters_empty, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_grad_is_view, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_static_graph, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_integer_list, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_torch_device_list, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_2gpu_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_4gpu_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module_grad_is_view, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output_with_unused_parameters, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_sharded_tensor, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_invalid_powerSGD_state, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_save_load_checkpoint, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients_grad_is_view, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward, test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_optimizer, test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_unused_parameters, test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_multi_bucket, test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_single_bucket, test/distributed/test_c10d_gloo.py::ReducerTest::test_single_dtype_single_bucket, test/distributed/test_c10d_gloo.py::CommTest::test_bool_tensors, test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cpu, test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cuda, test/distributed/test_c10d_gloo.py::CommTest::test_gloo_rank_membership, test/distributed/test_c10d_gloo.py::CommTest::test_gloo_warn_not_in_group, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_default, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_subgroup, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_default_pg_gloo, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_gloo_new_group, test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_complex, test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_mismatch, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allgather_coalesced, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_collectives, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_monitored_barrier, test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync, test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_duplicate_pg, test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_sanity_check 2024-12-18T02:56:17.1058191Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::RendezvousEnvTest::test_logging_init 2024-12-18T02:56:17.1059292Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::TimeoutTest::test_default_store_timeout_gloo 2024-12-18T02:56:17.1060310Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics 2024-12-18T02:56:17.1061340Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics_cuda 2024-12-18T02:56:17.1062372Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_checks 2024-12-18T02:56:17.1063427Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_async 2024-12-18T02:56:17.1064525Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_checks 2024-12-18T02:56:17.1065606Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_inference_mode 2024-12-18T02:56:17.1066729Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_into_tensor_coalesced 2024-12-18T02:56:17.1067996Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_noncontiguous_input 2024-12-18T02:56:17.1069068Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress 2024-12-18T02:56:17.1070097Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress_cuda 2024-12-18T02:56:17.1071227Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics 2024-12-18T02:56:17.1072311Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics_cuda 2024-12-18T02:56:17.1073265Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_checks 2024-12-18T02:56:17.1074253Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_async 2024-12-18T02:56:17.1075289Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_basics 2024-12-18T02:56:17.1076324Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks 2024-12-18T02:56:17.1077383Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks_cuda 2024-12-18T02:56:17.1078534Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_stress 2024-12-18T02:56:17.1079527Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress 2024-12-18T02:56:17.1080584Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress_cuda 2024-12-18T02:56:17.1081582Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_barrier_implies_wait 2024-12-18T02:56:17.1082555Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics 2024-12-18T02:56:17.1083528Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics_cuda 2024-12-18T02:56:17.1084500Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_checks 2024-12-18T02:56:17.1085450Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress 2024-12-18T02:56:17.1086422Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress_cuda 2024-12-18T02:56:17.1087371Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_empty_tensors 2024-12-18T02:56:17.1088296Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics 2024-12-18T02:56:17.1089416Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics_cuda 2024-12-18T02:56:17.1090394Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_checks 2024-12-18T02:56:17.1091498Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_noncontiguous_input 2024-12-18T02:56:17.1092480Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress 2024-12-18T02:56:17.1093425Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress_cuda 2024-12-18T02:56:17.1094404Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_multi_device_constructor 2024-12-18T02:56:17.1095372Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics 2024-12-18T02:56:17.1096315Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics_cuda 2024-12-18T02:56:17.1097320Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_checks 2024-12-18T02:56:17.1098283Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor 2024-12-18T02:56:17.1099598Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor_coalesced 2024-12-18T02:56:17.1100672Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress 2024-12-18T02:56:17.1101661Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress_cuda 2024-12-18T02:56:17.1102676Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics 2024-12-18T02:56:17.1103694Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics_cuda 2024-12-18T02:56:17.1104707Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_checks 2024-12-18T02:56:17.1105693Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress 2024-12-18T02:56:17.1106704Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress_cuda 2024-12-18T02:56:17.1107727Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_send_recv_all_to_all 2024-12-18T02:56:17.1108760Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_set_gloo_pg_timeout 2024-12-18T02:56:17.1109886Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics 2024-12-18T02:56:17.1111097Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics_cuda 2024-12-18T02:56:17.1112202Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_checks 2024-12-18T02:56:17.1113199Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_cuda_dispatched 2024-12-18T02:56:17.1114200Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output 2024-12-18T02:56:17.1115214Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output_unused_param 2024-12-18T02:56:17.1116324Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module 2024-12-18T02:56:17.1117492Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing 2024-12-18T02:56:17.1118706Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False 2024-12-18T02:56:17.1119918Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True 2024-12-18T02:56:17.1121377Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False 2024-12-18T02:56:17.1123144Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True 2024-12-18T02:56:17.1124570Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False 2024-12-18T02:56:17.1125932Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True 2024-12-18T02:56:17.1127379Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing 2024-12-18T02:56:17.1128758Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False 2024-12-18T02:56:17.1130196Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True 2024-12-18T02:56:17.1131639Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False 2024-12-18T02:56:17.1133094Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True 2024-12-18T02:56:17.1134454Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_cpu 2024-12-18T02:56:17.1135717Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_gloo 2024-12-18T02:56:17.1136854Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_register_just_once 2024-12-18T02:56:17.1137959Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_sparse_gradients 2024-12-18T02:56:17.1139263Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_init 2024-12-18T02:56:17.1140584Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_return_type 2024-12-18T02:56:17.1141935Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_find_unused_parameters_when_unused_parameters_empty 2024-12-18T02:56:17.1143282Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad 2024-12-18T02:56:17.1144618Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_grad_is_view 2024-12-18T02:56:17.1146030Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_static_graph 2024-12-18T02:56:17.1147433Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_integer_list 2024-12-18T02:56:17.1148851Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_torch_device_list 2024-12-18T02:56:17.1150158Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_2gpu_module 2024-12-18T02:56:17.1151433Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_4gpu_module 2024-12-18T02:56:17.1152481Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module 2024-12-18T02:56:17.1153572Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module_grad_is_view 2024-12-18T02:56:17.1154627Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output 2024-12-18T02:56:17.1155686Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output_with_unused_parameters 2024-12-18T02:56:17.1156785Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_sharded_tensor 2024-12-18T02:56:17.1157812Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_invalid_powerSGD_state 2024-12-18T02:56:17.1158865Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_save_load_checkpoint 2024-12-18T02:56:17.1159843Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients 2024-12-18T02:56:17.1160876Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients_grad_is_view 2024-12-18T02:56:17.1161961Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input 2024-12-18T02:56:17.1163061Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input 2024-12-18T02:56:17.1164035Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward 2024-12-18T02:56:17.1164903Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_optimizer 2024-12-18T02:56:17.1165816Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_unused_parameters 2024-12-18T02:56:17.1166739Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_multi_bucket 2024-12-18T02:56:17.1167619Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_single_bucket 2024-12-18T02:56:17.1168509Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_single_dtype_single_bucket 2024-12-18T02:56:17.1169335Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_bool_tensors 2024-12-18T02:56:17.1170220Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cpu 2024-12-18T02:56:17.1171116Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cuda 2024-12-18T02:56:17.1171971Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_gloo_rank_membership 2024-12-18T02:56:17.1172809Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_gloo_warn_not_in_group 2024-12-18T02:56:17.1173704Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_default 2024-12-18T02:56:17.1174658Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_subgroup 2024-12-18T02:56:17.1175594Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_default_pg_gloo 2024-12-18T02:56:17.1176516Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_gloo_new_group 2024-12-18T02:56:17.1177385Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_complex 2024-12-18T02:56:17.1178216Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_mismatch 2024-12-18T02:56:17.1179617Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single 2024-12-18T02:56:17.1180986Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allgather_coalesced 2024-12-18T02:56:17.1182368Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced 2024-12-18T02:56:17.1183707Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_collectives 2024-12-18T02:56:17.1185123Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2024-12-18T02:56:17.1186759Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2024-12-18T02:56:17.1188195Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_monitored_barrier 2024-12-18T02:56:17.1189362Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync 2024-12-18T02:56:17.1190400Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_duplicate_pg 2024-12-18T02:56:17.1191678Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_sanity_check 2024-12-18T02:56:17.1192231Z 2024-12-18T02:56:17.1192505Z Running distributed/elastic/timer/api_test 1/1 ... [2024-12-18 02:56:17.097195] 2024-12-18T02:56:17.1193016Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T02:56:17.1194181Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/timer/api_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 02:56:17.097583] 2024-12-18T02:56:18.8212532Z 2024-12-18T02:56:18.8213832Z distributed/elastic/timer/api_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.timer.api_test_1.1_c4073b0fa122f11b_.log 2024-12-18T02:56:18.8214776Z 2024-12-18T02:56:19.5926488Z Running test batch 'tests to run' cost 10068.0 seconds 2024-12-18T02:56:20.4164394Z 2024-12-18T02:56:20.4164853Z real 167m54.168s 2024-12-18T02:56:20.4165180Z user 435m36.504s 2024-12-18T02:56:20.4165445Z sys 166m16.028s 2024-12-18T02:56:20.4165995Z + assert_git_not_dirty 2024-12-18T02:56:20.4166363Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-12-18T02:56:20.4166801Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *xla* ]] 2024-12-18T02:56:20.4168051Z ++ git status --porcelain 2024-12-18T02:56:20.4168870Z ++ grep -v '?? third_party' 2024-12-18T02:56:23.0221413Z ++ true 2024-12-18T02:56:23.0221721Z + git_status= 2024-12-18T02:56:23.0222169Z + [[ -n '' ]] 2024-12-18T02:56:23.0222498Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-12-18T02:56:23.0222911Z + [[ 2 == 1 ]] 2024-12-18T02:56:23.0223152Z + [[ 2 == 1 ]] 2024-12-18T02:56:23.0223426Z + cleanup_workspace 2024-12-18T02:56:23.0224017Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-12-18T02:56:23.0224937Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-12-18T02:56:23.0225679Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-12-18T02:56:23.0226259Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-12-18T02:56:23.0226888Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-12-18T02:56:23.0227585Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-12-18T02:56:23.0228152Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-12-18T02:56:23.5188241Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@release/2.6 2024-12-18T02:56:23.5188848Z with: 2024-12-18T02:56:23.5189140Z benchmark-results-dir: test/test-reports 2024-12-18T02:56:23.5189525Z dry-run: false 2024-12-18T02:56:23.5189790Z schema-version: v3 2024-12-18T02:56:23.5190303Z github-token: *** 2024-12-18T02:56:23.5190579Z env: 2024-12-18T02:56:23.5190832Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:56:23.5191334Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:56:23.5192137Z DOCKER_CONTAINER_ID: 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T02:56:23.5192645Z ##[endgroup] 2024-12-18T02:56:23.5222277Z ##[group]Run set -eux 2024-12-18T02:56:23.5222590Z set -eux 2024-12-18T02:56:23.5222896Z python3 -mpip install boto3==1.35.33 2024-12-18T02:56:23.5233022Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:56:23.5233592Z env: 2024-12-18T02:56:23.5233877Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:56:23.5234234Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:56:23.5234796Z DOCKER_CONTAINER_ID: 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T02:56:23.5235306Z ##[endgroup] 2024-12-18T02:56:23.5268567Z + python3 -mpip install boto3==1.35.33 2024-12-18T02:56:23.7836893Z Defaulting to user installation because normal site-packages is not writeable 2024-12-18T02:56:24.9516236Z Collecting boto3==1.35.33 2024-12-18T02:56:24.9933471Z Downloading boto3-1.35.33-py3-none-any.whl (139 kB) 2024-12-18T02:56:25.0615213Z Collecting s3transfer<0.11.0,>=0.10.0 2024-12-18T02:56:25.0654395Z Downloading s3transfer-0.10.4-py3-none-any.whl (83 kB) 2024-12-18T02:56:26.2679459Z Collecting botocore<1.36.0,>=1.35.33 2024-12-18T02:56:26.2735591Z Downloading botocore-1.35.83-py3-none-any.whl (13.3 MB) 2024-12-18T02:56:26.5078658Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.0) 2024-12-18T02:56:26.5140368Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.8.1) 2024-12-18T02:56:26.5148600Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.25.10) 2024-12-18T02:56:26.7170840Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.15.0) 2024-12-18T02:56:26.8054905Z Installing collected packages: botocore, s3transfer, boto3 2024-12-18T02:56:27.4090774Z Successfully installed boto3-1.35.33 botocore-1.35.83 s3transfer-0.10.4 2024-12-18T02:56:27.5020336Z ##[group]Run set -eux 2024-12-18T02:56:27.5020664Z set -eux 2024-12-18T02:56:27.5020965Z  2024-12-18T02:56:27.5021236Z if [[ -z "${GITHUB_TOKEN}" ]]; then 2024-12-18T02:56:27.5021669Z  echo "Missing github-token input" 2024-12-18T02:56:27.5022243Z  exit 1 2024-12-18T02:56:27.5022511Z fi 2024-12-18T02:56:27.5030605Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:56:27.5031046Z env: 2024-12-18T02:56:27.5031281Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:56:27.5031664Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:56:27.5032287Z DOCKER_CONTAINER_ID: 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T02:56:27.5033148Z GITHUB_TOKEN: *** 2024-12-18T02:56:27.5033425Z ##[endgroup] 2024-12-18T02:56:27.5058549Z + [[ -z *** ]] 2024-12-18T02:56:27.5114763Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main 2024-12-18T02:56:27.5115225Z with: 2024-12-18T02:56:27.5115589Z github-token: *** 2024-12-18T02:56:27.5115840Z env: 2024-12-18T02:56:27.5116086Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:56:27.5116427Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:56:27.5117155Z DOCKER_CONTAINER_ID: 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T02:56:27.5117667Z ##[endgroup] 2024-12-18T02:56:27.5143809Z ##[group]Run set -eux 2024-12-18T02:56:27.5144123Z set -eux 2024-12-18T02:56:27.5144393Z  2024-12-18T02:56:27.5144947Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-12-18T02:56:27.5151107Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:56:27.5151652Z env: 2024-12-18T02:56:27.5151883Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:56:27.5152240Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:56:27.5152810Z DOCKER_CONTAINER_ID: 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T02:56:27.5153503Z GITHUB_TOKEN: *** 2024-12-18T02:56:27.5153899Z ##[endgroup] 2024-12-18T02:56:27.5178050Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 12383255652 i-0a62689d000b23974 2024-12-18T02:56:30.6140482Z setting job-id=34566462868 2024-12-18T02:56:30.6141360Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T02:56:30.6288068Z ##[group]Run set -eux 2024-12-18T02:56:30.6288516Z set -eux 2024-12-18T02:56:30.6288910Z  2024-12-18T02:56:30.6289450Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \ 2024-12-18T02:56:30.6290089Z  --schema-version "${SCHEMA_VERSION}" \ 2024-12-18T02:56:30.6290602Z  --repo "${REPO}" \ 2024-12-18T02:56:30.6291044Z  --head-branch "${HEAD_BRANCH}" \ 2024-12-18T02:56:30.6291561Z  --head-sha "${HEAD_SHA}" \ 2024-12-18T02:56:30.6292038Z  --workflow-id "${WORKFLOW_RUN_ID}" \ 2024-12-18T02:56:30.6292545Z  --run-attempt "${RUN_ATTEMPT}" \ 2024-12-18T02:56:30.6293031Z  --job-id "${JOB_ID}" \ 2024-12-18T02:56:30.6293471Z  --job-name "${JOB_NAME}" 2024-12-18T02:56:30.6300114Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:56:30.6300712Z env: 2024-12-18T02:56:30.6301123Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:56:30.6301671Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:56:30.6302398Z DOCKER_CONTAINER_ID: 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T02:56:30.6303134Z SCHEMA_VERSION: v3 2024-12-18T02:56:30.6303582Z REPO: pytorch/pytorch 2024-12-18T02:56:30.6304042Z HEAD_BRANCH: refs/heads/release/2.6 2024-12-18T02:56:30.6304618Z HEAD_SHA: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T02:56:30.6305114Z WORKFLOW_RUN_ID: 12383255652 2024-12-18T02:56:30.6305668Z RUN_ATTEMPT: 1 2024-12-18T02:56:30.6305949Z JOB_ID: 34566462868 2024-12-18T02:56:30.6306482Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-12-18T02:56:30.6307092Z ##[endgroup] 2024-12-18T02:56:30.6335489Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/release/2.6/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/heads/release/2.6 --head-sha 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 --workflow-id 12383255652 --run-attempt 1 --job-id 34566462868 --job-name 'linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.g4dn.12xlarge.nvidia.gpu)' 2024-12-18T02:56:30.6707131Z ##[group]Run set -eux 2024-12-18T02:56:30.6707431Z set -eux 2024-12-18T02:56:30.6707671Z  2024-12-18T02:56:30.6707948Z # TODO (huydhn): Implement this part 2024-12-18T02:56:30.6708348Z echo "runners=[]" >> "${GITHUB_OUTPUT}" 2024-12-18T02:56:30.6713696Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:56:30.6714119Z env: 2024-12-18T02:56:30.6714360Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:56:30.6714872Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:56:30.6715461Z DOCKER_CONTAINER_ID: 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T02:56:30.6715980Z ##[endgroup] 2024-12-18T02:56:30.6740395Z + echo 'runners=[]' 2024-12-18T02:56:30.6773988Z ##[group]Run set -eux 2024-12-18T02:56:30.6774290Z set -eux 2024-12-18T02:56:30.6774527Z  2024-12-18T02:56:30.6774810Z # TODO (huydhn): Implement this part 2024-12-18T02:56:30.6775230Z echo "dependencies={}" >> "${GITHUB_OUTPUT}" 2024-12-18T02:56:30.6780971Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:56:30.6781432Z env: 2024-12-18T02:56:30.6781698Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:56:30.6782086Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:56:30.6782739Z DOCKER_CONTAINER_ID: 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T02:56:30.6783496Z ##[endgroup] 2024-12-18T02:56:30.6808395Z + echo 'dependencies={}' 2024-12-18T02:56:30.6845886Z ##[group]Run set -eux 2024-12-18T02:56:30.6846192Z set -eux 2024-12-18T02:56:30.6846443Z  2024-12-18T02:56:30.6846735Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then 2024-12-18T02:56:30.6847208Z  echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping" 2024-12-18T02:56:30.6847716Z  # We don't want the job to fail if the directory doesn't exist 2024-12-18T02:56:30.6848158Z  exit 0 2024-12-18T02:56:30.6848413Z fi 2024-12-18T02:56:30.6848651Z  2024-12-18T02:56:30.6848899Z if [[ "${DRY_RUN}" == "true" ]]; then 2024-12-18T02:56:30.6849418Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2024-12-18T02:56:30.6850017Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2024-12-18T02:56:30.6850498Z  --metadata "${BENCHMARK_METADATA}" \ 2024-12-18T02:56:30.6850897Z  --runners "${RUNNER_INFO}" \ 2024-12-18T02:56:30.6851285Z  --dependencies "${DEPENDENCIES}" \ 2024-12-18T02:56:30.6851638Z  --dry-run 2024-12-18T02:56:30.6851921Z else 2024-12-18T02:56:30.6852334Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2024-12-18T02:56:30.6852928Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2024-12-18T02:56:30.6853397Z  --metadata "${BENCHMARK_METADATA}" \ 2024-12-18T02:56:30.6853772Z  --runners "${RUNNER_INFO}" \ 2024-12-18T02:56:30.6854160Z  --dependencies "${DEPENDENCIES}" 2024-12-18T02:56:30.6854513Z fi 2024-12-18T02:56:30.6859971Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:56:30.6860437Z env: 2024-12-18T02:56:30.6860702Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:56:30.6861101Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:56:30.6861758Z DOCKER_CONTAINER_ID: 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T02:56:30.6862371Z BENCHMARK_RESULTS_DIR: test/test-reports 2024-12-18T02:56:30.6862755Z DRY_RUN: false 2024-12-18T02:56:30.6864597Z BENCHMARK_METADATA: {"timestamp": 1734490590, "schema_version": "v3", "name": "linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.g4dn.12xlarge.nvidia.gpu)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/release/2.6", "head_sha": "0cdf8b1d09254cfda66191d1bd01e3041c3c76f7", "workflow_id": 12383255652, "run_attempt": 1, "job_id": 34566462868} 2024-12-18T02:56:30.6866329Z RUNNER_INFO: [] 2024-12-18T02:56:30.6866618Z DEPENDENCIES: {} 2024-12-18T02:56:30.6866886Z ##[endgroup] 2024-12-18T02:56:30.6890395Z + [[ ! -d test/test-reports ]] 2024-12-18T02:56:30.6890909Z + [[ false == \t\r\u\e ]] 2024-12-18T02:56:30.6893995Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/release/2.6/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1734490590, "schema_version": "v3", "name": "linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.g4dn.12xlarge.nvidia.gpu)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/release/2.6", "head_sha": "0cdf8b1d09254cfda66191d1bd01e3041c3c76f7", "workflow_id": 12383255652, "run_attempt": 1, "job_id": 34566462868}' --runners '[]' --dependencies '{}' 2024-12-18T02:56:30.8654670Z ##[group]Run cat test/**/*_toprint.log || true 2024-12-18T02:56:30.8655217Z cat test/**/*_toprint.log || true 2024-12-18T02:56:30.8661387Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:56:30.8661839Z env: 2024-12-18T02:56:30.8662096Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:56:30.8662492Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:56:30.8663132Z DOCKER_CONTAINER_ID: 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T02:56:30.8663842Z ##[endgroup] 2024-12-18T02:56:30.8737359Z cat: 'test/**/*_toprint.log': No such file or directory 2024-12-18T02:56:30.8775671Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-12-18T02:56:30.8776067Z kill "$MONITOR_SCRIPT_PID" 2024-12-18T02:56:30.8782275Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:56:30.8782727Z env: 2024-12-18T02:56:30.8782986Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:56:30.8783379Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:56:30.8784025Z DOCKER_CONTAINER_ID: 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T02:56:30.8784596Z MONITOR_SCRIPT_PID: 57285 2024-12-18T02:56:30.8784910Z ##[endgroup] 2024-12-18T02:56:30.8945948Z Prepare all required actions 2024-12-18T02:56:30.8946433Z Getting action download info 2024-12-18T02:56:31.0333139Z Download action repository 'actions/upload-artifact@v4' (SHA:6f51ac03b9356f520e9adb1b1b7802705f340c2b) 2024-12-18T02:56:31.3907222Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-12-18T02:56:31.3907646Z with: 2024-12-18T02:56:31.3908105Z file-suffix: test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_34566462868 2024-12-18T02:56:31.3908669Z s3-bucket: gha-artifacts 2024-12-18T02:56:31.3908978Z env: 2024-12-18T02:56:31.3909231Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:56:31.3909626Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:56:31.3910267Z DOCKER_CONTAINER_ID: 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T02:56:31.3910866Z ##[endgroup] 2024-12-18T02:56:31.3971427Z ##[group]Run # Remove any previous test jsons if they exist 2024-12-18T02:56:31.3971982Z # Remove any previous test jsons if they exist 2024-12-18T02:56:31.3972365Z rm -f test-jsons-*.zip 2024-12-18T02:56:31.3972818Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json' 2024-12-18T02:56:31.3978610Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:56:31.3979280Z env: 2024-12-18T02:56:31.3979540Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:56:31.3979938Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:56:31.3980565Z DOCKER_CONTAINER_ID: 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T02:56:31.3981335Z FILE_SUFFIX: test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_34566462868 2024-12-18T02:56:31.3981888Z ##[endgroup] 2024-12-18T02:56:31.4203856Z adding: test/test-reports/td_exclusions-ef00ec9b7f08a73e3ca1.json (deflated 86%) 2024-12-18T02:56:31.4237958Z ##[group]Run # Remove any previous test reports if they exist 2024-12-18T02:56:31.4238489Z # Remove any previous test reports if they exist 2024-12-18T02:56:31.4238936Z rm -f test-reports-*.zip 2024-12-18T02:56:31.4239454Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv' 2024-12-18T02:56:31.4245060Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:56:31.4245470Z env: 2024-12-18T02:56:31.4245702Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:56:31.4246054Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:56:31.4246627Z DOCKER_CONTAINER_ID: 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T02:56:31.4247303Z FILE_SUFFIX: test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_34566462868 2024-12-18T02:56:31.4247793Z ##[endgroup] 2024-12-18T02:56:31.4345346Z adding: test/test-reports/python-pytest/distributed._tools.test_runtime_estimator/distributed._tools.test_runtime_estimator-d41680a91aac2ab3.xml (deflated 59%) 2024-12-18T02:56:31.4346856Z adding: test/test-reports/python-pytest/distributed.test_device_mesh/distributed.test_device_mesh-5cbb5861f5447018.xml (deflated 85%) 2024-12-18T02:56:31.4348323Z adding: test/test-reports/python-pytest/distributed._tensor.test_experimental_ops/distributed._tensor.test_experimental_ops-2a0d5e851764af13.xml (deflated 57%) 2024-12-18T02:56:31.4350048Z adding: test/test-reports/python-pytest/distributed._tensor.test_tensor_ops/distributed._tensor.test_tensor_ops-0bca591db812c451.xml (deflated 86%) 2024-12-18T02:56:31.4351813Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_extensions/distributed._composable.fsdp.test_fully_shard_extensions-2d759e72a665362a.xml (deflated 72%) 2024-12-18T02:56:31.4353349Z adding: test/test-reports/python-pytest/distributed.fsdp.test_wrap/distributed.fsdp.test_wrap-05f682f1206dfb1f.xml (deflated 86%) 2024-12-18T02:56:31.4354889Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_autograd/distributed._composable.fsdp.test_fully_shard_autograd-e4bf652b4a78a7ac.xml (deflated 68%) 2024-12-18T02:56:31.4356680Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_clip_grad_norm/distributed.fsdp.test_fsdp_clip_grad_norm-d3cf83625aa9ee22.xml (deflated 64%) 2024-12-18T02:56:31.4358110Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_misc/distributed.fsdp.test_fsdp_misc-a2362395756df2fd.xml (deflated 79%) 2024-12-18T02:56:31.4359426Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-17fd5ff599c79589.xml (deflated 38%) 2024-12-18T02:56:31.4360710Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-481d25da9e42eef9.xml (deflated 37%) 2024-12-18T02:56:31.4361990Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-d7508c01bb737c40.xml (deflated 37%) 2024-12-18T02:56:31.4363267Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-b569e60b1f451c88.xml (deflated 36%) 2024-12-18T02:56:31.4364548Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-538340ceb192db4c.xml (deflated 38%) 2024-12-18T02:56:31.4365812Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-072e79f516c0acb0.xml (deflated 38%) 2024-12-18T02:56:31.4367105Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-bbc5482206b1ef96.xml (deflated 37%) 2024-12-18T02:56:31.4368394Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-1f3807e9dd9eacff.xml (deflated 37%) 2024-12-18T02:56:31.4369675Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-2acc05a625e119ee.xml (deflated 37%) 2024-12-18T02:56:31.4370957Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-3fd2b81984e7ed85.xml (deflated 37%) 2024-12-18T02:56:31.4372241Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-90c248a4cb1cff08.xml (deflated 37%) 2024-12-18T02:56:31.4373524Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-4fcfa017e256bc05.xml (deflated 37%) 2024-12-18T02:56:31.4374825Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-edfd75d6f5e4eeac.xml (deflated 37%) 2024-12-18T02:56:31.4376113Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-fc0f9dcdb04ef52c.xml (deflated 37%) 2024-12-18T02:56:31.4377380Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-ad31ee0d1093a43d.xml (deflated 37%) 2024-12-18T02:56:31.4378743Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-47da096ff9cda6a8.xml (deflated 38%) 2024-12-18T02:56:31.4380266Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-ae705ae366157237.xml (deflated 38%) 2024-12-18T02:56:31.4381589Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-fe5bef6aa1c873ed.xml (deflated 37%) 2024-12-18T02:56:31.4382994Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-b4c3576e376cec38.xml (deflated 38%) 2024-12-18T02:56:31.4384320Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-a4015f41677fda5f.xml (deflated 38%) 2024-12-18T02:56:31.4385648Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-efaf765f18753803.xml (deflated 38%) 2024-12-18T02:56:31.4386966Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-4cbc6c31aee5d5f9.xml (deflated 37%) 2024-12-18T02:56:31.4388377Z adding: test/test-reports/python-pytest/distributed.test_c10d_common/distributed.test_c10d_common-70b05437df49c5ed.xml (deflated 37%) 2024-12-18T02:56:31.4389768Z adding: test/test-reports/python-pytest/distributed._tools.test_mod_tracker/distributed._tools.test_mod_tracker-561ac64404b6bcba.xml (deflated 65%) 2024-12-18T02:56:31.4391476Z adding: test/test-reports/python-pytest/distributed.pipelining.test_schedule_multiproc/distributed.pipelining.test_schedule_multiproc-8adf8c19beae2910.xml (deflated 87%) 2024-12-18T02:56:31.4393181Z adding: test/test-reports/python-pytest/distributed.pipelining.test_schedule_multiproc/distributed.pipelining.test_schedule_multiproc-c3e8e1e4014c7b7a.xml (deflated 88%) 2024-12-18T02:56:31.4394916Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_micro_pipeline_tp/distributed.tensor.parallel.test_micro_pipeline_tp-6088d1ec0ccd2422.xml (deflated 89%) 2024-12-18T02:56:31.4396554Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_nested_dict/distributed.checkpoint.test_nested_dict-ed523c5163ff1c39.xml (deflated 51%) 2024-12-18T02:56:31.4398107Z adding: test/test-reports/python-pytest/distributed.pipelining.test_transformer/distributed.pipelining.test_transformer-f55c55ca57c27f60.xml (deflated 39%) 2024-12-18T02:56:31.4399637Z adding: test/test-reports/python-pytest/distributed.pipelining.test_backward/distributed.pipelining.test_backward-2e582f138d275de1.xml (deflated 67%) 2024-12-18T02:56:31.4401175Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.test_logger/distributed._shard.sharded_tensor.test_logger-bacd8dcd80e353c6.xml (deflated 37%) 2024-12-18T02:56:31.4402685Z adding: test/test-reports/python-pytest/distributed.pipelining.test_pipe/distributed.pipelining.test_pipe-2a777299d13923d4.xml (deflated 65%) 2024-12-18T02:56:31.4404114Z adding: test/test-reports/python-pytest/distributed._tools.test_sac_estimator/distributed._tools.test_sac_estimator-cad78b21ab4cd2e6.xml (deflated 51%) 2024-12-18T02:56:31.4405502Z adding: test/test-reports/python-pytest/distributed._tools.test_sac_ilp/distributed._tools.test_sac_ilp-68293cd590a36c96.xml (deflated 66%) 2024-12-18T02:56:31.4406957Z adding: test/test-reports/python-pytest/distributed.test_c10d_functional_native/distributed.test_c10d_functional_native-23f9e9a4ddbba643.xml (deflated 82%) 2024-12-18T02:56:31.4408487Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_save_load_api/distributed.checkpoint.test_save_load_api-a8393272d3cf8bb3.xml (deflated 36%) 2024-12-18T02:56:31.4410054Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_dedup_tensors/distributed.checkpoint.test_dedup_tensors-6271bab90154ba5d.xml (deflated 38%) 2024-12-18T02:56:31.4411587Z adding: test/test-reports/python-pytest/distributed._composable.test_contract/distributed._composable.test_contract-943d7affd632de55.xml (deflated 69%) 2024-12-18T02:56:31.4413095Z adding: test/test-reports/python-pytest/distributed.pipelining.test_microbatch/distributed.pipelining.test_microbatch-fbc00d066556d632.xml (deflated 54%) 2024-12-18T02:56:31.4414637Z adding: test/test-reports/python-pytest/distributed.checkpoint.e2e.test_fsdp_ep/distributed.checkpoint.e2e.test_fsdp_ep-523e6dcff0a61c3b.xml (deflated 50%) 2024-12-18T02:56:31.4416324Z adding: test/test-reports/python-pytest/distributed._tensor.experimental.test_register_sharding/distributed._tensor.experimental.test_register_sharding-ac88eb47c29a2d7a.xml (deflated 50%) 2024-12-18T02:56:31.4418031Z adding: test/test-reports/python-pytest/distributed._tensor.test_attention/distributed._tensor.test_attention-bedcb47a2f2abf5f.xml (deflated 92%) 2024-12-18T02:56:31.4419573Z adding: test/test-reports/python-pytest/distributed.test_fake_pg/distributed.test_fake_pg-7598e95da50b3188.xml (deflated 79%) 2024-12-18T02:56:31.4421021Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_fsdp_optim_state/distributed.checkpoint.test_fsdp_optim_state-d1d6a62433527493.xml (deflated 53%) 2024-12-18T02:56:31.4422945Z adding: test/test-reports/python-pytest/distributed._composable.test_checkpoint/distributed._composable.test_checkpoint-f3a7b7ec9c53ec55.xml (deflated 72%) 2024-12-18T02:56:31.4424610Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_dtensor_resharding/distributed.checkpoint.test_dtensor_resharding-72e3b5a7a53f784c.xml (deflated 68%) 2024-12-18T02:56:31.4426233Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_fsspec/distributed.checkpoint.test_fsspec-54854e0f2fdf77a7.xml (deflated 48%) 2024-12-18T02:56:31.4427803Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_style/distributed.tensor.parallel.test_tp_style-8507d5f27fc88590.xml (deflated 78%) 2024-12-18T02:56:31.4429479Z adding: test/test-reports/python-pytest/distributed.checkpoint.e2e.test_e2e_save_and_load/distributed.checkpoint.e2e.test_e2e_save_and_load-dca14ded70b769a7.xml (deflated 81%) 2024-12-18T02:56:31.4431295Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_examples/distributed.tensor.parallel.test_tp_examples-a522e81036355164.xml (deflated 85%) 2024-12-18T02:56:31.4432970Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_state/distributed._composable.fsdp.test_fully_shard_state-776bf0b253317a6b.xml (deflated 73%) 2024-12-18T02:56:31.4434554Z adding: test/test-reports/python-pytest/distributed.test_control_collectives/distributed.test_control_collectives-13b1ad69b65317a4.xml (deflated 84%) 2024-12-18T02:56:31.4436038Z adding: test/test-reports/python-pytest/distributed._tensor.debug.test_comm_mode/distributed._tensor.debug.test_comm_mode-5aa21c198af17ee3.xml (deflated 66%) 2024-12-18T02:56:31.4437537Z adding: test/test-reports/python-pytest/distributed._composable.test_replicate/distributed._composable.test_replicate-74d8bc7e74220281.xml (deflated 83%) 2024-12-18T02:56:31.4439009Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_state_dict/distributed.fsdp.test_fsdp_state_dict-cb2d20bebf01882c.xml (deflated 95%) 2024-12-18T02:56:31.4440364Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b3ded225c16be403.xml (deflated 33%) 2024-12-18T02:56:31.4441608Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-dbdc5c4e1d533fbd.xml (deflated 34%) 2024-12-18T02:56:31.4442844Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2f8681e27eb1c411.xml (deflated 34%) 2024-12-18T02:56:31.4444078Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-64af929e2d8a37ab.xml (deflated 34%) 2024-12-18T02:56:31.4445300Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ec5d0c2182f6fb2f.xml (deflated 36%) 2024-12-18T02:56:31.4446533Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3ecf20272294f2f6.xml (deflated 49%) 2024-12-18T02:56:31.4447769Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c89f09a677b671b6.xml (deflated 36%) 2024-12-18T02:56:31.4449005Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ecc2ab4cb5a03082.xml (deflated 34%) 2024-12-18T02:56:31.4450323Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f247ecb19ed9ac09.xml (deflated 34%) 2024-12-18T02:56:31.4451558Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4620d0d63535df4a.xml (deflated 35%) 2024-12-18T02:56:31.4452787Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-e2f1d191926159cd.xml (deflated 34%) 2024-12-18T02:56:31.4454016Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9fd3ef60e1eb42e7.xml (deflated 36%) 2024-12-18T02:56:31.4455242Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-e4df9f7b81fa33a7.xml (deflated 37%) 2024-12-18T02:56:31.4456540Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-45d5f73f0646f7d0.xml (deflated 36%) 2024-12-18T02:56:31.4457769Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9748182d0807a713.xml (deflated 37%) 2024-12-18T02:56:31.4459228Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-773d2baf5f518d1c.xml (deflated 36%) 2024-12-18T02:56:31.4460499Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2439e4fd131225e1.xml (deflated 35%) 2024-12-18T02:56:31.4461778Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-fd76f4af806ce7fb.xml (deflated 36%) 2024-12-18T02:56:31.4463051Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-90d19c96fc5d3076.xml (deflated 36%) 2024-12-18T02:56:31.4464330Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-73aca0788f1d16ab.xml (deflated 36%) 2024-12-18T02:56:31.4465595Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d1584b19f1986fda.xml (deflated 36%) 2024-12-18T02:56:31.4466871Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6e7d98e5e6071c28.xml (deflated 36%) 2024-12-18T02:56:31.4468145Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5d27fff5ba10d87f.xml (deflated 37%) 2024-12-18T02:56:31.4469417Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f16a868f152a6adb.xml (deflated 36%) 2024-12-18T02:56:31.4470688Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5a8ba3999b4d6585.xml (deflated 37%) 2024-12-18T02:56:31.4472042Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-69ebea566519e7b8.xml (deflated 36%) 2024-12-18T02:56:31.4473272Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-770365acab91f25c.xml (deflated 35%) 2024-12-18T02:56:31.4474496Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-11fc661baed02bb2.xml (deflated 37%) 2024-12-18T02:56:31.4475728Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d3e889dbca726715.xml (deflated 36%) 2024-12-18T02:56:31.4476965Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-754d92121b98dce8.xml (deflated 36%) 2024-12-18T02:56:31.4478196Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9632990eeb9104af.xml (deflated 36%) 2024-12-18T02:56:31.4479422Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-80b3297898132ead.xml (deflated 35%) 2024-12-18T02:56:31.4480659Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-0cdbee02358e8336.xml (deflated 48%) 2024-12-18T02:56:31.4481895Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-86da800f1ac262ea.xml (deflated 35%) 2024-12-18T02:56:31.4483199Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b2922fed0ad3c39e.xml (deflated 45%) 2024-12-18T02:56:31.4484424Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-52b35c088a73e2b5.xml (deflated 49%) 2024-12-18T02:56:31.4485651Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-30e73ec073a21723.xml (deflated 35%) 2024-12-18T02:56:31.4486882Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-77c6e1d766bcf48b.xml (deflated 35%) 2024-12-18T02:56:31.4488179Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-3bbb1783392a50ba.xml (deflated 35%) 2024-12-18T02:56:31.4489410Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-5e1953d6a05b139d.xml (deflated 36%) 2024-12-18T02:56:31.4490645Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-f8086dd27eb86573.xml (deflated 36%) 2024-12-18T02:56:31.4491879Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-4fd693f624255c0e.xml (deflated 58%) 2024-12-18T02:56:31.4493104Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-bf2221e31be7bcb8.xml (deflated 36%) 2024-12-18T02:56:31.4494340Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6e1b048c34da894b.xml (deflated 34%) 2024-12-18T02:56:31.4495580Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-0388a6708f93b4de.xml (deflated 34%) 2024-12-18T02:56:31.4496812Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b27bc2625a3b3451.xml (deflated 34%) 2024-12-18T02:56:31.4498050Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d8bdd924b49b9026.xml (deflated 34%) 2024-12-18T02:56:31.4499525Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9ac2e3650b423dd8.xml (deflated 35%) 2024-12-18T02:56:31.4500801Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-2fe599e9574acf93.xml (deflated 35%) 2024-12-18T02:56:31.4502078Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6041e0d71d2a7885.xml (deflated 35%) 2024-12-18T02:56:31.4503338Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-b8ea37370935ba14.xml (deflated 34%) 2024-12-18T02:56:31.4504625Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9f0d9cc3ef5eda93.xml (deflated 35%) 2024-12-18T02:56:31.4505908Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-db89512f0720c682.xml (deflated 35%) 2024-12-18T02:56:31.4507189Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-16d57039cf586cf1.xml (deflated 35%) 2024-12-18T02:56:31.4508459Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ecccff2d19d669f6.xml (deflated 34%) 2024-12-18T02:56:31.4509737Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-9c35c0b6dc9b5290.xml (deflated 34%) 2024-12-18T02:56:31.4511111Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-d8c27515768a32fa.xml (deflated 35%) 2024-12-18T02:56:31.4512347Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-802f28acaa25399b.xml (deflated 35%) 2024-12-18T02:56:31.4513568Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-fe9157b867a4f6d1.xml (deflated 33%) 2024-12-18T02:56:31.4514866Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-57592c58655e8b2a.xml (deflated 35%) 2024-12-18T02:56:31.4516105Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-ce0ac507d42cdf4c.xml (deflated 35%) 2024-12-18T02:56:31.4517361Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-8dd612b5c5fe977e.xml (deflated 35%) 2024-12-18T02:56:31.4518599Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-6f3b5298b96e4440.xml (deflated 34%) 2024-12-18T02:56:31.4519889Z adding: test/test-reports/python-pytest/distributed.test_c10d_nccl/distributed.test_c10d_nccl-c72927a6250c670c.xml (deflated 34%) 2024-12-18T02:56:31.4521357Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_comm/distributed._composable.fsdp.test_fully_shard_comm-392874c097ec815d.xml (deflated 81%) 2024-12-18T02:56:31.4523105Z adding: test/test-reports/python-pytest/distributed.nn.jit.test_instantiator/distributed.nn.jit.test_instantiator-144b933fa18a15cd.xml (deflated 61%) 2024-12-18T02:56:31.4524622Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_backward_prefetch/distributed.fsdp.test_fsdp_backward_prefetch-6c5053d150687607.xml (deflated 40%) 2024-12-18T02:56:31.4526129Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_pure_fp16/distributed.fsdp.test_fsdp_pure_fp16-0a6cee2e2389ab28.xml (deflated 50%) 2024-12-18T02:56:31.4527602Z adding: test/test-reports/python-pytest/distributed.test_c10d_object_collectives/distributed.test_c10d_object_collectives-661b427838d5bfba.xml (deflated 77%) 2024-12-18T02:56:31.4528992Z adding: test/test-reports/python-pytest/distributed.test_c10d_pypg/distributed.test_c10d_pypg-27d6acbff9cf91a9.xml (deflated 87%) 2024-12-18T02:56:31.4530408Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_freezing_weights/distributed.fsdp.test_fsdp_freezing_weights-73a5e5ee368c71bc.xml (deflated 94%) 2024-12-18T02:56:31.4532063Z adding: test/test-reports/python-pytest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks-c6f2f0b93a6ce88d.xml (deflated 74%) 2024-12-18T02:56:31.4533661Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_exec_order/distributed.fsdp.test_fsdp_exec_order-74dbb55465caa76e.xml (deflated 80%) 2024-12-18T02:56:31.4535192Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_grad_acc/distributed.fsdp.test_fsdp_grad_acc-d81f07fbdc7ec53d.xml (deflated 72%) 2024-12-18T02:56:31.4536676Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_ignored_modules/distributed.fsdp.test_fsdp_ignored_modules-87d743c773295a8d.xml (deflated 76%) 2024-12-18T02:56:31.4538248Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_multiple_wrapping/distributed.fsdp.test_fsdp_multiple_wrapping-87647a3f0be84774.xml (deflated 37%) 2024-12-18T02:56:31.4540263Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_mixed_precision/distributed._composable.fsdp.test_fully_shard_mixed_precision-bd6449e80b0c9f6e.xml (deflated 73%) 2024-12-18T02:56:31.4542302Z adding: test/test-reports/python-pytest/distributed._composable.test_composability.test_2d_composability/distributed._composable.test_composability.test_2d_composability-129c3b917b28da07.xml (deflated 84%) 2024-12-18T02:56:31.4544083Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm_hooks/distributed.fsdp.test_fsdp_comm_hooks-c854f59fd142b072.xml (deflated 90%) 2024-12-18T02:56:31.4545530Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-ffa1369a7e0b678d.xml (deflated 86%) 2024-12-18T02:56:31.4546920Z adding: test/test-reports/python-pytest/distributed._shard.test_sharder/distributed._shard.test_sharder-80d32eca8a6d3c87.xml (deflated 51%) 2024-12-18T02:56:31.4548394Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_comm/distributed.fsdp.test_fsdp_comm-30f95b768ad7cedf.xml (deflated 83%) 2024-12-18T02:56:31.4549928Z adding: test/test-reports/python-pytest/distributed.fsdp.test_distributed_checkpoint/distributed.fsdp.test_distributed_checkpoint-91d946e08ae42faa.xml (deflated 62%) 2024-12-18T02:56:31.4551657Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_use_orig_params/distributed.fsdp.test_fsdp_use_orig_params-5376f8cf14a44808.xml (deflated 85%) 2024-12-18T02:56:31.4553039Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9fc2cc4166282ad5.xml (deflated 37%) 2024-12-18T02:56:31.4554348Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-98643193f92e4d67.xml (deflated 38%) 2024-12-18T02:56:31.4555582Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-d90d822118a042b5.xml (deflated 37%) 2024-12-18T02:56:31.4556811Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-866b356cb43365e8.xml (deflated 37%) 2024-12-18T02:56:31.4558046Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-b2fde42939a3dc07.xml (deflated 37%) 2024-12-18T02:56:31.4559278Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-5a93966dcd3ccd79.xml (deflated 37%) 2024-12-18T02:56:31.4560514Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-e6cdb6a4dea08429.xml (deflated 37%) 2024-12-18T02:56:31.4561760Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-d5b24943cbd6580c.xml (deflated 37%) 2024-12-18T02:56:31.4562996Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-d7bc9e4a28af84db.xml (deflated 37%) 2024-12-18T02:56:31.4564237Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-396251cf1edb554a.xml (deflated 37%) 2024-12-18T02:56:31.4565477Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-8e660e3d2697f6e6.xml (deflated 37%) 2024-12-18T02:56:31.4566708Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-4d73f86c82181a00.xml (deflated 37%) 2024-12-18T02:56:31.4567940Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9dd4729df6f84c3d.xml (deflated 38%) 2024-12-18T02:56:31.4569163Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-aff5cd0c952d0e15.xml (deflated 38%) 2024-12-18T02:56:31.4570408Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-d6aa02adc19d5b39.xml (deflated 37%) 2024-12-18T02:56:31.4571644Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-f400b8c647e2d168.xml (deflated 37%) 2024-12-18T02:56:31.4572886Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-e7a66ccd4268f753.xml (deflated 38%) 2024-12-18T02:56:31.4574117Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7d33e14871cab6bf.xml (deflated 37%) 2024-12-18T02:56:31.4575358Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-aca4c36adae5f5e1.xml (deflated 37%) 2024-12-18T02:56:31.4576598Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-a3eabd7548c65540.xml (deflated 38%) 2024-12-18T02:56:31.4577833Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-60a25807260d9cbe.xml (deflated 37%) 2024-12-18T02:56:31.4579280Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-5be1229c1aa0c1c3.xml (deflated 37%) 2024-12-18T02:56:31.4580627Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-6cc7c3b794e9457d.xml (deflated 37%) 2024-12-18T02:56:31.4581907Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-c09ca90c7b351a72.xml (deflated 38%) 2024-12-18T02:56:31.4583184Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-d8b54a3a2f0a11d6.xml (deflated 38%) 2024-12-18T02:56:31.4584467Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-b22b2231a6b5bd27.xml (deflated 37%) 2024-12-18T02:56:31.4585800Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2dc48aad5007417b.xml (deflated 38%) 2024-12-18T02:56:31.4587070Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-8b4f767800b3764c.xml (deflated 38%) 2024-12-18T02:56:31.4588327Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-ac9c15e66b647d85.xml (deflated 38%) 2024-12-18T02:56:31.4589608Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-469eda7896d4fbb0.xml (deflated 38%) 2024-12-18T02:56:31.4590885Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9ab8281be37d8a6d.xml (deflated 37%) 2024-12-18T02:56:31.4592244Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-b1a3ce8cd74d8af9.xml (deflated 37%) 2024-12-18T02:56:31.4593484Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7941546f787ccc96.xml (deflated 37%) 2024-12-18T02:56:31.4594716Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-4d35462c083f4332.xml (deflated 38%) 2024-12-18T02:56:31.4595953Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-cd673e3a06316d0e.xml (deflated 37%) 2024-12-18T02:56:31.4597188Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-6d472cc5fa784686.xml (deflated 37%) 2024-12-18T02:56:31.4598407Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2d3ed6a3c87923a6.xml (deflated 37%) 2024-12-18T02:56:31.4599638Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-f3eb239026155b11.xml (deflated 38%) 2024-12-18T02:56:31.4600874Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-e82f2d021ff31176.xml (deflated 37%) 2024-12-18T02:56:31.4602114Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-e2f2f1c2238a4400.xml (deflated 37%) 2024-12-18T02:56:31.4603345Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-0a5f14fa2d98d655.xml (deflated 38%) 2024-12-18T02:56:31.4604586Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2b431bdbe4218e23.xml (deflated 38%) 2024-12-18T02:56:31.4605991Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-6a6ea596ac8c93ba.xml (deflated 38%) 2024-12-18T02:56:31.4607217Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-cf516bc2a53c3000.xml (deflated 37%) 2024-12-18T02:56:31.4608453Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-48cc125c65ca9e64.xml (deflated 37%) 2024-12-18T02:56:31.4609688Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-ef1afd2ce37ce823.xml (deflated 37%) 2024-12-18T02:56:31.4610923Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-20665ee68114c653.xml (deflated 37%) 2024-12-18T02:56:31.4612150Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-788a188254ab0ea5.xml (deflated 47%) 2024-12-18T02:56:31.4613442Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-8736a9075d7a7ab9.xml (deflated 38%) 2024-12-18T02:56:31.4614680Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9e7af2a5b1209be3.xml (deflated 38%) 2024-12-18T02:56:31.4615912Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2c5203fb9503a50d.xml (deflated 38%) 2024-12-18T02:56:31.4617136Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-8979ffbbc62f9da0.xml (deflated 38%) 2024-12-18T02:56:31.4618428Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-e223aa9193266dfb.xml (deflated 37%) 2024-12-18T02:56:31.4619930Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-40f65e20182446dd.xml (deflated 37%) 2024-12-18T02:56:31.4621211Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-1ba3f10acc9333e3.xml (deflated 39%) 2024-12-18T02:56:31.4622634Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-41683a059662207c.xml (deflated 39%) 2024-12-18T02:56:31.4623907Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-3cd6daa1c4aa739d.xml (deflated 38%) 2024-12-18T02:56:31.4625183Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-111c76d93814e02f.xml (deflated 37%) 2024-12-18T02:56:31.4626457Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-11ddeba608577205.xml (deflated 38%) 2024-12-18T02:56:31.4627711Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-8fc607d1799353a3.xml (deflated 37%) 2024-12-18T02:56:31.4628991Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-af8f8c00aedcc61d.xml (deflated 37%) 2024-12-18T02:56:31.4630267Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9be848e9297ef4e5.xml (deflated 37%) 2024-12-18T02:56:31.4631636Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-6d8c0dd6cd736e35.xml (deflated 37%) 2024-12-18T02:56:31.4632878Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-59f2b87e23eb6050.xml (deflated 37%) 2024-12-18T02:56:31.4634277Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-8cd395ae5f559dfd.xml (deflated 37%) 2024-12-18T02:56:31.4635486Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7b1d2c5ecaf0a69e.xml (deflated 38%) 2024-12-18T02:56:31.4636675Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-6595b7700c16a35e.xml (deflated 37%) 2024-12-18T02:56:31.4637881Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-0cbfd995c629b0f1.xml (deflated 37%) 2024-12-18T02:56:31.4639093Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-68032bcebff839e3.xml (deflated 38%) 2024-12-18T02:56:31.4640292Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-6c863306e1607f93.xml (deflated 38%) 2024-12-18T02:56:31.4641494Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-0c2f730394c11cf2.xml (deflated 38%) 2024-12-18T02:56:31.4642705Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-15af03a83db2c3b9.xml (deflated 38%) 2024-12-18T02:56:31.4643909Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-91869c21d647f24c.xml (deflated 38%) 2024-12-18T02:56:31.4645195Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-1ad1008a30ebc625.xml (deflated 38%) 2024-12-18T02:56:31.4646384Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-5a5135f3b6b4e536.xml (deflated 37%) 2024-12-18T02:56:31.4647594Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-c9bb8e37ece056a5.xml (deflated 40%) 2024-12-18T02:56:31.4648797Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9e567713b6786bb6.xml (deflated 38%) 2024-12-18T02:56:31.4650140Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-23884acb5972e4f5.xml (deflated 38%) 2024-12-18T02:56:31.4651335Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-07d31490e289a98f.xml (deflated 38%) 2024-12-18T02:56:31.4652533Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-75797ff60c117186.xml (deflated 38%) 2024-12-18T02:56:31.4670244Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-0b4efab5f32be931.xml (deflated 38%) 2024-12-18T02:56:31.4671621Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-e83b4265a0715a59.xml (deflated 38%) 2024-12-18T02:56:31.4672758Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-b863c773a866fc2c.xml (deflated 49%) 2024-12-18T02:56:31.4673887Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-79168aa7ab9f0de9.xml (deflated 38%) 2024-12-18T02:56:31.4675036Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-80d7a84775839090.xml (deflated 38%) 2024-12-18T02:56:31.4676145Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-d21e3bd59090941e.xml (deflated 38%) 2024-12-18T02:56:31.4677276Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9176c26f81a180fc.xml (deflated 38%) 2024-12-18T02:56:31.4678401Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-b02f78580406f7d8.xml (deflated 38%) 2024-12-18T02:56:31.4679530Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-3cff8e73bc461e98.xml (deflated 38%) 2024-12-18T02:56:31.4680660Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-b9300def6d3325c5.xml (deflated 59%) 2024-12-18T02:56:31.4681791Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-3bbc243db1f90d24.xml (deflated 38%) 2024-12-18T02:56:31.4682916Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-18dfb6244589b065.xml (deflated 59%) 2024-12-18T02:56:31.4684049Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-d364affc8a641dff.xml (deflated 38%) 2024-12-18T02:56:31.4685172Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-282b07fb108ff6df.xml (deflated 37%) 2024-12-18T02:56:31.4686299Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-925ce71e7b926a4a.xml (deflated 36%) 2024-12-18T02:56:31.4687431Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-59a220c60bd5b405.xml (deflated 36%) 2024-12-18T02:56:31.4688567Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-3273dda78fc7b599.xml (deflated 36%) 2024-12-18T02:56:31.4689704Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-d42fe8a7cb8155eb.xml (deflated 36%) 2024-12-18T02:56:31.4690836Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-869b21ef9fa6c491.xml (deflated 36%) 2024-12-18T02:56:31.4692101Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-42fff57f10ddb237.xml (deflated 37%) 2024-12-18T02:56:31.4693214Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-d34a47359f681c64.xml (deflated 37%) 2024-12-18T02:56:31.4694343Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-59578ae90673bbf1.xml (deflated 38%) 2024-12-18T02:56:31.4695472Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-3b0a0661c642fbdf.xml (deflated 38%) 2024-12-18T02:56:31.4696667Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-6a8506215d202296.xml (deflated 37%) 2024-12-18T02:56:31.4697793Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-0474f3920942ba53.xml (deflated 37%) 2024-12-18T02:56:31.4699030Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-d9dbcd99a7d597ad.xml (deflated 38%) 2024-12-18T02:56:31.4700469Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-640ae26b4ec5ff47.xml (deflated 38%) 2024-12-18T02:56:31.4701745Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-352797af551c8462.xml (deflated 38%) 2024-12-18T02:56:31.4702997Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-3b693754c4341b77.xml (deflated 38%) 2024-12-18T02:56:31.4704272Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-ba4d7944a56453b2.xml (deflated 59%) 2024-12-18T02:56:31.4705541Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-00412899524f98ee.xml (deflated 37%) 2024-12-18T02:56:31.4706809Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-3618786340a3fa14.xml (deflated 36%) 2024-12-18T02:56:31.4708074Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7003b4a3c2956701.xml (deflated 36%) 2024-12-18T02:56:31.4709329Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-60e06c3e4601ba46.xml (deflated 36%) 2024-12-18T02:56:31.4710597Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-3b4d1b8a814fe340.xml (deflated 37%) 2024-12-18T02:56:31.4711921Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-4436a5fffe541a28.xml (deflated 37%) 2024-12-18T02:56:31.4713102Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-da2f2054df782f31.xml (deflated 36%) 2024-12-18T02:56:31.4714300Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7cb52c80c871313f.xml (deflated 36%) 2024-12-18T02:56:31.4715597Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-fc7d7643770beead.xml (deflated 36%) 2024-12-18T02:56:31.4716722Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2118f0eb5e15df3a.xml (deflated 36%) 2024-12-18T02:56:31.4717843Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-69c97a4322962988.xml (deflated 36%) 2024-12-18T02:56:31.4719022Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-163b304f442fe520.xml (deflated 38%) 2024-12-18T02:56:31.4720274Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-410f155e2a5ac9f1.xml (deflated 39%) 2024-12-18T02:56:31.4721501Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4e536f41bd3d06cb.xml (deflated 42%) 2024-12-18T02:56:31.4723243Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-542afd4ff1441727.xml (deflated 45%) 2024-12-18T02:56:31.4724585Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b5a96a4c1060c498.xml (deflated 43%) 2024-12-18T02:56:31.4725937Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7e803eb807c2af09.xml (deflated 42%) 2024-12-18T02:56:31.4727286Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-038f20046e26ec42.xml (deflated 43%) 2024-12-18T02:56:31.4728753Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6d794cf5a0c3c52c.xml (deflated 38%) 2024-12-18T02:56:31.4730111Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f9d312af6b63d450.xml (deflated 44%) 2024-12-18T02:56:31.4731470Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-13645c3c522138db.xml (deflated 43%) 2024-12-18T02:56:31.4732802Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ba395ba3e119020.xml (deflated 38%) 2024-12-18T02:56:31.4734151Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a15a29901d8f18b0.xml (deflated 38%) 2024-12-18T02:56:31.4735642Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-500c03e6b38a5b4a.xml (deflated 37%) 2024-12-18T02:56:31.4736881Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-49f3333fbe87a8b3.xml (deflated 37%) 2024-12-18T02:56:31.4738120Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4e0630d28c2bd69a.xml (deflated 48%) 2024-12-18T02:56:31.4739647Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a76ee87b7ab4cb2d.xml (deflated 38%) 2024-12-18T02:56:31.4741038Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e07582bfc56f9a55.xml (deflated 37%) 2024-12-18T02:56:31.4742428Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-76563b950dc328db.xml (deflated 38%) 2024-12-18T02:56:31.4743819Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-03d0f64fd192d687.xml (deflated 47%) 2024-12-18T02:56:31.4745197Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0652212fb1a46c82.xml (deflated 38%) 2024-12-18T02:56:31.4746584Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aebf49c633695920.xml (deflated 38%) 2024-12-18T02:56:31.4747973Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5ad098264dad944e.xml (deflated 37%) 2024-12-18T02:56:31.4749369Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6bea477a65f12ac2.xml (deflated 38%) 2024-12-18T02:56:31.4750773Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5cffebc5dcd5dc12.xml (deflated 37%) 2024-12-18T02:56:31.4752182Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9f838e32857aea7d.xml (deflated 37%) 2024-12-18T02:56:31.4753420Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ca23d00ec76b165e.xml (deflated 38%) 2024-12-18T02:56:31.4754655Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-45c01a6b64cb6734.xml (deflated 37%) 2024-12-18T02:56:31.4755955Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8138e5f943e1e9c5.xml (deflated 45%) 2024-12-18T02:56:31.4757187Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0b4d51859e371606.xml (deflated 38%) 2024-12-18T02:56:31.4758425Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3997bcb6e8cc992c.xml (deflated 39%) 2024-12-18T02:56:31.4759713Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-97e9c8216b1ffe89.xml (deflated 42%) 2024-12-18T02:56:31.4760950Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-60d42ab9fcf0de9b.xml (deflated 45%) 2024-12-18T02:56:31.4762189Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-39a87f2748ec2d5a.xml (deflated 43%) 2024-12-18T02:56:31.4763421Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b1261254afc58610.xml (deflated 43%) 2024-12-18T02:56:31.4764654Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8d1924ebea224e2f.xml (deflated 43%) 2024-12-18T02:56:31.4765878Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-006785eb40831012.xml (deflated 38%) 2024-12-18T02:56:31.4767116Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0904ced601673030.xml (deflated 44%) 2024-12-18T02:56:31.4768348Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aad299c39f645a62.xml (deflated 44%) 2024-12-18T02:56:31.4769586Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a45c2d541d23da75.xml (deflated 38%) 2024-12-18T02:56:31.4770819Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17ac78e7d677b629.xml (deflated 37%) 2024-12-18T02:56:31.4772049Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-24518bb6361d8446.xml (deflated 37%) 2024-12-18T02:56:31.4773286Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-97a9388e29a9465f.xml (deflated 37%) 2024-12-18T02:56:31.4774516Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ead74316698330b5.xml (deflated 48%) 2024-12-18T02:56:31.4775739Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bf112c2746fa7579.xml (deflated 37%) 2024-12-18T02:56:31.4776969Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-384bc89ee0470ba4.xml (deflated 37%) 2024-12-18T02:56:31.4778205Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b2b4db5b6088fcf5.xml (deflated 37%) 2024-12-18T02:56:31.4779728Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-40046bf241b1acdb.xml (deflated 47%) 2024-12-18T02:56:31.4781126Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a7d29a4de2efe46d.xml (deflated 38%) 2024-12-18T02:56:31.4782533Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-651996a17b6cff7c.xml (deflated 37%) 2024-12-18T02:56:31.4783925Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d0f0e29828d908c4.xml (deflated 37%) 2024-12-18T02:56:31.4785405Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2db4ae106f3b89dd.xml (deflated 38%) 2024-12-18T02:56:31.4786791Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e28efcb3280b74cf.xml (deflated 37%) 2024-12-18T02:56:31.4788190Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a59dddc263d6b207.xml (deflated 37%) 2024-12-18T02:56:31.4789586Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a3ed5a12e5f40dde.xml (deflated 37%) 2024-12-18T02:56:31.4791149Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d0f235a3d5cc2571.xml (deflated 38%) 2024-12-18T02:56:31.4792522Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-de4ec65186491329.xml (deflated 45%) 2024-12-18T02:56:31.4793758Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-656b037e704189b1.xml (deflated 58%) 2024-12-18T02:56:31.4794997Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-223ef968dc58015f.xml (deflated 37%) 2024-12-18T02:56:31.4796227Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-43ed2e7e785b3ed8.xml (deflated 45%) 2024-12-18T02:56:31.4797449Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8803fae689b08f82.xml (deflated 46%) 2024-12-18T02:56:31.4798686Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-74ad7bed2ade9957.xml (deflated 43%) 2024-12-18T02:56:31.4799924Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-da995ed662e09e1d.xml (deflated 37%) 2024-12-18T02:56:31.4801164Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c324a49b52f5735d.xml (deflated 38%) 2024-12-18T02:56:31.4802398Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0e9db36851c946a0.xml (deflated 43%) 2024-12-18T02:56:31.4803638Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2e50d2fa9ac843ba.xml (deflated 42%) 2024-12-18T02:56:31.4804881Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a16bfec511dd692c.xml (deflated 38%) 2024-12-18T02:56:31.4806123Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2847fd5462eff977.xml (deflated 44%) 2024-12-18T02:56:31.4807350Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-53ad4e5ea968f2a9.xml (deflated 44%) 2024-12-18T02:56:31.4808591Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e16675fa8add69c4.xml (deflated 38%) 2024-12-18T02:56:31.4809827Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-975dfec81e3580bc.xml (deflated 38%) 2024-12-18T02:56:31.4811059Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c032e1dae38e8652.xml (deflated 37%) 2024-12-18T02:56:31.4812297Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9bc12a610e9814a2.xml (deflated 37%) 2024-12-18T02:56:31.4813536Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c60c39bda605d26d.xml (deflated 37%) 2024-12-18T02:56:31.4814780Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c4caef09e7e4ba3d.xml (deflated 38%) 2024-12-18T02:56:31.4816076Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9a4cf75889cd1c8f.xml (deflated 38%) 2024-12-18T02:56:31.4817312Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-78a6dd3faa77c576.xml (deflated 38%) 2024-12-18T02:56:31.4818543Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-920bbde9a6a58981.xml (deflated 37%) 2024-12-18T02:56:31.4820102Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-91a6443c14687800.xml (deflated 37%) 2024-12-18T02:56:31.4821554Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-932a1514564fe6c3.xml (deflated 38%) 2024-12-18T02:56:31.4823116Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9d1500b90f3ee2c3.xml (deflated 37%) 2024-12-18T02:56:31.4824518Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e2dafcc4c789316b.xml (deflated 38%) 2024-12-18T02:56:31.4825917Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5fac61007a9c0ce9.xml (deflated 37%) 2024-12-18T02:56:31.4827316Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d7310e6063f52316.xml (deflated 38%) 2024-12-18T02:56:31.4828708Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8bef803ea0f18566.xml (deflated 37%) 2024-12-18T02:56:31.4830097Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8bf305051f5e7e3a.xml (deflated 44%) 2024-12-18T02:56:31.4831588Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e6562eb65b25654b.xml (deflated 43%) 2024-12-18T02:56:31.4832944Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0b940de118d8e938.xml (deflated 37%) 2024-12-18T02:56:31.4834460Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c089cd8a8986f7f4.xml (deflated 43%) 2024-12-18T02:56:31.4835698Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c54c7742b4ec1d07.xml (deflated 43%) 2024-12-18T02:56:31.4836934Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a9b7d4a678aac8d7.xml (deflated 38%) 2024-12-18T02:56:31.4838169Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-369b6c1b93b37ef4.xml (deflated 43%) 2024-12-18T02:56:31.4839403Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ca112fbd80ca1587.xml (deflated 45%) 2024-12-18T02:56:31.4840631Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e920f923518b7672.xml (deflated 45%) 2024-12-18T02:56:31.4841866Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a60e34ebc86340bb.xml (deflated 58%) 2024-12-18T02:56:31.4843089Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a21c500b5e356c94.xml (deflated 37%) 2024-12-18T02:56:31.4844316Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ef82ef9e90fbbf83.xml (deflated 46%) 2024-12-18T02:56:31.4845553Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-edb5d6d46615b520.xml (deflated 45%) 2024-12-18T02:56:31.4846772Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a9c05c438fb7d52c.xml (deflated 42%) 2024-12-18T02:56:31.4848094Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e59000399d271fcd.xml (deflated 37%) 2024-12-18T02:56:31.4849318Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-be2d5c0fbd4e75de.xml (deflated 38%) 2024-12-18T02:56:31.4850548Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f566e84db5fbe61c.xml (deflated 43%) 2024-12-18T02:56:31.4851853Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ed83535c4cceead4.xml (deflated 42%) 2024-12-18T02:56:31.4853082Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-308c96e25ba62cec.xml (deflated 38%) 2024-12-18T02:56:31.4854313Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f525e7396055a336.xml (deflated 45%) 2024-12-18T02:56:31.4855541Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dc19f0b1c5ae1460.xml (deflated 44%) 2024-12-18T02:56:31.4856772Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f7d7dd7152a00351.xml (deflated 38%) 2024-12-18T02:56:31.4857990Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d0013a7cd43d0383.xml (deflated 38%) 2024-12-18T02:56:31.4859475Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e9eb0d04f219f469.xml (deflated 37%) 2024-12-18T02:56:31.4860860Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2c64a5697bc68b2c.xml (deflated 37%) 2024-12-18T02:56:31.4862251Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3eaa3c64ee54fb3a.xml (deflated 37%) 2024-12-18T02:56:31.4863636Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2bf340e84a121da1.xml (deflated 38%) 2024-12-18T02:56:31.4865015Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1ed706f2ff70a264.xml (deflated 38%) 2024-12-18T02:56:31.4866397Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4e28953b32a6e2b8.xml (deflated 38%) 2024-12-18T02:56:31.4867776Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-64856d285999d5c0.xml (deflated 38%) 2024-12-18T02:56:31.4869149Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6065b5df009a9eda.xml (deflated 38%) 2024-12-18T02:56:31.4870530Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5fa06011b31b4619.xml (deflated 38%) 2024-12-18T02:56:31.4871948Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cdc5b2ca98346260.xml (deflated 37%) 2024-12-18T02:56:31.4873170Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ec42d440a0a78e32.xml (deflated 38%) 2024-12-18T02:56:31.4874391Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c97f24bd5061ec72.xml (deflated 37%) 2024-12-18T02:56:31.4875617Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a4c5551a1276508c.xml (deflated 38%) 2024-12-18T02:56:31.4876843Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b51ffc0c76a4ce62.xml (deflated 37%) 2024-12-18T02:56:31.4878131Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a64ce1e03186d77d.xml (deflated 44%) 2024-12-18T02:56:31.4879346Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c1e5cc9f71c90b09.xml (deflated 44%) 2024-12-18T02:56:31.4880577Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bb9f9b57be15ba57.xml (deflated 37%) 2024-12-18T02:56:31.4881797Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3d941c2753d437c4.xml (deflated 43%) 2024-12-18T02:56:31.4883093Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9d4e730b177d832d.xml (deflated 43%) 2024-12-18T02:56:31.4884323Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6c229f9579317d5e.xml (deflated 37%) 2024-12-18T02:56:31.4885554Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6a6d43a2a7a545c6.xml (deflated 43%) 2024-12-18T02:56:31.4886785Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-41fa418ca9ddf88b.xml (deflated 45%) 2024-12-18T02:56:31.4888020Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6e6735f948eba2a3.xml (deflated 45%) 2024-12-18T02:56:31.4889241Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7ffad14a9927ea2e.xml (deflated 37%) 2024-12-18T02:56:31.4890472Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ff72773a981ea689.xml (deflated 38%) 2024-12-18T02:56:31.4891704Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fccc3b01dcafbba8.xml (deflated 44%) 2024-12-18T02:56:31.4892941Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b6e2b3a2a9d37300.xml (deflated 43%) 2024-12-18T02:56:31.4894167Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a13d983c08feecbe.xml (deflated 38%) 2024-12-18T02:56:31.4895394Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-51e7b00dad4f8482.xml (deflated 44%) 2024-12-18T02:56:31.4896628Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0d24bb30fe7b1fbe.xml (deflated 37%) 2024-12-18T02:56:31.4897850Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-529714745803f326.xml (deflated 38%) 2024-12-18T02:56:31.4899337Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-71b349748bf2302e.xml (deflated 44%) 2024-12-18T02:56:31.4900729Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cf7867383be23402.xml (deflated 43%) 2024-12-18T02:56:31.4902112Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-877926219e0fb791.xml (deflated 37%) 2024-12-18T02:56:31.4903492Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-20f73fc25837e352.xml (deflated 38%) 2024-12-18T02:56:31.4904871Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c32742d0ab139da6.xml (deflated 37%) 2024-12-18T02:56:31.4906267Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dd112648eadcdd32.xml (deflated 37%) 2024-12-18T02:56:31.4907649Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eb9c3e10b5929150.xml (deflated 44%) 2024-12-18T02:56:31.4909119Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3196f4cbcc2ec76d.xml (deflated 37%) 2024-12-18T02:56:31.4910495Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f75d20751bde4493.xml (deflated 44%) 2024-12-18T02:56:31.4911951Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-092f75c56f12fac7.xml (deflated 44%) 2024-12-18T02:56:31.4913260Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a08fe424c4cd2456.xml (deflated 37%) 2024-12-18T02:56:31.4914626Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8eed95fd55c549c5.xml (deflated 37%) 2024-12-18T02:56:31.4915933Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eae45e31262fd347.xml (deflated 42%) 2024-12-18T02:56:31.4917231Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6454f10c4610c0b9.xml (deflated 43%) 2024-12-18T02:56:31.4918529Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d66758eaa4726aac.xml (deflated 43%) 2024-12-18T02:56:31.4919920Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-db907232d7ab17c0.xml (deflated 45%) 2024-12-18T02:56:31.4921146Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1af0ce4f912556fe.xml (deflated 38%) 2024-12-18T02:56:31.4922694Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-93b47ddf527ed77d.xml (deflated 38%) 2024-12-18T02:56:31.4924044Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5c430df21fa15e43.xml (deflated 37%) 2024-12-18T02:56:31.4925396Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5d778c726332df74.xml (deflated 38%) 2024-12-18T02:56:31.4926744Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6d3be20926364ce0.xml (deflated 44%) 2024-12-18T02:56:31.4928080Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6d4690fd6305df90.xml (deflated 43%) 2024-12-18T02:56:31.4929428Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-010f33b5ea30fbf2.xml (deflated 38%) 2024-12-18T02:56:31.4930776Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aa4931bf3e4e8ba4.xml (deflated 44%) 2024-12-18T02:56:31.4932126Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-acdc9dadcb1f43b2.xml (deflated 37%) 2024-12-18T02:56:31.4933465Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e6b5c3d6b05a9234.xml (deflated 38%) 2024-12-18T02:56:31.4934804Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b9c7fb275099ea3c.xml (deflated 45%) 2024-12-18T02:56:31.4936247Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-657336a02f5bd890.xml (deflated 43%) 2024-12-18T02:56:31.4937551Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-caa6ace9048fbf15.xml (deflated 37%) 2024-12-18T02:56:31.4938908Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2da144cf79e9aef4.xml (deflated 38%) 2024-12-18T02:56:31.4940448Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-38e64993937c9c10.xml (deflated 37%) 2024-12-18T02:56:31.4942314Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-655a061439a71d8c.xml (deflated 37%) 2024-12-18T02:56:31.4943701Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9ebeca45aa5055a4.xml (deflated 44%) 2024-12-18T02:56:31.4945075Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8b8eb8d07a027bbd.xml (deflated 37%) 2024-12-18T02:56:31.4946534Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cc5e9b1cbc9a3dea.xml (deflated 43%) 2024-12-18T02:56:31.4947925Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-628960c8bda20a33.xml (deflated 44%) 2024-12-18T02:56:31.4949304Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-12c2bd02ca986fa9.xml (deflated 38%) 2024-12-18T02:56:31.4950686Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fd14cef8c9ea3914.xml (deflated 37%) 2024-12-18T02:56:31.4952186Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cee47e6dc764326f.xml (deflated 43%) 2024-12-18T02:56:31.4953409Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c24c96dff410a4b6.xml (deflated 43%) 2024-12-18T02:56:31.4954634Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b17b9ce84cbafc16.xml (deflated 43%) 2024-12-18T02:56:31.4955854Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c0c20f20bb1b9063.xml (deflated 45%) 2024-12-18T02:56:31.4957082Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0214cb19c2b468ed.xml (deflated 38%) 2024-12-18T02:56:31.4958308Z adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c3bd1c31ff48d95c.xml (deflated 37%) 2024-12-18T02:56:31.4959540Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-804776cb512cca0c.xml (deflated 38%) 2024-12-18T02:56:31.4960759Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0791b1d661a00532.xml (deflated 39%) 2024-12-18T02:56:31.4961988Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-96a1eaaa9710baa3.xml (deflated 42%) 2024-12-18T02:56:31.4963214Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-266a217765f3d110.xml (deflated 37%) 2024-12-18T02:56:31.4964439Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-207546e54c8aa9ba.xml (deflated 37%) 2024-12-18T02:56:31.4965659Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-94453fec9b3c8d95.xml (deflated 49%) 2024-12-18T02:56:31.4966886Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4222d75d84c2aafe.xml (deflated 43%) 2024-12-18T02:56:31.4968117Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0918eeff1bc87344.xml (deflated 45%) 2024-12-18T02:56:31.4969351Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-24ef2e0e54c0b727.xml (deflated 44%) 2024-12-18T02:56:31.4970578Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d315f6911bd6a775.xml (deflated 49%) 2024-12-18T02:56:31.4971860Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b9b2506101878912.xml (deflated 38%) 2024-12-18T02:56:31.4973092Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7d3dd17d280eab83.xml (deflated 38%) 2024-12-18T02:56:31.4974309Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c7f1e970549f28b3.xml (deflated 37%) 2024-12-18T02:56:31.4975525Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e84de0f82ff84275.xml (deflated 37%) 2024-12-18T02:56:31.4976804Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ff5a75d771e79806.xml (deflated 48%) 2024-12-18T02:56:31.4978034Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2f40ed82ad13feb2.xml (deflated 38%) 2024-12-18T02:56:31.4979527Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-096e2ffb2aecdb28.xml (deflated 37%) 2024-12-18T02:56:31.4980972Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5c869fa03438ae3d.xml (deflated 38%) 2024-12-18T02:56:31.4982347Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f93606f5177c8bc3.xml (deflated 47%) 2024-12-18T02:56:31.4983741Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7993d402a9d24860.xml (deflated 38%) 2024-12-18T02:56:31.4985144Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-75b0e8f5cb5826d7.xml (deflated 37%) 2024-12-18T02:56:31.4986538Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8662e93adb867634.xml (deflated 37%) 2024-12-18T02:56:31.4987938Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4ba8a412e043e1ae.xml (deflated 45%) 2024-12-18T02:56:31.4989334Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-91829d3d626cc19f.xml (deflated 37%) 2024-12-18T02:56:31.4990734Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2b1b1cf880f6783d.xml (deflated 37%) 2024-12-18T02:56:31.4992148Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f0d3f5a59fd5e33f.xml (deflated 37%) 2024-12-18T02:56:31.4993385Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-052d1c15f0cd7715.xml (deflated 37%) 2024-12-18T02:56:31.4994623Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3d986396c1a45669.xml (deflated 38%) 2024-12-18T02:56:31.4995858Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-358c6d9548b39241.xml (deflated 38%) 2024-12-18T02:56:31.4997100Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-306feb2efe3f6805.xml (deflated 39%) 2024-12-18T02:56:31.4998387Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6a793c2fd066180c.xml (deflated 42%) 2024-12-18T02:56:31.4999629Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-48a35a210f8f494d.xml (deflated 37%) 2024-12-18T02:56:31.5000879Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-624fe1723e981b2b.xml (deflated 37%) 2024-12-18T02:56:31.5002123Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-06a3dfb04e91e16b.xml (deflated 49%) 2024-12-18T02:56:31.5003412Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2dc7126a018b2bfc.xml (deflated 43%) 2024-12-18T02:56:31.5004657Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4e71c60fdc4980ed.xml (deflated 45%) 2024-12-18T02:56:31.5005901Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-eb4b6e84d52097ca.xml (deflated 44%) 2024-12-18T02:56:31.5007147Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-becd1223bf8d77dc.xml (deflated 49%) 2024-12-18T02:56:31.5008457Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-34120de046964fe6.xml (deflated 38%) 2024-12-18T02:56:31.5009704Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5cebd96a7bbc169b.xml (deflated 38%) 2024-12-18T02:56:31.5010954Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3ab96a78eacb7b89.xml (deflated 37%) 2024-12-18T02:56:31.5012199Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e0848b33e19f5a76.xml (deflated 37%) 2024-12-18T02:56:31.5013425Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1956bb75cf1364e8.xml (deflated 48%) 2024-12-18T02:56:31.5014667Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-876b9223d5369115.xml (deflated 38%) 2024-12-18T02:56:31.5015916Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b94e397e770a895b.xml (deflated 37%) 2024-12-18T02:56:31.5017159Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-75d7491c88b70954.xml (deflated 38%) 2024-12-18T02:56:31.5018404Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9fff816245ffe22c.xml (deflated 47%) 2024-12-18T02:56:31.5019958Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-43a628eec42a6db5.xml (deflated 38%) 2024-12-18T02:56:31.5021398Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-876f91bf19bb9c31.xml (deflated 37%) 2024-12-18T02:56:31.5023001Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6bb9fbd25d0a7521.xml (deflated 37%) 2024-12-18T02:56:31.5024400Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bd76df6cc822bc00.xml (deflated 45%) 2024-12-18T02:56:31.5025811Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6964b8ba08945db1.xml (deflated 37%) 2024-12-18T02:56:31.5027223Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bf1f6a8c01cdf0e7.xml (deflated 37%) 2024-12-18T02:56:31.5028630Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f72ed0e3b3faae32.xml (deflated 37%) 2024-12-18T02:56:31.5030036Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cfaae96e62309bef.xml (deflated 38%) 2024-12-18T02:56:31.5031538Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0a83ab8abeb101c2.xml (deflated 38%) 2024-12-18T02:56:31.5032900Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1fa23d706d92c737.xml (deflated 58%) 2024-12-18T02:56:31.5034364Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-663fac3a7d72307c.xml (deflated 37%) 2024-12-18T02:56:31.5035693Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-27817b30e8d80e3a.xml (deflated 37%) 2024-12-18T02:56:31.5036938Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-53d8ee003e414379.xml (deflated 38%) 2024-12-18T02:56:31.5038181Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f8acb8c819a0b1f6.xml (deflated 37%) 2024-12-18T02:56:31.5039497Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-57ba8906a7de4b8c.xml (deflated 43%) 2024-12-18T02:56:31.5040740Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-560bb2f2f1612345.xml (deflated 44%) 2024-12-18T02:56:31.5041986Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5c9dc5bc340dcfa3.xml (deflated 38%) 2024-12-18T02:56:31.5043240Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-aaa9df89191990f4.xml (deflated 37%) 2024-12-18T02:56:31.5044487Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-079ddaa1fc665e32.xml (deflated 38%) 2024-12-18T02:56:31.5045730Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6cd797dad5132064.xml (deflated 44%) 2024-12-18T02:56:31.5046965Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-35efb686c84c1f30.xml (deflated 44%) 2024-12-18T02:56:31.5048205Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1e4e214ce3c50da0.xml (deflated 42%) 2024-12-18T02:56:31.5049450Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-60464ab084e6bceb.xml (deflated 42%) 2024-12-18T02:56:31.5050696Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-925ccc0d32f2901b.xml (deflated 37%) 2024-12-18T02:56:31.5051935Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-45069194bf6855e4.xml (deflated 38%) 2024-12-18T02:56:31.5053173Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bcc11e5cd34f2c6c.xml (deflated 37%) 2024-12-18T02:56:31.5054424Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a1a96c6c883b3139.xml (deflated 38%) 2024-12-18T02:56:31.5055661Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-24b17b283966348a.xml (deflated 38%) 2024-12-18T02:56:31.5056891Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-37cdba786bb5eae9.xml (deflated 38%) 2024-12-18T02:56:31.5058136Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8caa0338a96e1770.xml (deflated 37%) 2024-12-18T02:56:31.5059658Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-05e6f4ee96727ada.xml (deflated 38%) 2024-12-18T02:56:31.5061119Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7331bd0b5991284f.xml (deflated 37%) 2024-12-18T02:56:31.5062524Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7210ade3fff64f27.xml (deflated 37%) 2024-12-18T02:56:31.5063927Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4e1efbcb4cf71509.xml (deflated 43%) 2024-12-18T02:56:31.5065397Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-87c8bc74b68d928b.xml (deflated 37%) 2024-12-18T02:56:31.5066797Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-129a15dda1a7fcff.xml (deflated 38%) 2024-12-18T02:56:31.5068192Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e473ed52abcb39d9.xml (deflated 37%) 2024-12-18T02:56:31.5069592Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cb6991f0b7df1d14.xml (deflated 37%) 2024-12-18T02:56:31.5071051Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-dcf6516713eefcb8.xml (deflated 38%) 2024-12-18T02:56:31.5072434Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9ff273d0b58c2025.xml (deflated 37%) 2024-12-18T02:56:31.5073678Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6c3c32b6757f5ee7.xml (deflated 49%) 2024-12-18T02:56:31.5074914Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8b30eea96dfa9075.xml (deflated 38%) 2024-12-18T02:56:31.5076143Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f30e495ba47f0937.xml (deflated 44%) 2024-12-18T02:56:31.5077378Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9fce3ad9d2859daa.xml (deflated 38%) 2024-12-18T02:56:31.5078613Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4dea95712d10b721.xml (deflated 37%) 2024-12-18T02:56:31.5079832Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4d529db6728bdc67.xml (deflated 38%) 2024-12-18T02:56:31.5081054Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-22a96b7121837e74.xml (deflated 58%) 2024-12-18T02:56:31.5082287Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bc6d4764715b7f3b.xml (deflated 37%) 2024-12-18T02:56:31.5083524Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-22dadcf6f8b75c42.xml (deflated 37%) 2024-12-18T02:56:31.5084756Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-46b141de31c0b32d.xml (deflated 38%) 2024-12-18T02:56:31.5085991Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4f6cb9fa5e93f712.xml (deflated 38%) 2024-12-18T02:56:31.5087224Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d99c70a2c8b18319.xml (deflated 43%) 2024-12-18T02:56:31.5088450Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ac388bb4600d0115.xml (deflated 44%) 2024-12-18T02:56:31.5089684Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-01ee6a41dc832fc6.xml (deflated 38%) 2024-12-18T02:56:31.5090916Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0ca50df4919fc2ee.xml (deflated 38%) 2024-12-18T02:56:31.5092152Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fe88f8b7385fdb32.xml (deflated 38%) 2024-12-18T02:56:31.5093384Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-97ba41d78b0d45c8.xml (deflated 44%) 2024-12-18T02:56:31.5094615Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e6a75b15bb6b7e38.xml (deflated 44%) 2024-12-18T02:56:31.5095910Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5e43d5df8c06b5f9.xml (deflated 42%) 2024-12-18T02:56:31.5097141Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-48612e553f0dfc39.xml (deflated 42%) 2024-12-18T02:56:31.5098363Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-08fa2aa6487ff0fd.xml (deflated 37%) 2024-12-18T02:56:31.5099870Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cf561743d607cea4.xml (deflated 37%) 2024-12-18T02:56:31.5101312Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b13b09e8b8535a44.xml (deflated 37%) 2024-12-18T02:56:31.5102703Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-2c996847f19b2011.xml (deflated 38%) 2024-12-18T02:56:31.5104086Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-99dc18432b35ab24.xml (deflated 38%) 2024-12-18T02:56:31.5105465Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-be90147d065ecb11.xml (deflated 38%) 2024-12-18T02:56:31.5106847Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-918f77e30f238ef9.xml (deflated 37%) 2024-12-18T02:56:31.5108230Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c336ba1c79670f7e.xml (deflated 37%) 2024-12-18T02:56:31.5109611Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-901d8cf862e9218d.xml (deflated 38%) 2024-12-18T02:56:31.5110991Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-a465e36f235ad326.xml (deflated 37%) 2024-12-18T02:56:31.5112481Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-489e3bd035692656.xml (deflated 43%) 2024-12-18T02:56:31.5113709Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9a21a9c32d3f3f36.xml (deflated 37%) 2024-12-18T02:56:31.5114935Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ea6b73efa4ca83ed.xml (deflated 38%) 2024-12-18T02:56:31.5116162Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-781bf9c78483526c.xml (deflated 37%) 2024-12-18T02:56:31.5117394Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e58e9c9af8bbd509.xml (deflated 37%) 2024-12-18T02:56:31.5118624Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-0d6c526306cd82dc.xml (deflated 38%) 2024-12-18T02:56:31.5119845Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-146df3781e764b8d.xml (deflated 37%) 2024-12-18T02:56:31.5121071Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-caad8b846c5b72c3.xml (deflated 49%) 2024-12-18T02:56:31.5122666Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-83c25092808a1529.xml (deflated 37%) 2024-12-18T02:56:31.5124007Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-9b660bf360c49e02.xml (deflated 45%) 2024-12-18T02:56:31.5125353Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-54c29cc43880813e.xml (deflated 38%) 2024-12-18T02:56:31.5126700Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8a8fea9d4c4b3d20.xml (deflated 37%) 2024-12-18T02:56:31.5128160Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-7300b49789b1cbb3.xml (deflated 38%) 2024-12-18T02:56:31.5129491Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-29058df81e4386f0.xml (deflated 37%) 2024-12-18T02:56:31.5130824Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-b14a98dcda427f04.xml (deflated 38%) 2024-12-18T02:56:31.5132165Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-beaf0dc730d2ea18.xml (deflated 38%) 2024-12-18T02:56:31.5133567Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1884c28145237656.xml (deflated 43%) 2024-12-18T02:56:31.5134978Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d844dc40a90e7ef9.xml (deflated 43%) 2024-12-18T02:56:31.5136213Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-17b0180bc281cfa9.xml (deflated 44%) 2024-12-18T02:56:31.5137440Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-4e389225e07c747e.xml (deflated 44%) 2024-12-18T02:56:31.5138715Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e295163469b61851.xml (deflated 44%) 2024-12-18T02:56:31.5140227Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-10927a7b76e6f85f.xml (deflated 50%) 2024-12-18T02:56:31.5141616Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-679a8ba1dd7f56fe.xml (deflated 49%) 2024-12-18T02:56:31.5142990Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c2963b7957c832ce.xml (deflated 37%) 2024-12-18T02:56:31.5144376Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-603c7020d48c0e98.xml (deflated 38%) 2024-12-18T02:56:31.5145762Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d955c76f5351904c.xml (deflated 37%) 2024-12-18T02:56:31.5147145Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-703839de3d5592eb.xml (deflated 38%) 2024-12-18T02:56:31.5148540Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cc19a32a00b69d30.xml (deflated 44%) 2024-12-18T02:56:31.5149923Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-be62818975618ae8.xml (deflated 38%) 2024-12-18T02:56:31.5151421Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-73805a9211e0c30d.xml (deflated 38%) 2024-12-18T02:56:31.5152763Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6186d73dc31c276c.xml (deflated 37%) 2024-12-18T02:56:31.5153983Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-bb42d959f2546926.xml (deflated 38%) 2024-12-18T02:56:31.5155215Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c6f7bb3eb1fb06d3.xml (deflated 37%) 2024-12-18T02:56:31.5156453Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-56eb8338509ba5b9.xml (deflated 37%) 2024-12-18T02:56:31.5157682Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-21fa499f54281639.xml (deflated 49%) 2024-12-18T02:56:31.5159036Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-5b9a465f91d665d4.xml (deflated 49%) 2024-12-18T02:56:31.5160266Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-f2107dceb6574b03.xml (deflated 38%) 2024-12-18T02:56:31.5161500Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ebcf1291b3b7780f.xml (deflated 38%) 2024-12-18T02:56:31.5162732Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c75b72c479d63a3b.xml (deflated 37%) 2024-12-18T02:56:31.5164002Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-fc36c7a038f4f036.xml (deflated 37%) 2024-12-18T02:56:31.5165233Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-98c3cad9c61019bc.xml (deflated 38%) 2024-12-18T02:56:31.5166460Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-73514e77e3301442.xml (deflated 38%) 2024-12-18T02:56:31.5167681Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-863bb03a3e871129.xml (deflated 43%) 2024-12-18T02:56:31.5168905Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6d84cbb35790dcc4.xml (deflated 43%) 2024-12-18T02:56:31.5170129Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-11651bf3dbe88b0e.xml (deflated 44%) 2024-12-18T02:56:31.5171363Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-abb78507beca5172.xml (deflated 44%) 2024-12-18T02:56:31.5172592Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-cbc314f0aa8f5d6c.xml (deflated 44%) 2024-12-18T02:56:31.5173818Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d223c25b6fd12cd8.xml (deflated 50%) 2024-12-18T02:56:31.5175052Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-8ded8b8fcb0a13f3.xml (deflated 49%) 2024-12-18T02:56:31.5176275Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c87c795811135911.xml (deflated 37%) 2024-12-18T02:56:31.5177493Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-3df2d40c0649c85d.xml (deflated 38%) 2024-12-18T02:56:31.5178772Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-70dddd87d4cfc1f5.xml (deflated 37%) 2024-12-18T02:56:31.5180288Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-e94222720259d4ac.xml (deflated 37%) 2024-12-18T02:56:31.5181674Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-56ffbf593051e84f.xml (deflated 44%) 2024-12-18T02:56:31.5183067Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-d3ab3dbe3abf8e47.xml (deflated 37%) 2024-12-18T02:56:31.5184454Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-09d09e5f7bd33e6f.xml (deflated 38%) 2024-12-18T02:56:31.5185842Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-898f78c57b7cfae2.xml (deflated 37%) 2024-12-18T02:56:31.5187237Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-41ebe1dab89825d0.xml (deflated 37%) 2024-12-18T02:56:31.5188628Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-ef5deeb3962a69fd.xml (deflated 37%) 2024-12-18T02:56:31.5190081Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-399d4ee8509a201f.xml (deflated 37%) 2024-12-18T02:56:31.5191659Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-c8819546434ccce3.xml (deflated 49%) 2024-12-18T02:56:31.5192889Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-6e8d623de0d791d4.xml (deflated 49%) 2024-12-18T02:56:31.5194122Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-046b9aa6ee2c5ab2.xml (deflated 38%) 2024-12-18T02:56:31.5195400Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-1292d8faed7d9850.xml (deflated 38%) 2024-12-18T02:56:31.5196627Z adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/distributed.test_distributed_spawn-42c8b43f9be9a7a9.xml (deflated 37%) 2024-12-18T02:56:31.5227765Z ##[group]Run # Remove any previous usage logs if they exist 2024-12-18T02:56:31.5228303Z # Remove any previous usage logs if they exist 2024-12-18T02:56:31.5228731Z rm -f logs-*.zip 2024-12-18T02:56:31.5229272Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-12-18T02:56:31.5229900Z # so check to see if the file exists first 2024-12-18T02:56:31.5230298Z if [ -f 'usage_log.txt' ]; then 2024-12-18T02:56:31.5230726Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-12-18T02:56:31.5231233Z fi 2024-12-18T02:56:31.5231654Z if find "test/test-reports" -name "*.log" 2>/dev/null | grep -q .; then 2024-12-18T02:56:31.5232275Z  zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' 2024-12-18T02:56:31.5232718Z fi 2024-12-18T02:56:31.5237974Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:56:31.5238369Z env: 2024-12-18T02:56:31.5238588Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:56:31.5238924Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:56:31.5239480Z DOCKER_CONTAINER_ID: 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T02:56:31.5240155Z FILE_SUFFIX: test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_34566462868 2024-12-18T02:56:31.5240629Z ##[endgroup] 2024-12-18T02:56:31.5722849Z adding: usage_log.txt (deflated 97%) 2024-12-18T02:56:31.5794057Z adding: test/test-reports/distributed._tools.test_runtime_estimator_1.1_85fe516bc8b708a2_.log (deflated 59%) 2024-12-18T02:56:31.5797367Z adding: test/test-reports/distributed.test_device_mesh_1.1_20328142d33f817f_.log (deflated 90%) 2024-12-18T02:56:31.5798451Z adding: test/test-reports/distributed._tensor.test_experimental_ops_1.1_824e98994873664b_.log (deflated 61%) 2024-12-18T02:56:31.5800280Z adding: test/test-reports/distributed._tensor.test_tensor_ops_1.1_655f3bc36620a502_.log (deflated 91%) 2024-12-18T02:56:31.5801325Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_extensions_1.1_085a69f5359d19d6_.log (deflated 73%) 2024-12-18T02:56:31.5804213Z adding: test/test-reports/distributed.fsdp.test_wrap_1.1_1cbb7aec53fa7ba0_.log (deflated 93%) 2024-12-18T02:56:31.5805230Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_autograd_1.1_7b8f662490ea681d_.log (deflated 72%) 2024-12-18T02:56:31.5806582Z adding: test/test-reports/distributed.fsdp.test_fsdp_clip_grad_norm_1.1_f28d8deddf23c59c_.log (deflated 91%) 2024-12-18T02:56:31.5808657Z adding: test/test-reports/distributed.fsdp.test_fsdp_misc_1.1_c9f4bc7a3a7dc932_.log (deflated 84%) 2024-12-18T02:56:31.5812177Z adding: test/test-reports/distributed.test_c10d_common_1.1_e947e4b06e84b5f7_.log (deflated 94%) 2024-12-18T02:56:31.5813078Z adding: test/test-reports/distributed._tools.test_mod_tracker_1.1_44cf98526b1f1078_.log (deflated 63%) 2024-12-18T02:56:31.5814751Z adding: test/test-reports/distributed.pipelining.test_schedule_multiproc_1.1_0bc390351f80e4ff_.log (deflated 91%) 2024-12-18T02:56:31.5816068Z adding: test/test-reports/distributed.tensor.parallel.test_micro_pipeline_tp_1.1_29ea31d878dfd070_.log (deflated 87%) 2024-12-18T02:56:31.5817126Z adding: test/test-reports/distributed.checkpoint.test_nested_dict_1.1_9ddf00a6a608549f_.log (deflated 59%) 2024-12-18T02:56:31.5818130Z adding: test/test-reports/distributed.pipelining.test_transformer_1.1_a8a30047e9c1a9b5_.log (deflated 54%) 2024-12-18T02:56:31.5819407Z adding: test/test-reports/distributed.pipelining.test_backward_1.1_9682a21479527d48_.log (deflated 66%) 2024-12-18T02:56:31.5820547Z adding: test/test-reports/distributed._shard.sharded_tensor.test_logger_1.1_d94a14034fee1f41_.log (deflated 55%) 2024-12-18T02:56:31.5821573Z adding: test/test-reports/distributed.pipelining.test_pipe_1.1_1a85a1dad4f76019_.log (deflated 61%) 2024-12-18T02:56:31.5822721Z adding: test/test-reports/distributed._tools.test_sac_estimator_1.1_efac988a30c89dc4_.log (deflated 59%) 2024-12-18T02:56:31.5823665Z adding: test/test-reports/distributed._tools.test_sac_ilp_1.1_83de39b7eb08edaa_.log (deflated 61%) 2024-12-18T02:56:31.5824627Z adding: test/test-reports/distributed.test_c10d_functional_native_1.1_dcafd296dbbfb2ff_.log (deflated 79%) 2024-12-18T02:56:31.5825665Z adding: test/test-reports/distributed.checkpoint.test_save_load_api_1.1_34ae5c10d327c468_.log (deflated 53%) 2024-12-18T02:56:31.5826723Z adding: test/test-reports/distributed.checkpoint.test_dedup_tensors_1.1_60b0df5cf9ccfd18_.log (deflated 52%) 2024-12-18T02:56:31.5827747Z adding: test/test-reports/distributed._composable.test_contract_1.1_dcc08c236508da95_.log (deflated 66%) 2024-12-18T02:56:31.5828764Z adding: test/test-reports/distributed.pipelining.test_microbatch_1.1_3c7add31fd4d5aaf_.log (deflated 58%) 2024-12-18T02:56:31.5829768Z adding: test/test-reports/distributed.optim.test_named_optimizer_1.1_c2176295cfe0f188_.log (stored 0%) 2024-12-18T02:56:31.5830772Z adding: test/test-reports/distributed.checkpoint.e2e.test_fsdp_ep_1.1_4e2a9011772a3b00_.log (deflated 69%) 2024-12-18T02:56:31.5831958Z adding: test/test-reports/distributed._tensor.experimental.test_register_sharding_1.1_14320eed10b309d6_.log (deflated 60%) 2024-12-18T02:56:31.5833002Z adding: test/test-reports/distributed._tensor.test_attention_1.1_55b3b1d3a8d03733_.log (deflated 88%) 2024-12-18T02:56:31.5833890Z adding: test/test-reports/distributed.test_fake_pg_1.1_e84fb7c933bbe00d_.log (deflated 72%) 2024-12-18T02:56:31.5834845Z adding: test/test-reports/distributed.checkpoint.test_fsdp_optim_state_1.1_e0c2fd9a3de4f808_.log (deflated 95%) 2024-12-18T02:56:31.5835859Z adding: test/test-reports/distributed._composable.test_checkpoint_1.1_6638656fd917ce3f_.log (deflated 69%) 2024-12-18T02:56:31.5836874Z adding: test/test-reports/distributed.checkpoint.test_dtensor_resharding_1.1_2adb3d8109de5f87_.log (deflated 88%) 2024-12-18T02:56:31.5837883Z adding: test/test-reports/distributed.test_distributed_spawn_2.9_9abe803ee66f3e4b_.log (deflated 12%) 2024-12-18T02:56:31.5838808Z adding: test/test-reports/distributed.test_distributed_spawn_2.9_c53285b5fb75255a_.log (deflated 12%) 2024-12-18T02:56:31.5839750Z adding: test/test-reports/distributed.test_distributed_spawn_2.9_30a30dc6053d7711_.log (deflated 93%) 2024-12-18T02:56:31.5843958Z adding: test/test-reports/distributed.test_distributed_spawn_2.9_6da35615c1074dea_.log (deflated 93%) 2024-12-18T02:56:31.5849271Z adding: test/test-reports/distributed.test_distributed_spawn_2.9_45751ecb2d5e33e0_.log (deflated 93%) 2024-12-18T02:56:31.5854435Z adding: test/test-reports/distributed.test_distributed_spawn_2.9_4c89dbc6c1a5f9d9_.log (deflated 93%) 2024-12-18T02:56:31.5855555Z adding: test/test-reports/distributed.test_distributed_spawn_5.9_302bb113143f2f98_.log (deflated 12%) 2024-12-18T02:56:31.5856489Z adding: test/test-reports/distributed.test_distributed_spawn_5.9_87b762fd44fa80f5_.log (deflated 12%) 2024-12-18T02:56:31.5861812Z adding: test/test-reports/distributed.test_distributed_spawn_5.9_d2d3c4a56c261414_.log (deflated 93%) 2024-12-18T02:56:31.5868261Z adding: test/test-reports/distributed.test_distributed_spawn_5.9_d6d5f31063ca4623_.log (deflated 93%) 2024-12-18T02:56:31.5874240Z adding: test/test-reports/distributed.test_distributed_spawn_5.9_0735238399ddf8cc_.log (deflated 94%) 2024-12-18T02:56:31.5879911Z adding: test/test-reports/distributed.test_distributed_spawn_5.9_5682993814e1d560_.log (deflated 94%) 2024-12-18T02:56:31.5881009Z adding: test/test-reports/distributed.test_distributed_spawn_8.9_af74545114a96014_.log (deflated 12%) 2024-12-18T02:56:31.5882024Z adding: test/test-reports/distributed.test_distributed_spawn_8.9_d7699238841b4071_.log (deflated 12%) 2024-12-18T02:56:31.5884415Z adding: test/test-reports/distributed.test_distributed_spawn_8.9_95d273721db6a18d_.log (deflated 94%) 2024-12-18T02:56:31.5888345Z adding: test/test-reports/distributed.test_distributed_spawn_8.9_c4199e6eb08a7a72_.log (deflated 94%) 2024-12-18T02:56:31.5892504Z adding: test/test-reports/distributed.test_distributed_spawn_8.9_fc9d2d2f31142769_.log (deflated 93%) 2024-12-18T02:56:31.5896433Z adding: test/test-reports/distributed.test_distributed_spawn_8.9_b2097b31d6561d68_.log (deflated 93%) 2024-12-18T02:56:31.5898302Z adding: test/test-reports/distributed.checkpoint.test_fsspec_1.1_1967e82aa101dc3f_.log (deflated 84%) 2024-12-18T02:56:31.5899723Z adding: test/test-reports/distributed.tensor.parallel.test_tp_style_1.1_d193ded39d05106a_.log (deflated 77%) 2024-12-18T02:56:31.5903673Z adding: test/test-reports/distributed.checkpoint.e2e.test_e2e_save_and_load_1.1_bda595a177166fc0_.log (deflated 93%) 2024-12-18T02:56:31.5904802Z adding: test/test-reports/distributed.tensor.parallel.test_tp_examples_1.1_161991ee1f835690_.log (deflated 83%) 2024-12-18T02:56:31.5905895Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_state_1.1_3fded1c04c318bf8_.log (deflated 70%) 2024-12-18T02:56:31.5906972Z adding: test/test-reports/distributed.test_control_collectives_1.1_a6881c3ddab1d34c_.log (deflated 77%) 2024-12-18T02:56:31.5907977Z adding: test/test-reports/distributed._tensor.debug.test_comm_mode_1.1_b90474a629a5b2cc_.log (deflated 61%) 2024-12-18T02:56:31.5908987Z adding: test/test-reports/distributed._composable.test_replicate_1.1_326cf4636ea1e768_.log (deflated 80%) 2024-12-18T02:56:31.5910033Z adding: test/test-reports/distributed.optim.test_apply_optimizer_in_backward_1.1_70a212ff41a5eac3_.log (stored 0%) 2024-12-18T02:56:31.5926370Z adding: test/test-reports/distributed.fsdp.test_fsdp_state_dict_2.2_15f9c0c5290fadb7_.log (deflated 97%) 2024-12-18T02:56:31.5936677Z adding: test/test-reports/distributed.test_c10d_nccl_3.3_1a50706c964e8d56_.log (deflated 93%) 2024-12-18T02:56:31.5960892Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_comm_1.1_7cea8c95aad49f2f_.log (deflated 95%) 2024-12-18T02:56:31.5962004Z adding: test/test-reports/distributed.nn.jit.test_instantiator_1.1_e3de813d6dce6007_.log (deflated 63%) 2024-12-18T02:56:31.5962991Z adding: test/test-reports/distributed.fsdp.test_fsdp_backward_prefetch_1.1_1bd477d2a4bcab10_.log (deflated 66%) 2024-12-18T02:56:31.5963980Z adding: test/test-reports/distributed.fsdp.test_fsdp_pure_fp16_1.1_db41536cef1a8abe_.log (deflated 73%) 2024-12-18T02:56:31.5964954Z adding: test/test-reports/distributed.test_c10d_object_collectives_1.1_d8c25a97bcc0c130_.log (deflated 87%) 2024-12-18T02:56:31.5965862Z adding: test/test-reports/distributed.test_c10d_pypg_1.1_4f4bb7400b0f840c_.log (deflated 86%) 2024-12-18T02:56:31.5967777Z adding: test/test-reports/distributed.fsdp.test_fsdp_freezing_weights_1.1_7ce3ffff2ed59e82_.log (deflated 94%) 2024-12-18T02:56:31.5968856Z adding: test/test-reports/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks_1.1_5c9f7d5f923de9e9_.log (deflated 73%) 2024-12-18T02:56:31.5970038Z adding: test/test-reports/distributed.fsdp.test_fsdp_exec_order_1.1_97e5115fb10e78d2_.log (deflated 93%) 2024-12-18T02:56:31.5971412Z adding: test/test-reports/distributed.fsdp.test_fsdp_grad_acc_1.1_a7183b063ff1fe6c_.log (deflated 90%) 2024-12-18T02:56:31.5972368Z adding: test/test-reports/distributed.fsdp.test_fsdp_ignored_modules_1.1_ee345141a32e10d3_.log (deflated 76%) 2024-12-18T02:56:31.5973386Z adding: test/test-reports/distributed.fsdp.test_fsdp_multiple_wrapping_1.1_a8564f6030a24b15_.log (deflated 72%) 2024-12-18T02:56:31.5974489Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_25f44a3f728dc1fe_.log (deflated 76%) 2024-12-18T02:56:31.5977083Z adding: test/test-reports/distributed._composable.test_composability.test_2d_composability_1.1_77c0c2692f2612dc_.log (deflated 92%) 2024-12-18T02:56:31.5979570Z adding: test/test-reports/distributed.fsdp.test_fsdp_comm_hooks_1.1_aa579ddb4e97d7b8_.log (deflated 95%) 2024-12-18T02:56:31.5989963Z adding: test/test-reports/distributed.fsdp.test_fsdp_core_2.2_a81ab11c8b7f6336_.log (deflated 97%) 2024-12-18T02:56:31.5991615Z adding: test/test-reports/distributed._shard.test_sharder_1.1_525d4c73ec376388_.log (deflated 88%) 2024-12-18T02:56:31.5994313Z adding: test/test-reports/distributed.fsdp.test_fsdp_comm_1.1_4cee44d5f80cf3f7_.log (deflated 93%) 2024-12-18T02:56:31.5997569Z adding: test/test-reports/distributed.fsdp.test_distributed_checkpoint_1.1_a59c6f4c0a14e29b_.log (deflated 91%) 2024-12-18T02:56:31.6002622Z adding: test/test-reports/distributed.fsdp.test_fsdp_use_orig_params_1.1_d7c4eb0d244e68e4_.log (deflated 92%) 2024-12-18T02:56:31.6021259Z adding: test/test-reports/distributed.test_c10d_gloo_1.1_37d1f0fa984c921f_.log (deflated 96%) 2024-12-18T02:56:31.6022392Z adding: test/test-reports/distributed.elastic.timer.api_test_1.1_c4073b0fa122f11b_.log (stored 0%) 2024-12-18T02:56:31.6062206Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-12-18T02:56:31.6062824Z # Remove any previous debugging artifacts if they exist 2024-12-18T02:56:31.6063308Z rm -f debug-*.zip 2024-12-18T02:56:31.6063630Z if [ -d 'test/debug' ]; then 2024-12-18T02:56:31.6064048Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-12-18T02:56:31.6064441Z fi 2024-12-18T02:56:31.6070037Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:56:31.6070472Z env: 2024-12-18T02:56:31.6070731Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:56:31.6071208Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:56:31.6071803Z DOCKER_CONTAINER_ID: 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T02:56:31.6072526Z FILE_SUFFIX: test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_34566462868 2024-12-18T02:56:31.6073039Z ##[endgroup] 2024-12-18T02:56:31.6157663Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-12-18T02:56:31.6158033Z with: 2024-12-18T02:56:31.6158259Z s3-bucket: gha-artifacts 2024-12-18T02:56:31.6158729Z s3-prefix: pytorch/pytorch/12383255652/1/artifact 2024-12-18T02:56:31.6159103Z retention-days: 14 2024-12-18T02:56:31.6159370Z if-no-files-found: warn 2024-12-18T02:56:31.6159651Z path: test-jsons-*.zip 2024-12-18T02:56:31.6159906Z name: artifact 2024-12-18T02:56:31.6160150Z region: us-east-1 2024-12-18T02:56:31.6160389Z env: 2024-12-18T02:56:31.6160610Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:56:31.6160959Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:56:31.6161513Z DOCKER_CONTAINER_ID: 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T02:56:31.6162019Z ##[endgroup] 2024-12-18T02:56:31.9899395Z NOTE: s3-prefix specified, ignoring name parameter 2024-12-18T02:56:31.9900123Z With the provided path, there will be 1 file uploaded 2024-12-18T02:56:31.9900696Z Uploading to s3 prefix: pytorch/pytorch/12383255652/1/artifact 2024-12-18T02:56:31.9944034Z Starting upload of test-jsons-test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_34566462868.zip 2024-12-18T02:56:32.1257149Z Finished upload of test-jsons-test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_34566462868.zip 2024-12-18T02:56:32.1446969Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-12-18T02:56:32.1447320Z with: 2024-12-18T02:56:32.1447569Z s3-bucket: gha-artifacts 2024-12-18T02:56:32.1447919Z s3-prefix: pytorch/pytorch/12383255652/1/artifact 2024-12-18T02:56:32.1448296Z retention-days: 14 2024-12-18T02:56:32.1448574Z if-no-files-found: error 2024-12-18T02:56:32.1448863Z path: test-reports-*.zip 2024-12-18T02:56:32.1449148Z name: artifact 2024-12-18T02:56:32.1449404Z region: us-east-1 2024-12-18T02:56:32.1449657Z env: 2024-12-18T02:56:32.1449898Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:56:32.1450242Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:56:32.1450973Z DOCKER_CONTAINER_ID: 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T02:56:32.1451491Z ##[endgroup] 2024-12-18T02:56:32.4896671Z NOTE: s3-prefix specified, ignoring name parameter 2024-12-18T02:56:32.4897250Z With the provided path, there will be 1 file uploaded 2024-12-18T02:56:32.4897772Z Uploading to s3 prefix: pytorch/pytorch/12383255652/1/artifact 2024-12-18T02:56:32.4940356Z Starting upload of test-reports-test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_34566462868.zip 2024-12-18T02:56:33.0940680Z Finished upload of test-reports-test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_34566462868.zip 2024-12-18T02:56:33.1150828Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-12-18T02:56:33.1151256Z with: 2024-12-18T02:56:33.1151634Z s3-bucket: gha-artifacts 2024-12-18T02:56:33.1152176Z s3-prefix: pytorch/pytorch/12383255652/1/artifact 2024-12-18T02:56:33.1152588Z retention-days: 14 2024-12-18T02:56:33.1152898Z if-no-files-found: ignore 2024-12-18T02:56:33.1153239Z path: logs-*.zip 2024-12-18T02:56:33.1153500Z name: artifact 2024-12-18T02:56:33.1153771Z region: us-east-1 2024-12-18T02:56:33.1154044Z env: 2024-12-18T02:56:33.1154298Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:56:33.1154701Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:56:33.1155299Z DOCKER_CONTAINER_ID: 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T02:56:33.1155847Z ##[endgroup] 2024-12-18T02:56:33.4659721Z NOTE: s3-prefix specified, ignoring name parameter 2024-12-18T02:56:33.4660324Z With the provided path, there will be 1 file uploaded 2024-12-18T02:56:33.4660870Z Uploading to s3 prefix: pytorch/pytorch/12383255652/1/artifact 2024-12-18T02:56:33.4702671Z Starting upload of logs-test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_34566462868.zip 2024-12-18T02:56:33.6686823Z Finished upload of logs-test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_34566462868.zip 2024-12-18T02:56:33.6896225Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-12-18T02:56:33.6896629Z with: 2024-12-18T02:56:33.6896870Z s3-bucket: gha-artifacts 2024-12-18T02:56:33.6897237Z s3-prefix: pytorch/pytorch/12383255652/1/artifact 2024-12-18T02:56:33.6897629Z retention-days: 14 2024-12-18T02:56:33.6897939Z if-no-files-found: ignore 2024-12-18T02:56:33.6898249Z path: debug-*.zip 2024-12-18T02:56:33.6898496Z name: artifact 2024-12-18T02:56:33.6898883Z region: us-east-1 2024-12-18T02:56:33.6899330Z env: 2024-12-18T02:56:33.6899598Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:56:33.6900012Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:56:33.6900650Z DOCKER_CONTAINER_ID: 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T02:56:33.6901232Z ##[endgroup] 2024-12-18T02:56:34.0326517Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-12-18T02:56:34.0515449Z ##[group]Run # shellcheck disable=SC2156 2024-12-18T02:56:34.0515862Z # shellcheck disable=SC2156 2024-12-18T02:56:34.0516483Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-12-18T02:56:34.0523079Z shell: /usr/bin/bash -e {0} 2024-12-18T02:56:34.0523546Z env: 2024-12-18T02:56:34.0523796Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:56:34.0524185Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:56:34.0524805Z DOCKER_CONTAINER_ID: 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T02:56:34.0525362Z ##[endgroup] 2024-12-18T02:56:34.2932649Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@release/2.6 2024-12-18T02:56:34.2933182Z with: 2024-12-18T02:56:34.2933433Z env: 2024-12-18T02:56:34.2933673Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:56:34.2934066Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:56:34.2934697Z DOCKER_CONTAINER_ID: 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T02:56:34.2935255Z ##[endgroup] 2024-12-18T02:56:34.2970409Z ##[group]Run set -eou pipefail 2024-12-18T02:56:34.2970772Z set -eou pipefail 2024-12-18T02:56:34.2971077Z  2024-12-18T02:56:34.2971484Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-12-18T02:56:34.2972024Z for _ in $(seq 1440); do 2024-12-18T02:56:34.2972401Z  # Break if no ssh session exists anymore 2024-12-18T02:56:34.2972795Z  if [ "$(who)" = "" ]; then 2024-12-18T02:56:34.2973130Z  break 2024-12-18T02:56:34.2973431Z  fi 2024-12-18T02:56:34.2973691Z  echo "." 2024-12-18T02:56:34.2973952Z  sleep 5 2024-12-18T02:56:34.2974220Z done 2024-12-18T02:56:34.2980239Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:56:34.2980692Z env: 2024-12-18T02:56:34.2980951Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:56:34.2981338Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:56:34.2981980Z DOCKER_CONTAINER_ID: 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T02:56:34.2982550Z ##[endgroup] 2024-12-18T02:56:34.3007020Z Holding runner for 2 hours until all ssh sessions have logged out 2024-12-18T02:56:34.3124927Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T02:56:34.3125580Z # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T02:56:34.3126079Z # shellcheck disable=SC2046 2024-12-18T02:56:34.3126470Z docker stop $(docker ps -q) || true 2024-12-18T02:56:34.3126876Z # Prune all of the docker images 2024-12-18T02:56:34.3127258Z docker system prune -af 2024-12-18T02:56:34.3132823Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:56:34.3133261Z env: 2024-12-18T02:56:34.3133517Z GIT_DEFAULT_BRANCH: main 2024-12-18T02:56:34.3133888Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T02:56:34.3134604Z DOCKER_CONTAINER_ID: 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T02:56:34.3135119Z ##[endgroup] 2024-12-18T02:56:35.0045993Z 680575e591d7 2024-12-18T02:56:35.4305745Z Deleted Containers: 2024-12-18T02:56:35.4306244Z 680575e591d746470ac64357996618bdf00b3d6022e254512b031771af78c937 2024-12-18T02:56:35.4306618Z 2024-12-18T02:56:40.7624374Z Deleted Images: 2024-12-18T02:56:40.7625496Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T02:56:40.7627128Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9@sha256:52918bcdb0e370d1b9413099049464295e6c36e0a09cff6cff645b519e6da502 2024-12-18T02:56:40.7628401Z deleted: sha256:166dd14a00627fbd0d4e062a733b69b733b63b1ac12ec3bd5285db8a8befaaf5 2024-12-18T02:56:40.7629168Z deleted: sha256:2e828a4f851a71a08b9f06842643025da951b3d1a26bc40be945b5dc00f5015f 2024-12-18T02:56:40.7629907Z deleted: sha256:d27246a7406fc9214ce9db75df344bdd8e0a31300352beaf8ee0e0668a1ace5b 2024-12-18T02:56:40.7630671Z deleted: sha256:68c9fa6063c3f2de81e86b8de02d9c520f91cebed9343aecd84d4621681e108b 2024-12-18T02:56:40.7631552Z deleted: sha256:7fba561e060d45fd930ced565d615e1e1f5b5e1266212caaddcbdce3b6ce2157 2024-12-18T02:56:40.7632543Z deleted: sha256:5b5728232cece21786a91a2b7109215ddacbea370a452a5725759be3f9782f36 2024-12-18T02:56:40.7633268Z deleted: sha256:dc257cb01440638b647e8f88d00e791443a113161d0d4f469015690bab702c18 2024-12-18T02:56:40.7634135Z deleted: sha256:3ee1ba287c278eaf901c09c7e521f0a405bbb9fb8062aa51674fb9ea3ab18789 2024-12-18T02:56:40.7634870Z deleted: sha256:2d66661697c95587c70e4f77dc01731d10271a2a2d5ae52b240971a0554e6d0e 2024-12-18T02:56:40.7635594Z deleted: sha256:eaa304b04f52287b7bdefe4e276ed9444a9f2924aab757a4112d857162d00aae 2024-12-18T02:56:40.7636325Z deleted: sha256:4b999f756c8d5d7c8444b89e5466899b4bbd4f3adec20d788c443754a4cb2cf6 2024-12-18T02:56:40.7637061Z deleted: sha256:031bf045cac35b9a41c06ca4b85ca3419cf104dcaef931590111a4cbc99225d5 2024-12-18T02:56:40.7637805Z deleted: sha256:4a6da4e6593178794969f47a35ec4c667bd2a620051de682f4f6476200528ad7 2024-12-18T02:56:40.7638529Z deleted: sha256:d49018d0fd9477af622a59aae4a080650d82601c3ff10f5e413d54b61605cbc7 2024-12-18T02:56:40.7639367Z deleted: sha256:eaeb72d8e211a811359bbae6cdb944e506296e3593e02a7845d1de2b020db64d 2024-12-18T02:56:40.7640077Z deleted: sha256:4443a93278f4af70ebbc86c00611e1229086bcd73ba6f46cfbac959f5a3ea13e 2024-12-18T02:56:40.7640797Z deleted: sha256:7da4d3af5a970120b30198ce7c0d5758fd4180887250d2e1d20b1bb19a7ac558 2024-12-18T02:56:40.7641699Z deleted: sha256:51d1e0cf0c3b813663526e6d529bfb7f0374c2474f5a9581ccfd66f6dd722384 2024-12-18T02:56:40.7642441Z deleted: sha256:6d152e7f1f9932d4367ae21fbdcaf7e300d258e862c3c3104defca22e1f67f82 2024-12-18T02:56:40.7643179Z deleted: sha256:5f3d462f5b23f997262b6a535998f846bc071e1cbfb2039b507881fb8d873078 2024-12-18T02:56:40.7643918Z deleted: sha256:b5aa4897eb3166e0b560548c1ccbf716422a3b134bc63aa4da378db45a578506 2024-12-18T02:56:40.7644637Z deleted: sha256:19e8d1b6e7033c3cf97c2220cc96c9d4860bc5a09ad25b706a7260176331e59b 2024-12-18T02:56:40.7645372Z deleted: sha256:f82be94fab4a78f872875e44a2a54d01d6b7c7c9a73ad193593549037b0c9c6c 2024-12-18T02:56:40.7646108Z deleted: sha256:45e1395581d647e78b6fae987a785c08acea4b6176c97188b5a5010e9004d947 2024-12-18T02:56:40.7646836Z deleted: sha256:6a116d37e354cd470a6dee7600017fd2c240860da82a19939bd1451416af0972 2024-12-18T02:56:40.7647568Z deleted: sha256:96d8d1f880e7957a35c4060d1ae56eaa05c1e73ff5e7128cc5b42531a996ffeb 2024-12-18T02:56:40.7648303Z deleted: sha256:a186beea437cb709b244be99550467206b5bdcb190e032c091f15a6a065ddc36 2024-12-18T02:56:40.7649036Z deleted: sha256:e5f8bcff21a8fbb2d507150a82a1b26ebcec9caf9ee8fa7dce844574ef8da2ad 2024-12-18T02:56:40.7649777Z deleted: sha256:13091955acb9200fc139971ba2434c90657de11ff38ac954e35f3461fa4a6b8a 2024-12-18T02:56:40.7650504Z deleted: sha256:be10df8394bb253584f3f2f4c84a28bd53744e8c6892f1efcb4b12b67abe23da 2024-12-18T02:56:40.7651257Z deleted: sha256:ca58fcd63e3576eddbb8aaef07131e18631c53e4e9b6e8d9dbec5b8ddfa7726a 2024-12-18T02:56:40.7652009Z deleted: sha256:57c7d561a44e019b14a0f4aae3354ec1734b088b539cab6fb0e8ecca850ee714 2024-12-18T02:56:40.7652752Z deleted: sha256:a960411fa30719c90510a13403dcde638fbee3cbbb4ab71f14e52f40e9dd4533 2024-12-18T02:56:40.7653477Z deleted: sha256:8a2ef497a8ab2819802ad459d2dafe5195a85d11b38ef6d6d4d4fe872d9c6b6b 2024-12-18T02:56:40.7654242Z deleted: sha256:cbf589ae40cc4496ffa854c9108d58905a2ea5ac3162669b3b4606287b751d28 2024-12-18T02:56:40.7654973Z deleted: sha256:88c378f8d5104ea7a03bf49a1a7db4df53518519236345cf29bc2df76377d0ac 2024-12-18T02:56:40.7655709Z deleted: sha256:db2a844a68747daa51bb2b28c083df270c19bfbcb7e038723a1db1bacf20462e 2024-12-18T02:56:40.7656446Z deleted: sha256:f548fd4a27a545794a88afc9f9f3460001efb98677ae40f9737387bffedd901d 2024-12-18T02:56:40.7657184Z deleted: sha256:42f4cf4d61dad97d4c5f3bd5cb9374fe260e5166e36f4ae2d00a9c59127abb3a 2024-12-18T02:56:40.7657924Z deleted: sha256:1e465635bd263f73940e977a6d9ab8b3beceabc5eaa9d08c65a28aafdf72abf0 2024-12-18T02:56:40.7658642Z deleted: sha256:b890305c75102c7bd0a645b98b0e1893d45433319a1394c457afb7ffc3ac1070 2024-12-18T02:56:40.7659897Z deleted: sha256:c9d4902893199761be10cecc8c1de9ce1242b5df3c161bc452e5ffcf2d3bff38 2024-12-18T02:56:40.7660801Z deleted: sha256:254c14be940e16bbbe4d6da795ac3a987a11fde9842d614f31c1caae910c7263 2024-12-18T02:56:40.7661557Z deleted: sha256:137abe9e7f1c1726800d1ae709a40b92e48267f52dd7fa7b966be15345a94850 2024-12-18T02:56:40.7662414Z deleted: sha256:771d7f9e8bf6f532b16c3daa3a9167d4030f919eec29b18d99387ad1d1a277cb 2024-12-18T02:56:40.7663186Z deleted: sha256:22ee16d76a83cee720de0aafa52532ff1f3335a9048ff43877cee2aa6a370f5f 2024-12-18T02:56:40.7663934Z deleted: sha256:dbbad5d1e6d1785ee41424b56457a16a1d882ff1c09f41486b443b8c1e7c7eb4 2024-12-18T02:56:40.7664687Z deleted: sha256:df9a3657ce5f83f7ab775078ff74a472438fc5c21e1e0de53cc66509788e12bc 2024-12-18T02:56:40.7665452Z deleted: sha256:245eebe495c94ddc69ab83759089101d2edfdbf90474a3cb772e7bf1cce893bd 2024-12-18T02:56:40.7666211Z deleted: sha256:235e5b71d4fbb1eaf808e6fd789604752f653c1176b8acb1ba97604f35418396 2024-12-18T02:56:40.7666984Z deleted: sha256:c96d511cee1be7af8a8fedcab42cac5233cf66be05d188a2ba32ba4db1d2f715 2024-12-18T02:56:40.7667749Z deleted: sha256:704760c42e98a22e0f5ce40f10bdc3baec3e4ca679c2b046e90e6e12228a7164 2024-12-18T02:56:40.7668514Z deleted: sha256:78ec6b9f299f8114edc65d21c0c2e730f8423fcc496b819fe25df22af7477bd6 2024-12-18T02:56:40.7669280Z deleted: sha256:3d2201aa2c2a2db1e240c872ac8b84d4c92eed331c5c0053eaea4a8d292dd29a 2024-12-18T02:56:40.7670038Z deleted: sha256:2f8151f506d01c5be81b7547d7c20a36a2f30f6713adecaa91adb633fab9d809 2024-12-18T02:56:40.7670788Z deleted: sha256:d1f62ced0636553b623466ced7568917bb2b453997168ceb2635926f5c069ae6 2024-12-18T02:56:40.7671745Z deleted: sha256:bcb018d693dd3d8346bc43a253a42de37aad6ae11f42ee748dd9fdf1d3b259fc 2024-12-18T02:56:40.7672566Z deleted: sha256:40f224dcd4345d71537636d5c3bdb9c0659430771d9ac90c9e7cf7da30fabdd3 2024-12-18T02:56:40.7673235Z deleted: sha256:113e5e34c2859806f96efe4d47aa5e0d438e3f72211b7d539de8750bbfed7a98 2024-12-18T02:56:40.7673903Z deleted: sha256:a4786d6f71e2eec542ec8c4ff41191481de085b7e1b4e0139995017b6b7d6477 2024-12-18T02:56:40.7674581Z deleted: sha256:84faffca53886fb81b780b885baa8b6b4a4c16795bf93c9eed6f165c8d333545 2024-12-18T02:56:40.7675259Z deleted: sha256:22f20f90960f1663da6dbde7ea9b18026e0e1b03006890d56f64fc084a7550e3 2024-12-18T02:56:40.7675921Z deleted: sha256:d53c4a66cd75e088673985522b5ba099599960ac06350e59cfd330572327bff2 2024-12-18T02:56:40.7676578Z deleted: sha256:e95f4ec4da5fef7e54c22d270b51d9077a8c577a82bf0304e200fce309551084 2024-12-18T02:56:40.7677254Z deleted: sha256:ddbe476a0432522a73e2b052342ebc6f8610f7437c076ff6bf7e4501dc79efbc 2024-12-18T02:56:40.7677933Z deleted: sha256:ad7883a4c2954aebf555ea86e018fee1cc12d159b9e4d149263eb92c7cd21708 2024-12-18T02:56:40.7678616Z deleted: sha256:656ce67e45b8dab0cf77d1d21cd380f5ddfa64bfd486ebaa47b3af7c9b9df7b5 2024-12-18T02:56:40.7679301Z deleted: sha256:ae5754644efe96c6b01bbe838a7db9aac49333bf618c0a05160cdd0dd4b38fe8 2024-12-18T02:56:40.7679980Z deleted: sha256:23058ae084a55e5fd11db1aabe073f1886a1047be70eeb78dada3e6d4949c4d7 2024-12-18T02:56:40.7680643Z deleted: sha256:55c98b096fa43eb44df581de08d34a280f6eb359a04b1562fedccca5d56748e3 2024-12-18T02:56:40.7681325Z deleted: sha256:4a1d549deb2a6a9e8c52225d513aefeffbe0097e1eeb8217ca34963724db0b53 2024-12-18T02:56:40.7681997Z deleted: sha256:89e3b1c20a2d617b0ed103dc78296d410c1611d2044d8426fcc757b64c3113fe 2024-12-18T02:56:40.7682730Z deleted: sha256:89edf8c906678c217b9a88160e1561991f53909c3fd27a4a5e1f0ca0f7ff7888 2024-12-18T02:56:40.7683407Z deleted: sha256:d5ba67f8f9b0c2b6da06d2d5d9d5fedc9f17cbe1a5e13a1a4033a4619f4528ba 2024-12-18T02:56:40.7684086Z deleted: sha256:5d61311e1a293c0c9642e34871719525dfeec7028e4b45e08344a469e3fe24bf 2024-12-18T02:56:40.7684738Z deleted: sha256:b89bb70f745a94c0edd5e974aee294e160735f76fd666a1a06a0b9b1be3caf63 2024-12-18T02:56:40.7685416Z deleted: sha256:9e5477f1f21a38c3dfe21a56a7d77d1b1a42453aed3e11729afcbb9d5731770b 2024-12-18T02:56:40.7686100Z deleted: sha256:b76f4af91cffe2bb7b3f09960f5bdbb26f741419defda65c451c4407fc5b0a92 2024-12-18T02:56:40.7686776Z deleted: sha256:3f8748a00ebd394c0e9d9d481972d764dff58896841ef0150b3f7efaac0f4a4a 2024-12-18T02:56:40.7687511Z deleted: sha256:13989c771e3ba4d40f618344e55e5cb7062a35041b5efd69e2f740ba819f46ee 2024-12-18T02:56:40.7688182Z deleted: sha256:9ebb27837f6c67ad8c17e9af4e63c008faae03fa226c3641dd8bbbfeefb106c4 2024-12-18T02:56:40.7688902Z deleted: sha256:8c16907d735950f8093582bc3cf392b70e002456bf9c4224de5a5b86b78b3ea2 2024-12-18T02:56:40.7689566Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-12-18T02:56:40.7689972Z 2024-12-18T02:56:40.7690090Z Total reclaimed space: 36.91GB 2024-12-18T02:56:40.7774909Z Post job cleanup. 2024-12-18T02:56:40.7839845Z Post job cleanup. 2024-12-18T02:56:40.8613780Z [command]/usr/bin/git version 2024-12-18T02:56:40.8648891Z git version 2.40.1 2024-12-18T02:56:40.8693913Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/c8033525-2158-4138-a085-e7c717a5abbb' before making global git config changes 2024-12-18T02:56:40.8695025Z Adding repository directory to the temporary git global config as a safe directory 2024-12-18T02:56:40.8696770Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-18T02:56:40.8725672Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-12-18T02:56:40.8751085Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-12-18T02:56:40.9067871Z Entering 'android/libs/fbjni' 2024-12-18T02:56:40.9124538Z Entering 'third_party/FP16' 2024-12-18T02:56:40.9183321Z Entering 'third_party/FXdiv' 2024-12-18T02:56:40.9240703Z Entering 'third_party/NNPACK' 2024-12-18T02:56:40.9297166Z Entering 'third_party/NVTX' 2024-12-18T02:56:40.9356753Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T02:56:40.9413951Z Entering 'third_party/XNNPACK' 2024-12-18T02:56:40.9488070Z Entering 'third_party/benchmark' 2024-12-18T02:56:40.9544529Z Entering 'third_party/composable_kernel' 2024-12-18T02:56:40.9608372Z Entering 'third_party/cpp-httplib' 2024-12-18T02:56:40.9664264Z Entering 'third_party/cpuinfo' 2024-12-18T02:56:40.9720835Z Entering 'third_party/cudnn_frontend' 2024-12-18T02:56:40.9778068Z Entering 'third_party/cutlass' 2024-12-18T02:56:40.9843840Z Entering 'third_party/eigen' 2024-12-18T02:56:40.9900806Z Entering 'third_party/fbgemm' 2024-12-18T02:56:40.9958914Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T02:56:41.0015761Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T02:56:41.0076156Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T02:56:41.0140871Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T02:56:41.0198625Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T02:56:41.0256247Z Entering 'third_party/flatbuffers' 2024-12-18T02:56:41.0318146Z Entering 'third_party/fmt' 2024-12-18T02:56:41.0376092Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T02:56:41.0434304Z Entering 'third_party/gloo' 2024-12-18T02:56:41.0493165Z Entering 'third_party/googletest' 2024-12-18T02:56:41.0549801Z Entering 'third_party/ideep' 2024-12-18T02:56:41.0606394Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T02:56:41.0670486Z Entering 'third_party/ittapi' 2024-12-18T02:56:41.0729680Z Entering 'third_party/kineto' 2024-12-18T02:56:41.0787927Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T02:56:41.0844226Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T02:56:41.0901331Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T02:56:41.0959705Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T02:56:41.1016403Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T02:56:41.1073997Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T02:56:41.1134152Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T02:56:41.1191595Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T02:56:41.1250605Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T02:56:41.1307538Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T02:56:41.1367903Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T02:56:41.1422505Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T02:56:41.1482066Z Entering 'third_party/mimalloc' 2024-12-18T02:56:41.1541174Z Entering 'third_party/nccl/nccl' 2024-12-18T02:56:41.1599462Z Entering 'third_party/nlohmann' 2024-12-18T02:56:41.1660189Z Entering 'third_party/onnx' 2024-12-18T02:56:41.1732916Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T02:56:41.1794498Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T02:56:41.1852610Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T02:56:41.1909271Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T02:56:41.1965563Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T02:56:41.2020686Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T02:56:41.2083819Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T02:56:41.2140560Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T02:56:41.2197137Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T02:56:41.2252484Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T02:56:41.2308473Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T02:56:41.2369810Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T02:56:41.2448561Z Entering 'third_party/pocketfft' 2024-12-18T02:56:41.2503044Z Entering 'third_party/protobuf' 2024-12-18T02:56:41.2562162Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T02:56:41.2621010Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T02:56:41.2679736Z Entering 'third_party/psimd' 2024-12-18T02:56:41.2737818Z Entering 'third_party/pthreadpool' 2024-12-18T02:56:41.2793893Z Entering 'third_party/pybind11' 2024-12-18T02:56:41.2850038Z Entering 'third_party/python-peachpy' 2024-12-18T02:56:41.2904214Z Entering 'third_party/sleef' 2024-12-18T02:56:41.2962409Z Entering 'third_party/tensorpipe' 2024-12-18T02:56:41.3018463Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T02:56:41.3075061Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T02:56:41.3133405Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T02:56:41.3189439Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T02:56:41.3244171Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T02:56:41.3322432Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-12-18T02:56:41.3342161Z http.https://github.com/.extraheader 2024-12-18T02:56:41.3350836Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-12-18T02:56:41.3377318Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-12-18T02:56:41.3683166Z Entering 'android/libs/fbjni' 2024-12-18T02:56:41.3721490Z http.https://github.com/.extraheader 2024-12-18T02:56:41.3759614Z Entering 'third_party/FP16' 2024-12-18T02:56:41.3797237Z http.https://github.com/.extraheader 2024-12-18T02:56:41.3831997Z Entering 'third_party/FXdiv' 2024-12-18T02:56:41.3869279Z http.https://github.com/.extraheader 2024-12-18T02:56:41.3904802Z Entering 'third_party/NNPACK' 2024-12-18T02:56:41.3944297Z http.https://github.com/.extraheader 2024-12-18T02:56:41.3980392Z Entering 'third_party/NVTX' 2024-12-18T02:56:41.4019612Z http.https://github.com/.extraheader 2024-12-18T02:56:41.4056233Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T02:56:41.4093841Z http.https://github.com/.extraheader 2024-12-18T02:56:41.4130123Z Entering 'third_party/XNNPACK' 2024-12-18T02:56:41.4169485Z http.https://github.com/.extraheader 2024-12-18T02:56:41.4220263Z Entering 'third_party/benchmark' 2024-12-18T02:56:41.4258043Z http.https://github.com/.extraheader 2024-12-18T02:56:41.4294582Z Entering 'third_party/composable_kernel' 2024-12-18T02:56:41.4331520Z http.https://github.com/.extraheader 2024-12-18T02:56:41.4374074Z Entering 'third_party/cpp-httplib' 2024-12-18T02:56:41.4412295Z http.https://github.com/.extraheader 2024-12-18T02:56:41.4447729Z Entering 'third_party/cpuinfo' 2024-12-18T02:56:41.4487070Z http.https://github.com/.extraheader 2024-12-18T02:56:41.4522207Z Entering 'third_party/cudnn_frontend' 2024-12-18T02:56:41.4561801Z http.https://github.com/.extraheader 2024-12-18T02:56:41.4602286Z Entering 'third_party/cutlass' 2024-12-18T02:56:41.4642378Z http.https://github.com/.extraheader 2024-12-18T02:56:41.4686757Z Entering 'third_party/eigen' 2024-12-18T02:56:41.4728787Z http.https://github.com/.extraheader 2024-12-18T02:56:41.4767103Z Entering 'third_party/fbgemm' 2024-12-18T02:56:41.4806031Z http.https://github.com/.extraheader 2024-12-18T02:56:41.4841616Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T02:56:41.4878672Z http.https://github.com/.extraheader 2024-12-18T02:56:41.4916465Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T02:56:41.4954815Z http.https://github.com/.extraheader 2024-12-18T02:56:41.4994791Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T02:56:41.5030158Z http.https://github.com/.extraheader 2024-12-18T02:56:41.5073358Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T02:56:41.5108714Z http.https://github.com/.extraheader 2024-12-18T02:56:41.5142818Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T02:56:41.5179994Z http.https://github.com/.extraheader 2024-12-18T02:56:41.5221759Z Entering 'third_party/flatbuffers' 2024-12-18T02:56:41.5260393Z http.https://github.com/.extraheader 2024-12-18T02:56:41.5302614Z Entering 'third_party/fmt' 2024-12-18T02:56:41.5340893Z http.https://github.com/.extraheader 2024-12-18T02:56:41.5376806Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T02:56:41.5415066Z http.https://github.com/.extraheader 2024-12-18T02:56:41.5454815Z Entering 'third_party/gloo' 2024-12-18T02:56:41.5492894Z http.https://github.com/.extraheader 2024-12-18T02:56:41.5527859Z Entering 'third_party/googletest' 2024-12-18T02:56:41.5568349Z http.https://github.com/.extraheader 2024-12-18T02:56:41.5612352Z Entering 'third_party/ideep' 2024-12-18T02:56:41.5650728Z http.https://github.com/.extraheader 2024-12-18T02:56:41.5684571Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T02:56:41.5722198Z http.https://github.com/.extraheader 2024-12-18T02:56:41.5767200Z Entering 'third_party/ittapi' 2024-12-18T02:56:41.5806257Z http.https://github.com/.extraheader 2024-12-18T02:56:41.5839732Z Entering 'third_party/kineto' 2024-12-18T02:56:41.5878842Z http.https://github.com/.extraheader 2024-12-18T02:56:41.5922008Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T02:56:41.5960581Z http.https://github.com/.extraheader 2024-12-18T02:56:41.6001337Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T02:56:41.6039083Z http.https://github.com/.extraheader 2024-12-18T02:56:41.6084673Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T02:56:41.6125305Z http.https://github.com/.extraheader 2024-12-18T02:56:41.6161071Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T02:56:41.6198696Z http.https://github.com/.extraheader 2024-12-18T02:56:41.6236417Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T02:56:41.6274356Z http.https://github.com/.extraheader 2024-12-18T02:56:41.6310816Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T02:56:41.6349853Z http.https://github.com/.extraheader 2024-12-18T02:56:41.6395553Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T02:56:41.6433649Z http.https://github.com/.extraheader 2024-12-18T02:56:41.6475963Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T02:56:41.6513308Z http.https://github.com/.extraheader 2024-12-18T02:56:41.6548427Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T02:56:41.6584733Z http.https://github.com/.extraheader 2024-12-18T02:56:41.6622419Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T02:56:41.6660182Z http.https://github.com/.extraheader 2024-12-18T02:56:41.6702099Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T02:56:41.6740531Z http.https://github.com/.extraheader 2024-12-18T02:56:41.6779136Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T02:56:41.6814861Z http.https://github.com/.extraheader 2024-12-18T02:56:41.6853485Z Entering 'third_party/mimalloc' 2024-12-18T02:56:41.6891953Z http.https://github.com/.extraheader 2024-12-18T02:56:41.6925566Z Entering 'third_party/nccl/nccl' 2024-12-18T02:56:41.6965734Z http.https://github.com/.extraheader 2024-12-18T02:56:41.7001318Z Entering 'third_party/nlohmann' 2024-12-18T02:56:41.7040313Z http.https://github.com/.extraheader 2024-12-18T02:56:41.7078727Z Entering 'third_party/onnx' 2024-12-18T02:56:41.7116264Z http.https://github.com/.extraheader 2024-12-18T02:56:41.7168828Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T02:56:41.7206517Z http.https://github.com/.extraheader 2024-12-18T02:56:41.7244131Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T02:56:41.7282629Z http.https://github.com/.extraheader 2024-12-18T02:56:41.7320111Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T02:56:41.7356980Z http.https://github.com/.extraheader 2024-12-18T02:56:41.7396163Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T02:56:41.7431501Z http.https://github.com/.extraheader 2024-12-18T02:56:41.7473798Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T02:56:41.7508170Z http.https://github.com/.extraheader 2024-12-18T02:56:41.7541736Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T02:56:41.7578472Z http.https://github.com/.extraheader 2024-12-18T02:56:41.7616426Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T02:56:41.7653628Z http.https://github.com/.extraheader 2024-12-18T02:56:41.7692929Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T02:56:41.7729136Z http.https://github.com/.extraheader 2024-12-18T02:56:41.7764069Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T02:56:41.7801452Z http.https://github.com/.extraheader 2024-12-18T02:56:41.7837740Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T02:56:41.7874000Z http.https://github.com/.extraheader 2024-12-18T02:56:41.7918644Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T02:56:41.7957150Z http.https://github.com/.extraheader 2024-12-18T02:56:41.7996049Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T02:56:41.8032651Z http.https://github.com/.extraheader 2024-12-18T02:56:41.8092652Z Entering 'third_party/pocketfft' 2024-12-18T02:56:41.8130303Z http.https://github.com/.extraheader 2024-12-18T02:56:41.8166388Z Entering 'third_party/protobuf' 2024-12-18T02:56:41.8204135Z http.https://github.com/.extraheader 2024-12-18T02:56:41.8243696Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T02:56:41.8281491Z http.https://github.com/.extraheader 2024-12-18T02:56:41.8319178Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T02:56:41.8357328Z http.https://github.com/.extraheader 2024-12-18T02:56:41.8396289Z Entering 'third_party/psimd' 2024-12-18T02:56:41.8437239Z http.https://github.com/.extraheader 2024-12-18T02:56:41.8476576Z Entering 'third_party/pthreadpool' 2024-12-18T02:56:41.8515016Z http.https://github.com/.extraheader 2024-12-18T02:56:41.8548952Z Entering 'third_party/pybind11' 2024-12-18T02:56:41.8585654Z http.https://github.com/.extraheader 2024-12-18T02:56:41.8622000Z Entering 'third_party/python-peachpy' 2024-12-18T02:56:41.8660497Z http.https://github.com/.extraheader 2024-12-18T02:56:41.8698971Z Entering 'third_party/sleef' 2024-12-18T02:56:41.8738882Z http.https://github.com/.extraheader 2024-12-18T02:56:41.8780699Z Entering 'third_party/tensorpipe' 2024-12-18T02:56:41.8818360Z http.https://github.com/.extraheader 2024-12-18T02:56:41.8856361Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T02:56:41.8892886Z http.https://github.com/.extraheader 2024-12-18T02:56:41.8926919Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T02:56:41.8965802Z http.https://github.com/.extraheader 2024-12-18T02:56:41.9000293Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T02:56:41.9037065Z http.https://github.com/.extraheader 2024-12-18T02:56:41.9074668Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T02:56:41.9109623Z http.https://github.com/.extraheader 2024-12-18T02:56:41.9143618Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T02:56:41.9180409Z http.https://github.com/.extraheader 2024-12-18T02:56:41.9314738Z A job completed hook has been configured by the self-hosted runner administrator 2024-12-18T02:56:41.9342670Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-12-18T02:56:41.9348621Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T02:56:41.9349088Z ##[endgroup] 2024-12-18T02:56:41.9433446Z [!ALERT!] Swap in detected! [!ALERT!] 2024-12-18T02:56:53.6700024Z [!ALERT!] Swap out detected [!ALERT!] 2024-12-18T02:57:12.0145649Z Cleaning up orphan processes