2024-11-01T16:42:17.4328595Z Current runner version: '2.320.0' 2024-11-01T16:42:17.4335249Z Runner name: 'i-0961bf540f093269b' 2024-11-01T16:42:17.4335967Z Runner group name: 'Default' 2024-11-01T16:42:17.4336784Z Machine name: 'ip-10-0-37-113' 2024-11-01T16:42:17.4355699Z Testing runner upgrade compatibility 2024-11-01T16:42:17.5519352Z ##[group]GITHUB_TOKEN Permissions 2024-11-01T16:42:17.5521456Z Actions: read 2024-11-01T16:42:17.5522001Z Attestations: read 2024-11-01T16:42:17.5522508Z Checks: read 2024-11-01T16:42:17.5523030Z Contents: read 2024-11-01T16:42:17.5523479Z Deployments: read 2024-11-01T16:42:17.5523983Z Discussions: read 2024-11-01T16:42:17.5524516Z Issues: read 2024-11-01T16:42:17.5524985Z Metadata: read 2024-11-01T16:42:17.5525471Z Packages: read 2024-11-01T16:42:17.5525947Z Pages: read 2024-11-01T16:42:17.5526421Z PullRequests: read 2024-11-01T16:42:17.5526960Z RepositoryProjects: read 2024-11-01T16:42:17.5527568Z SecurityEvents: read 2024-11-01T16:42:17.5528089Z Statuses: read 2024-11-01T16:42:17.5528787Z ##[endgroup] 2024-11-01T16:42:17.5532320Z Secret source: Actions 2024-11-01T16:42:17.5533195Z Prepare workflow directory 2024-11-01T16:42:17.9352216Z Prepare all required actions 2024-11-01T16:42:17.9552135Z Getting action download info 2024-11-01T16:42:18.1525699Z Download action repository 'pytorch/test-infra@main' (SHA:49fb39b5efb49007791d74b09885044cd64544bf) 2024-11-01T16:42:18.5080991Z Download action repository 'pytorch/pytorch@main' (SHA:d79c5143d8e625947b93f8821bbe181419f47dee) 2024-11-01T16:42:21.8063879Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-11-01T16:42:21.9975842Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-11-01T16:42:22.2337998Z Getting action download info 2024-11-01T16:42:22.3363290Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-11-01T16:42:22.4836326Z Getting action download info 2024-11-01T16:42:22.5845873Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2024-11-01T16:42:22.7653560Z Getting action download info 2024-11-01T16:42:22.8645892Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-11-01T16:42:22.9831037Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/138766/merge (368da7510f5d755034681777a4e5ae6d33c07b38) 2024-11-01T16:42:22.9833883Z ##[group] Inputs 2024-11-01T16:42:22.9834466Z build-environment: linux-focal-cuda11.8-py3.10-gcc9 2024-11-01T16:42:22.9836567Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}]} 2024-11-01T16:42:22.9839231Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:42:22.9840435Z sync-tag: 2024-11-01T16:42:22.9841201Z timeout-minutes: 360 2024-11-01T16:42:22.9841559Z use-gha: 2024-11-01T16:42:22.9841859Z dashboard-tag: 2024-11-01T16:42:22.9842199Z s3-bucket: gha-artifacts 2024-11-01T16:42:22.9842567Z aws-role-to-assume: 2024-11-01T16:42:22.9842919Z ##[endgroup] 2024-11-01T16:42:22.9843913Z Complete job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-11-01T16:42:23.0753801Z A job started hook has been configured by the self-hosted runner administrator 2024-11-01T16:42:23.0918582Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-11-01T16:42:23.0929152Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:42:23.0929753Z ##[endgroup] 2024-11-01T16:42:24.6877219Z Runner Type: linux.g4dn.12xlarge.nvidia.gpu 2024-11-01T16:42:24.6878317Z Instance Type: g4dn.12xlarge 2024-11-01T16:42:24.6878746Z AMI Name: unknown 2024-11-01T16:42:24.6879398Z AMI ID: ami-0fff1b9a61dec8a5f 2024-11-01T16:42:30.9454246Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-11-01T16:42:30.9454875Z with: 2024-11-01T16:42:30.9455586Z github-secret: *** 2024-11-01T16:42:30.9456595Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-11-01T16:42:30.9457687Z activate-with-label: false 2024-11-01T16:42:30.9458052Z label: with-ssh 2024-11-01T16:42:30.9458372Z remove-existing-keys: true 2024-11-01T16:42:30.9458729Z fail-silently: true 2024-11-01T16:42:30.9459038Z env: 2024-11-01T16:42:30.9459299Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:42:30.9459644Z ##[endgroup] 2024-11-01T16:42:31.0455021Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-11-01T16:42:31.2879872Z Grabbing public ssh keys from https://github.com/c00w.keys 2024-11-01T16:42:31.3462959Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-11-01T16:42:31.3479490Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-11-01T16:42:31.3509728Z Login using: ssh ec2-user@ec2-100-26-186-158.compute-1.amazonaws.com 2024-11-01T16:42:31.3510737Z All testing is done inside the container, to start an interactive session run: 2024-11-01T16:42:31.3511744Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-11-01T16:42:31.3658992Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-11-01T16:42:31.3659625Z with: 2024-11-01T16:42:31.3659913Z submodules: recursive 2024-11-01T16:42:31.3660285Z fetch-depth: 0 2024-11-01T16:42:31.3660601Z env: 2024-11-01T16:42:31.3661072Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:42:31.3661657Z ##[endgroup] 2024-11-01T16:42:31.3894824Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:42:31.3896256Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:42:31.3905455Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:42:31.3906069Z env: 2024-11-01T16:42:31.3906391Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:42:31.3906796Z ##[endgroup] 2024-11-01T16:42:31.4002935Z ##[group]Run retry () { 2024-11-01T16:42:31.4003384Z retry () { 2024-11-01T16:42:31.4003930Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-11-01T16:42:31.4004535Z } 2024-11-01T16:42:31.4004863Z echo "${GITHUB_WORKSPACE}" 2024-11-01T16:42:31.4005326Z if [ -z "${NO_SUDO}" ]; then 2024-11-01T16:42:31.4005851Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-11-01T16:42:31.4006348Z else 2024-11-01T16:42:31.4006729Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-11-01T16:42:31.4007181Z fi 2024-11-01T16:42:31.4007507Z mkdir "${GITHUB_WORKSPACE}" 2024-11-01T16:42:31.4014069Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:42:31.4014619Z env: 2024-11-01T16:42:31.4014906Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:42:31.4015418Z NO_SUDO: 2024-11-01T16:42:31.4015693Z ##[endgroup] 2024-11-01T16:42:31.4041224Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-11-01T16:42:34.0441996Z ##[group]Run malfet/checkout@silent-checkout 2024-11-01T16:42:34.0442452Z with: 2024-11-01T16:42:34.0442785Z ref: d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:42:34.0443239Z fetch-depth: 0 2024-11-01T16:42:34.0443549Z submodules: recursive 2024-11-01T16:42:34.0443889Z quiet-checkout: true 2024-11-01T16:42:34.0444236Z repository: pytorch/pytorch 2024-11-01T16:42:34.0444742Z token: *** 2024-11-01T16:42:34.0445015Z ssh-strict: true 2024-11-01T16:42:34.0445532Z persist-credentials: true 2024-11-01T16:42:34.0445889Z clean: true 2024-11-01T16:42:34.0446205Z sparse-checkout-cone-mode: true 2024-11-01T16:42:34.0446597Z lfs: false 2024-11-01T16:42:34.0446894Z set-safe-directory: true 2024-11-01T16:42:34.0447217Z env: 2024-11-01T16:42:34.0447481Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:42:34.0447825Z ##[endgroup] 2024-11-01T16:42:34.1550173Z Syncing repository: pytorch/pytorch 2024-11-01T16:42:34.1552120Z ##[group]Getting Git version info 2024-11-01T16:42:34.1553154Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-11-01T16:42:34.1554154Z [command]/usr/bin/git version 2024-11-01T16:42:34.1554564Z git version 2.40.1 2024-11-01T16:42:34.1556105Z ##[endgroup] 2024-11-01T16:42:34.1569624Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/0a43c9f1-a0fd-47a1-b146-6ff14091460f' before making global git config changes 2024-11-01T16:42:34.1571053Z Adding repository directory to the temporary git global config as a safe directory 2024-11-01T16:42:34.1572371Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-11-01T16:42:34.1602970Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-11-01T16:42:34.1607401Z ##[group]Initializing the repository 2024-11-01T16:42:34.1610306Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-11-01T16:42:34.1643956Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-11-01T16:42:34.1645227Z hint: is subject to change. To configure the initial branch name to use in all 2024-11-01T16:42:34.1646176Z hint: of your new repositories, which will suppress this warning, call: 2024-11-01T16:42:34.1646819Z hint: 2024-11-01T16:42:34.1647300Z hint: git config --global init.defaultBranch 2024-11-01T16:42:34.1647825Z hint: 2024-11-01T16:42:34.1648413Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-11-01T16:42:34.1649408Z hint: 'development'. The just-created branch can be renamed via this command: 2024-11-01T16:42:34.1650076Z hint: 2024-11-01T16:42:34.1650409Z hint: git branch -m 2024-11-01T16:42:34.1651287Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-11-01T16:42:34.1653349Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-11-01T16:42:34.1687883Z ##[endgroup] 2024-11-01T16:42:34.1688537Z ##[group]Disabling automatic garbage collection 2024-11-01T16:42:34.1691033Z [command]/usr/bin/git config --local gc.auto 0 2024-11-01T16:42:34.1736896Z ##[endgroup] 2024-11-01T16:42:34.1737506Z ##[group]Setting up auth 2024-11-01T16:42:34.1738335Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-11-01T16:42:34.1764919Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-11-01T16:42:34.2066240Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-11-01T16:42:34.2098039Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-11-01T16:42:34.2390314Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-11-01T16:42:34.2440914Z ##[endgroup] 2024-11-01T16:42:34.2441664Z ##[group]Fetching the repository 2024-11-01T16:42:34.2446110Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-11-01T16:42:37.4213414Z remote: Enumerating objects: 1053138 2024-11-01T16:42:37.4214991Z remote: Enumerating objects: 1053710, done. 2024-11-01T16:42:37.4216698Z remote: Counting objects: 0% (1/572) 2024-11-01T16:42:37.4218641Z remote: Counting objects: 1% (6/572) 2024-11-01T16:42:37.4219401Z remote: Counting objects: 2% (12/572) 2024-11-01T16:42:37.4219977Z remote: Counting objects: 3% (18/572) 2024-11-01T16:42:37.4220546Z remote: Counting objects: 4% (23/572) 2024-11-01T16:42:37.4221223Z remote: Counting objects: 5% (29/572) 2024-11-01T16:42:37.4221797Z remote: Counting objects: 6% (35/572) 2024-11-01T16:42:37.4222370Z remote: Counting objects: 7% (41/572) 2024-11-01T16:42:37.4222947Z remote: Counting objects: 8% (46/572) 2024-11-01T16:42:37.4223495Z remote: Counting objects: 9% (52/572) 2024-11-01T16:42:37.4224062Z remote: Counting objects: 10% (58/572) 2024-11-01T16:42:37.4224631Z remote: Counting objects: 11% (63/572) 2024-11-01T16:42:37.4225194Z remote: Counting objects: 12% (69/572) 2024-11-01T16:42:37.4225760Z remote: Counting objects: 13% (75/572) 2024-11-01T16:42:37.4226330Z remote: Counting objects: 14% (81/572) 2024-11-01T16:42:37.4226990Z remote: Counting objects: 15% (86/572) 2024-11-01T16:42:37.4227536Z remote: Counting objects: 16% (92/572) 2024-11-01T16:42:37.4228079Z remote: Counting objects: 17% (98/572) 2024-11-01T16:42:37.4228631Z remote: Counting objects: 18% (103/572) 2024-11-01T16:42:37.4229238Z remote: Counting objects: 19% (109/572) 2024-11-01T16:42:37.4229802Z remote: Counting objects: 20% (115/572) 2024-11-01T16:42:37.4230528Z remote: Counting objects: 21% (121/572) 2024-11-01T16:42:37.4231081Z remote: Counting objects: 22% (126/572) 2024-11-01T16:42:37.4231642Z remote: Counting objects: 23% (132/572) 2024-11-01T16:42:37.4232204Z remote: Counting objects: 24% (138/572) 2024-11-01T16:42:37.4232759Z remote: Counting objects: 25% (143/572) 2024-11-01T16:42:37.4233320Z remote: Counting objects: 26% (149/572) 2024-11-01T16:42:37.4233887Z remote: Counting objects: 27% (155/572) 2024-11-01T16:42:37.4234426Z remote: Counting objects: 28% (161/572) 2024-11-01T16:42:37.4234983Z remote: Counting objects: 29% (166/572) 2024-11-01T16:42:37.4235541Z remote: Counting objects: 30% (172/572) 2024-11-01T16:42:37.4236096Z remote: Counting objects: 31% (178/572) 2024-11-01T16:42:37.4236651Z remote: Counting objects: 32% (184/572) 2024-11-01T16:42:37.4237190Z remote: Counting objects: 33% (189/572) 2024-11-01T16:42:37.4237750Z remote: Counting objects: 34% (195/572) 2024-11-01T16:42:37.4238305Z remote: Counting objects: 35% (201/572) 2024-11-01T16:42:37.4238859Z remote: Counting objects: 36% (206/572) 2024-11-01T16:42:37.4239413Z remote: Counting objects: 37% (212/572) 2024-11-01T16:42:37.4239964Z remote: Counting objects: 38% (218/572) 2024-11-01T16:42:37.4240502Z remote: Counting objects: 39% (224/572) 2024-11-01T16:42:37.4241065Z remote: Counting objects: 40% (229/572) 2024-11-01T16:42:37.4241619Z remote: Counting objects: 41% (235/572) 2024-11-01T16:42:37.4242172Z remote: Counting objects: 42% (241/572) 2024-11-01T16:42:37.4242727Z remote: Counting objects: 43% (246/572) 2024-11-01T16:42:37.4243279Z remote: Counting objects: 44% (252/572) 2024-11-01T16:42:37.4243818Z remote: Counting objects: 45% (258/572) 2024-11-01T16:42:37.4244369Z remote: Counting objects: 46% (264/572) 2024-11-01T16:42:37.4244930Z remote: Counting objects: 47% (269/572) 2024-11-01T16:42:37.4245483Z remote: Counting objects: 48% (275/572) 2024-11-01T16:42:37.4246042Z remote: Counting objects: 49% (281/572) 2024-11-01T16:42:37.4246578Z remote: Counting objects: 50% (286/572) 2024-11-01T16:42:37.4247128Z remote: Counting objects: 51% (292/572) 2024-11-01T16:42:37.4247686Z remote: Counting objects: 52% (298/572) 2024-11-01T16:42:37.4248242Z remote: Counting objects: 53% (304/572) 2024-11-01T16:42:37.4248891Z remote: Counting objects: 54% (309/572) 2024-11-01T16:42:37.4249452Z remote: Counting objects: 55% (315/572) 2024-11-01T16:42:37.4249997Z remote: Counting objects: 56% (321/572) 2024-11-01T16:42:37.4250553Z remote: Counting objects: 57% (327/572) 2024-11-01T16:42:37.4251111Z remote: Counting objects: 58% (332/572) 2024-11-01T16:42:37.4251667Z remote: Counting objects: 59% (338/572) 2024-11-01T16:42:37.4252225Z remote: Counting objects: 60% (344/572) 2024-11-01T16:42:37.4252763Z remote: Counting objects: 61% (349/572) 2024-11-01T16:42:37.4253321Z remote: Counting objects: 62% (355/572) 2024-11-01T16:42:37.4253887Z remote: Counting objects: 63% (361/572) 2024-11-01T16:42:37.4254441Z remote: Counting objects: 64% (367/572) 2024-11-01T16:42:37.4254997Z remote: Counting objects: 65% (372/572) 2024-11-01T16:42:37.4255547Z remote: Counting objects: 66% (378/572) 2024-11-01T16:42:37.4256106Z remote: Counting objects: 67% (384/572) 2024-11-01T16:42:37.4256656Z remote: Counting objects: 68% (389/572) 2024-11-01T16:42:37.4257194Z remote: Counting objects: 69% (395/572) 2024-11-01T16:42:37.4257750Z remote: Counting objects: 70% (401/572) 2024-11-01T16:42:37.4258303Z remote: Counting objects: 71% (407/572) 2024-11-01T16:42:37.4258858Z remote: Counting objects: 72% (412/572) 2024-11-01T16:42:37.4259481Z remote: Counting objects: 73% (418/572) 2024-11-01T16:42:37.4260023Z remote: Counting objects: 74% (424/572) 2024-11-01T16:42:37.4260578Z remote: Counting objects: 75% (429/572) 2024-11-01T16:42:37.4261389Z remote: Counting objects: 76% (435/572) 2024-11-01T16:42:37.4262042Z remote: Counting objects: 77% (441/572) 2024-11-01T16:42:37.4262614Z remote: Counting objects: 78% (447/572) 2024-11-01T16:42:37.4263191Z remote: Counting objects: 79% (452/572) 2024-11-01T16:42:37.4263765Z remote: Counting objects: 80% (458/572) 2024-11-01T16:42:37.4264339Z remote: Counting objects: 81% (464/572) 2024-11-01T16:42:37.4264910Z remote: Counting objects: 82% (470/572) 2024-11-01T16:42:37.4265476Z remote: Counting objects: 83% (475/572) 2024-11-01T16:42:37.4266044Z remote: Counting objects: 84% (481/572) 2024-11-01T16:42:37.4266599Z remote: Counting objects: 85% (487/572) 2024-11-01T16:42:37.4267268Z remote: Counting objects: 86% (492/572) 2024-11-01T16:42:37.4267828Z remote: Counting objects: 87% (498/572) 2024-11-01T16:42:37.4268382Z remote: Counting objects: 88% (504/572) 2024-11-01T16:42:37.4268937Z remote: Counting objects: 89% (510/572) 2024-11-01T16:42:37.4269487Z remote: Counting objects: 90% (515/572) 2024-11-01T16:42:37.4270032Z remote: Counting objects: 91% (521/572) 2024-11-01T16:42:37.4270587Z remote: Counting objects: 92% (527/572) 2024-11-01T16:42:37.4271147Z remote: Counting objects: 93% (532/572) 2024-11-01T16:42:37.4271699Z remote: Counting objects: 94% (538/572) 2024-11-01T16:42:37.4272249Z remote: Counting objects: 95% (544/572) 2024-11-01T16:42:37.4272801Z remote: Counting objects: 96% (550/572) 2024-11-01T16:42:37.4273338Z remote: Counting objects: 97% (555/572) 2024-11-01T16:42:37.4273892Z remote: Counting objects: 98% (561/572) 2024-11-01T16:42:37.4274444Z remote: Counting objects: 99% (567/572) 2024-11-01T16:42:37.4274999Z remote: Counting objects: 100% (572/572) 2024-11-01T16:42:37.4275597Z remote: Counting objects: 100% (572/572), done. 2024-11-01T16:42:37.4276198Z remote: Compressing objects: 0% (1/311) 2024-11-01T16:42:37.4279240Z remote: Compressing objects: 1% (4/311) 2024-11-01T16:42:37.4419880Z remote: Compressing objects: 2% (7/311) 2024-11-01T16:42:37.4607002Z remote: Compressing objects: 3% (10/311) 2024-11-01T16:42:37.4748693Z remote: Compressing objects: 4% (13/311) 2024-11-01T16:42:37.4881944Z remote: Compressing objects: 5% (16/311) 2024-11-01T16:42:37.5051120Z remote: Compressing objects: 6% (19/311) 2024-11-01T16:42:37.5286980Z remote: Compressing objects: 7% (22/311) 2024-11-01T16:42:37.5463341Z remote: Compressing objects: 8% (25/311) 2024-11-01T16:42:37.5558247Z remote: Compressing objects: 9% (28/311) 2024-11-01T16:42:37.5565792Z remote: Compressing objects: 10% (32/311) 2024-11-01T16:42:37.5566588Z remote: Compressing objects: 11% (35/311) 2024-11-01T16:42:37.5567203Z remote: Compressing objects: 12% (38/311) 2024-11-01T16:42:37.5568107Z remote: Compressing objects: 13% (41/311) 2024-11-01T16:42:37.5568710Z remote: Compressing objects: 14% (44/311) 2024-11-01T16:42:37.5569315Z remote: Compressing objects: 15% (47/311) 2024-11-01T16:42:37.5570148Z remote: Compressing objects: 16% (50/311) 2024-11-01T16:42:37.5570752Z remote: Compressing objects: 17% (53/311) 2024-11-01T16:42:37.5571350Z remote: Compressing objects: 18% (56/311) 2024-11-01T16:42:37.5571938Z remote: Compressing objects: 19% (60/311) 2024-11-01T16:42:37.5575035Z remote: Compressing objects: 20% (63/311) 2024-11-01T16:42:37.5582032Z remote: Compressing objects: 21% (66/311) 2024-11-01T16:42:37.5586008Z remote: Compressing objects: 22% (69/311) 2024-11-01T16:42:37.5588009Z remote: Compressing objects: 23% (72/311) 2024-11-01T16:42:37.5589663Z remote: Compressing objects: 24% (75/311) 2024-11-01T16:42:37.5590613Z remote: Compressing objects: 25% (78/311) 2024-11-01T16:42:37.5592198Z remote: Compressing objects: 26% (81/311) 2024-11-01T16:42:37.5594909Z remote: Compressing objects: 27% (84/311) 2024-11-01T16:42:37.5596271Z remote: Compressing objects: 28% (88/311) 2024-11-01T16:42:37.5597661Z remote: Compressing objects: 29% (91/311) 2024-11-01T16:42:37.5598938Z remote: Compressing objects: 30% (94/311) 2024-11-01T16:42:37.5600090Z remote: Compressing objects: 31% (97/311) 2024-11-01T16:42:37.5601922Z remote: Compressing objects: 32% (100/311) 2024-11-01T16:42:37.5603441Z remote: Compressing objects: 33% (103/311) 2024-11-01T16:42:37.5605700Z remote: Compressing objects: 34% (106/311) 2024-11-01T16:42:37.5613413Z remote: Compressing objects: 35% (109/311) 2024-11-01T16:42:37.5616081Z remote: Compressing objects: 36% (112/311) 2024-11-01T16:42:37.5618530Z remote: Compressing objects: 37% (116/311) 2024-11-01T16:42:37.5622533Z remote: Compressing objects: 38% (119/311) 2024-11-01T16:42:37.5639351Z remote: Compressing objects: 39% (122/311) 2024-11-01T16:42:37.5641633Z remote: Compressing objects: 40% (125/311) 2024-11-01T16:42:37.5643208Z remote: Compressing objects: 41% (128/311) 2024-11-01T16:42:37.5645701Z remote: Compressing objects: 42% (131/311) 2024-11-01T16:42:37.5649322Z remote: Compressing objects: 43% (134/311) 2024-11-01T16:42:37.5651331Z remote: Compressing objects: 44% (137/311) 2024-11-01T16:42:37.5660501Z remote: Compressing objects: 45% (140/311) 2024-11-01T16:42:37.5662536Z remote: Compressing objects: 46% (144/311) 2024-11-01T16:42:37.5665031Z remote: Compressing objects: 47% (147/311) 2024-11-01T16:42:37.5666904Z remote: Compressing objects: 48% (150/311) 2024-11-01T16:42:37.5669512Z remote: Compressing objects: 49% (153/311) 2024-11-01T16:42:37.5670653Z remote: Compressing objects: 50% (156/311) 2024-11-01T16:42:37.5675114Z remote: Compressing objects: 51% (159/311) 2024-11-01T16:42:37.5678363Z remote: Compressing objects: 52% (162/311) 2024-11-01T16:42:37.5679371Z remote: Compressing objects: 53% (165/311) 2024-11-01T16:42:37.5687431Z remote: Compressing objects: 54% (168/311) 2024-11-01T16:42:37.5690002Z remote: Compressing objects: 55% (172/311) 2024-11-01T16:42:37.5690786Z remote: Compressing objects: 56% (175/311) 2024-11-01T16:42:37.5692097Z remote: Compressing objects: 57% (178/311) 2024-11-01T16:42:37.5693288Z remote: Compressing objects: 58% (181/311) 2024-11-01T16:42:37.5694762Z remote: Compressing objects: 59% (184/311) 2024-11-01T16:42:37.5695412Z remote: Compressing objects: 60% (187/311) 2024-11-01T16:42:37.5696273Z remote: Compressing objects: 61% (190/311) 2024-11-01T16:42:37.5696888Z remote: Compressing objects: 62% (193/311) 2024-11-01T16:42:37.5697494Z remote: Compressing objects: 63% (196/311) 2024-11-01T16:42:37.5698109Z remote: Compressing objects: 64% (200/311) 2024-11-01T16:42:37.5698722Z remote: Compressing objects: 65% (203/311) 2024-11-01T16:42:37.5699334Z remote: Compressing objects: 66% (206/311) 2024-11-01T16:42:37.5699955Z remote: Compressing objects: 67% (209/311) 2024-11-01T16:42:37.5700567Z remote: Compressing objects: 68% (212/311) 2024-11-01T16:42:37.5701239Z remote: Compressing objects: 69% (215/311) 2024-11-01T16:42:37.5701858Z remote: Compressing objects: 70% (218/311) 2024-11-01T16:42:37.5702473Z remote: Compressing objects: 71% (221/311) 2024-11-01T16:42:37.5703086Z remote: Compressing objects: 72% (224/311) 2024-11-01T16:42:37.5703709Z remote: Compressing objects: 73% (228/311) 2024-11-01T16:42:37.5704441Z remote: Compressing objects: 74% (231/311) 2024-11-01T16:42:37.5705042Z remote: Compressing objects: 75% (234/311) 2024-11-01T16:42:37.5705653Z remote: Compressing objects: 76% (237/311) 2024-11-01T16:42:37.5706264Z remote: Compressing objects: 77% (240/311) 2024-11-01T16:42:37.5706877Z remote: Compressing objects: 78% (243/311) 2024-11-01T16:42:37.5707495Z remote: Compressing objects: 79% (246/311) 2024-11-01T16:42:37.5708109Z remote: Compressing objects: 80% (249/311) 2024-11-01T16:42:37.5708713Z remote: Compressing objects: 81% (252/311) 2024-11-01T16:42:37.5709555Z remote: Compressing objects: 82% (256/311) 2024-11-01T16:42:37.5710599Z remote: Compressing objects: 83% (259/311) 2024-11-01T16:42:37.5711988Z remote: Compressing objects: 84% (262/311) 2024-11-01T16:42:37.5713106Z remote: Compressing objects: 85% (265/311) 2024-11-01T16:42:37.5714640Z remote: Compressing objects: 86% (268/311) 2024-11-01T16:42:37.5716272Z remote: Compressing objects: 87% (271/311) 2024-11-01T16:42:37.5716931Z remote: Compressing objects: 88% (274/311) 2024-11-01T16:42:37.5717665Z remote: Compressing objects: 89% (277/311) 2024-11-01T16:42:37.5718587Z remote: Compressing objects: 90% (280/311) 2024-11-01T16:42:37.5719501Z remote: Compressing objects: 91% (284/311) 2024-11-01T16:42:37.5720451Z remote: Compressing objects: 92% (287/311) 2024-11-01T16:42:37.5721208Z remote: Compressing objects: 93% (290/311) 2024-11-01T16:42:37.5735300Z remote: Compressing objects: 94% (293/311) 2024-11-01T16:42:37.5738197Z remote: Compressing objects: 95% (296/311) 2024-11-01T16:42:37.5739803Z remote: Compressing objects: 96% (299/311) 2024-11-01T16:42:37.5740984Z remote: Compressing objects: 97% (302/311) 2024-11-01T16:42:37.5742279Z remote: Compressing objects: 98% (305/311) 2024-11-01T16:42:37.5743304Z remote: Compressing objects: 99% (308/311) 2024-11-01T16:42:37.5744234Z remote: Compressing objects: 100% (311/311) 2024-11-01T16:42:37.5744895Z remote: Compressing objects: 100% (311/311), done. 2024-11-01T16:43:00.8438465Z remote: Total 1053710 (delta 340), reused 441 (delta 260), pack-reused 1053138 (from 1) 2024-11-01T16:43:21.2111274Z [command]/usr/bin/git rev-parse --verify --quiet d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea^{object} 2024-11-01T16:43:21.2142580Z d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:43:21.2147505Z ##[endgroup] 2024-11-01T16:43:21.2148159Z ##[group]Determining the checkout info 2024-11-01T16:43:21.2148869Z ##[endgroup] 2024-11-01T16:43:21.2149444Z ##[group]Checking out the ref 2024-11-01T16:43:21.2151993Z [command]/usr/bin/git checkout --quiet --force d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:43:22.7313543Z ##[endgroup] 2024-11-01T16:43:22.7314341Z ##[group]Setting up auth for fetching submodules 2024-11-01T16:43:22.7319656Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-11-01T16:43:22.7378859Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-11-01T16:43:22.7410322Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-11-01T16:43:22.7443815Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-11-01T16:43:22.7474211Z ##[endgroup] 2024-11-01T16:43:22.7474800Z ##[group]Fetching submodules 2024-11-01T16:43:22.7478203Z [command]/usr/bin/git submodule sync --recursive 2024-11-01T16:43:22.7806336Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-11-01T16:43:22.8128710Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-11-01T16:43:22.8130721Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-11-01T16:43:22.8132718Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-11-01T16:43:22.8135245Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-11-01T16:43:22.8137936Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2024-11-01T16:43:22.8141642Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-11-01T16:43:22.8144448Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-11-01T16:43:22.8147779Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-11-01T16:43:22.8151408Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel' 2024-11-01T16:43:22.8154943Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-11-01T16:43:22.8158420Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-11-01T16:43:22.8162737Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-11-01T16:43:22.8166380Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-11-01T16:43:22.8170295Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-11-01T16:43:22.8174344Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-11-01T16:43:22.8179269Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-11-01T16:43:22.8185391Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-11-01T16:43:22.8190292Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:43:22.8194827Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-11-01T16:43:22.8199707Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-11-01T16:43:22.8204340Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-11-01T16:43:22.8209312Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-11-01T16:43:22.8214343Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-11-01T16:43:22.8219570Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-11-01T16:43:22.8225201Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-11-01T16:43:22.8230864Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-11-01T16:43:22.8236483Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-11-01T16:43:22.8242502Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-11-01T16:43:22.8247961Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-11-01T16:43:22.8254053Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-11-01T16:43:22.8259926Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-11-01T16:43:22.8266676Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-11-01T16:43:22.8272914Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-11-01T16:43:22.8281404Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-11-01T16:43:22.8287790Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-11-01T16:43:22.8294546Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-11-01T16:43:22.8326484Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-11-01T16:43:23.2137254Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-11-01T16:43:23.3990184Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-11-01T16:43:23.6023252Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-11-01T16:43:23.8465354Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2024-11-01T16:43:24.1734204Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-11-01T16:43:26.0439150Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-11-01T16:43:35.3096236Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-11-01T16:43:35.7546942Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'... 2024-11-01T16:43:37.4094931Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-11-01T16:43:37.8702524Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-11-01T16:43:38.4862358Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-11-01T16:43:39.7270832Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-11-01T16:43:41.9067832Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-11-01T16:43:47.3230570Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-11-01T16:43:48.7807977Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-11-01T16:43:50.4417800Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-11-01T16:43:51.5583786Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-11-01T16:43:52.0279807Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-11-01T16:43:52.4504864Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-11-01T16:43:53.4108093Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-11-01T16:43:53.7814962Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-11-01T16:43:54.1245172Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-11-01T16:43:55.6879675Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-11-01T16:43:56.4193190Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-11-01T16:43:56.8015955Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-11-01T16:44:02.4514081Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-11-01T16:44:04.2639097Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-11-01T16:44:11.5678529Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-11-01T16:44:11.8459603Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-11-01T16:44:20.9236216Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-11-01T16:44:21.1212179Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-11-01T16:44:21.3214704Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-11-01T16:44:22.3303228Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-11-01T16:44:22.6747534Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-11-01T16:44:23.3747956Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-11-01T16:44:23.7904655Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-11-01T16:44:23.8027942Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-11-01T16:44:23.8133208Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-11-01T16:44:23.8385443Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-11-01T16:44:23.8699738Z Submodule path 'third_party/NVTX': checked out 'e170594ac7cf1dac584da473d4ca9301087090c1' 2024-11-01T16:44:23.9105144Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-11-01T16:44:24.7501040Z Submodule path 'third_party/XNNPACK': checked out '87ee0b46b834f67bad9025d4a82ed5654f3403d3' 2024-11-01T16:44:24.7745961Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-11-01T16:44:24.9947695Z Submodule path 'third_party/composable_kernel': checked out 'cedccd59c94cb0c74e7ec0d0f6c791aed081febc' 2024-11-01T16:44:25.0447900Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-11-01T16:44:25.1459904Z Submodule path 'third_party/cpuinfo': checked out '1e83a2fdd3102f65c6f1fb602c1b320486218a99' 2024-11-01T16:44:25.1809638Z Submodule path 'third_party/cudnn_frontend': checked out '936021bfed8c91dc416af1588b2c4eca631a9e45' 2024-11-01T16:44:25.7102744Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-11-01T16:44:25.9791490Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-11-01T16:44:26.0762035Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-11-01T16:44:26.0778648Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:44:26.0780465Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:44:26.0782885Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:44:26.0785892Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:44:26.0789063Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:44:26.0819919Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-11-01T16:44:27.0128135Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-11-01T16:44:27.6214682Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-11-01T16:44:29.7728021Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-11-01T16:44:30.7272229Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-11-01T16:44:31.1331846Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-11-01T16:44:31.2334080Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-11-01T16:44:31.6575649Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-11-01T16:44:31.7247278Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-11-01T16:44:31.7386815Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-11-01T16:44:31.8618034Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-11-01T16:44:31.9024072Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-11-01T16:44:31.9428657Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-11-01T16:44:31.9699869Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-11-01T16:44:32.0170527Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-11-01T16:44:32.0304566Z Submodule path 'third_party/ideep': checked out '41d636c2bbcea6bff0faf97cdb65a48cdde987af' 2024-11-01T16:44:32.0323941Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-11-01T16:44:32.0348575Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-11-01T16:44:45.0135049Z Submodule path 'third_party/ideep/mkl-dnn': checked out '66f0cb9eb66affd2da3bf5f8d897376f04aae6af' 2024-11-01T16:44:45.0332598Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-11-01T16:44:45.1263932Z Submodule path 'third_party/kineto': checked out 'ed052ea024b9468908d558b15cd3f7584fb0f492' 2024-11-01T16:44:45.1281899Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:44:45.1283969Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:44:45.1286603Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:44:45.1318141Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-11-01T16:44:45.6554849Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-11-01T16:44:46.7044841Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-11-01T16:44:47.7257866Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-11-01T16:44:47.7277834Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:44:47.7280203Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:44:47.7282242Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:44:47.7284322Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:44:47.7286877Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:44:47.7289718Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:44:47.7293473Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:44:47.7296920Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:44:47.7328791Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-11-01T16:44:48.5628544Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-11-01T16:44:48.9679676Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-11-01T16:44:50.1793080Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-11-01T16:44:50.4763529Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-11-01T16:44:51.1200766Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-11-01T16:44:52.0591077Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-11-01T16:44:59.1090610Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-11-01T16:44:59.5168684Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-11-01T16:44:59.5370999Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-11-01T16:44:59.5754291Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-11-01T16:44:59.5895006Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-11-01T16:44:59.5910371Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:44:59.5943956Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-11-01T16:44:59.9081383Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-11-01T16:44:59.9277699Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-11-01T16:44:59.9701387Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-11-01T16:45:00.0783199Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-11-01T16:45:00.0959114Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-11-01T16:45:00.1360285Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-11-01T16:45:00.1969602Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-11-01T16:45:00.2349370Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-11-01T16:45:00.2663439Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-11-01T16:45:00.3738860Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-11-01T16:45:00.7483421Z Submodule path 'third_party/onnx': checked out 'b8baa8446686496da4cc8fda09f2b6fe65c2a02c' 2024-11-01T16:45:00.7521783Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-11-01T16:45:00.7552007Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-11-01T16:45:01.8304800Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-11-01T16:45:01.9018890Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-11-01T16:45:01.9040049Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:45:01.9042077Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:45:01.9044020Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:45:01.9046031Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:45:01.9049325Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:45:01.9052240Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:45:01.9055451Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:45:01.9059007Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:45:01.9088295Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-11-01T16:45:02.3474610Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-11-01T16:45:03.2612240Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-11-01T16:45:03.5880713Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-11-01T16:45:09.1679483Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-11-01T16:45:09.4723904Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-11-01T16:45:09.6928104Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-11-01T16:45:10.0075827Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-11-01T16:45:16.1978394Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-11-01T16:45:16.2407912Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-11-01T16:45:16.2571853Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-11-01T16:45:16.3606078Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-11-01T16:45:16.3742697Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-11-01T16:45:16.3895297Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-11-01T16:45:16.4059134Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-11-01T16:45:16.4078373Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:45:16.4080643Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:45:16.4106373Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-11-01T16:45:18.0725953Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-11-01T16:45:19.2142846Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-11-01T16:45:19.2640462Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-11-01T16:45:19.7443164Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-11-01T16:45:19.7570713Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-11-01T16:45:20.0634008Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-11-01T16:45:20.0658703Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:45:20.0660718Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-11-01T16:45:20.0689693Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-11-01T16:45:20.5269072Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-11-01T16:45:21.4339800Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-11-01T16:45:21.5105167Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-11-01T16:45:21.5203844Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-11-01T16:45:21.5334327Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-11-01T16:45:21.5704132Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2024-11-01T16:45:21.6011350Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-11-01T16:45:21.6457125Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-11-01T16:45:21.6732578Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-11-01T16:45:21.6749191Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:45:21.6751604Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:45:21.6754277Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:45:21.6757205Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:45:21.6784971Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-11-01T16:45:22.5603462Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-11-01T16:45:22.7827863Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-11-01T16:45:24.0282984Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-11-01T16:45:25.0845620Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-11-01T16:45:25.1003624Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-11-01T16:45:25.1736919Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-11-01T16:45:25.2039173Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-11-01T16:45:25.2056412Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:45:25.2082284Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-11-01T16:45:25.4065307Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-11-01T16:45:25.4108187Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-11-01T16:45:25.4448466Z Entering 'android/libs/fbjni' 2024-11-01T16:45:25.4495000Z Entering 'third_party/FP16' 2024-11-01T16:45:25.4541932Z Entering 'third_party/FXdiv' 2024-11-01T16:45:25.4587549Z Entering 'third_party/NNPACK' 2024-11-01T16:45:25.4632193Z Entering 'third_party/NVTX' 2024-11-01T16:45:25.4678644Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T16:45:25.4722203Z Entering 'third_party/XNNPACK' 2024-11-01T16:45:25.4781555Z Entering 'third_party/benchmark' 2024-11-01T16:45:25.4827881Z Entering 'third_party/composable_kernel' 2024-11-01T16:45:25.4878985Z Entering 'third_party/cpp-httplib' 2024-11-01T16:45:25.4923418Z Entering 'third_party/cpuinfo' 2024-11-01T16:45:25.4967095Z Entering 'third_party/cudnn_frontend' 2024-11-01T16:45:25.5013750Z Entering 'third_party/cutlass' 2024-11-01T16:45:25.5064806Z Entering 'third_party/eigen' 2024-11-01T16:45:25.5112726Z Entering 'third_party/fbgemm' 2024-11-01T16:45:25.5160892Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:45:25.5203178Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:45:25.5245357Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:45:25.5297139Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:45:25.5340823Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:45:25.5387031Z Entering 'third_party/flatbuffers' 2024-11-01T16:45:25.5434267Z Entering 'third_party/fmt' 2024-11-01T16:45:25.5477863Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:45:25.5521330Z Entering 'third_party/gloo' 2024-11-01T16:45:25.5564738Z Entering 'third_party/googletest' 2024-11-01T16:45:25.5609818Z Entering 'third_party/ideep' 2024-11-01T16:45:25.5653515Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T16:45:25.5705140Z Entering 'third_party/ittapi' 2024-11-01T16:45:25.5748371Z Entering 'third_party/kineto' 2024-11-01T16:45:25.5793992Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:45:25.5837596Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:45:25.5880256Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:45:25.5924373Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:45:25.5967116Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:45:25.6010724Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:45:25.6056778Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:45:25.6098704Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:45:25.6149650Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:45:25.6201171Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:45:25.6245564Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:45:25.6286606Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:45:25.6332560Z Entering 'third_party/mimalloc' 2024-11-01T16:45:25.6376450Z Entering 'third_party/nccl/nccl' 2024-11-01T16:45:25.6421219Z Entering 'third_party/nlohmann' 2024-11-01T16:45:25.6464016Z Entering 'third_party/onnx' 2024-11-01T16:45:25.6527469Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T16:45:25.6572292Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T16:45:25.6618939Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:45:25.6661206Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:45:25.6711061Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:45:25.6750804Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:45:25.6797060Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:45:25.6846453Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:45:25.6887417Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:45:25.6933711Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:45:25.6979222Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:45:25.7025454Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:45:25.7092663Z Entering 'third_party/pocketfft' 2024-11-01T16:45:25.7138666Z Entering 'third_party/protobuf' 2024-11-01T16:45:25.7185501Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:45:25.7230833Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T16:45:25.7277939Z Entering 'third_party/psimd' 2024-11-01T16:45:25.7321235Z Entering 'third_party/pthreadpool' 2024-11-01T16:45:25.7364160Z Entering 'third_party/pybind11' 2024-11-01T16:45:25.7407004Z Entering 'third_party/python-peachpy' 2024-11-01T16:45:25.7449876Z Entering 'third_party/sleef' 2024-11-01T16:45:25.7495375Z Entering 'third_party/tensorpipe' 2024-11-01T16:45:25.7542196Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:45:25.7583935Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:45:25.7625376Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:45:25.7668824Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:45:25.7717829Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:45:25.7777800Z ##[endgroup] 2024-11-01T16:45:25.7780062Z ##[group]Persisting credentials for submodules 2024-11-01T16:45:25.7782116Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-11-01T16:45:25.8104044Z Entering 'android/libs/fbjni' 2024-11-01T16:45:25.8160902Z Entering 'third_party/FP16' 2024-11-01T16:45:25.8221184Z Entering 'third_party/FXdiv' 2024-11-01T16:45:25.8278479Z Entering 'third_party/NNPACK' 2024-11-01T16:45:25.8335860Z Entering 'third_party/NVTX' 2024-11-01T16:45:25.8392439Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T16:45:25.8449759Z Entering 'third_party/XNNPACK' 2024-11-01T16:45:25.8525784Z Entering 'third_party/benchmark' 2024-11-01T16:45:25.8581349Z Entering 'third_party/composable_kernel' 2024-11-01T16:45:25.8647033Z Entering 'third_party/cpp-httplib' 2024-11-01T16:45:25.8702859Z Entering 'third_party/cpuinfo' 2024-11-01T16:45:25.8763962Z Entering 'third_party/cudnn_frontend' 2024-11-01T16:45:25.8821660Z Entering 'third_party/cutlass' 2024-11-01T16:45:25.8886517Z Entering 'third_party/eigen' 2024-11-01T16:45:25.8946009Z Entering 'third_party/fbgemm' 2024-11-01T16:45:25.9003135Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:45:25.9065004Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:45:25.9123465Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:45:25.9185895Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:45:25.9241002Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:45:25.9298026Z Entering 'third_party/flatbuffers' 2024-11-01T16:45:25.9358821Z Entering 'third_party/fmt' 2024-11-01T16:45:25.9419129Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:45:25.9477617Z Entering 'third_party/gloo' 2024-11-01T16:45:25.9534821Z Entering 'third_party/googletest' 2024-11-01T16:45:25.9592058Z Entering 'third_party/ideep' 2024-11-01T16:45:25.9648393Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T16:45:25.9710473Z Entering 'third_party/ittapi' 2024-11-01T16:45:25.9766843Z Entering 'third_party/kineto' 2024-11-01T16:45:25.9829430Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:45:25.9884503Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:45:25.9945778Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:45:26.0003529Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:45:26.0059866Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:45:26.0123662Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:45:26.0181179Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:45:26.0244709Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:45:26.0302308Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:45:26.0362405Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:45:26.0421962Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:45:26.0478313Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:45:26.0540207Z Entering 'third_party/mimalloc' 2024-11-01T16:45:26.0597884Z Entering 'third_party/nccl/nccl' 2024-11-01T16:45:26.0660601Z Entering 'third_party/nlohmann' 2024-11-01T16:45:26.0722104Z Entering 'third_party/onnx' 2024-11-01T16:45:26.0797367Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T16:45:26.0860859Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T16:45:26.0921013Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:45:26.0980776Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:45:26.1037324Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:45:26.1096991Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:45:26.1163169Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:45:26.1219652Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:45:26.1277397Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:45:26.1334523Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:45:26.1398900Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:45:26.1462320Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:45:26.1542621Z Entering 'third_party/pocketfft' 2024-11-01T16:45:26.1599203Z Entering 'third_party/protobuf' 2024-11-01T16:45:26.1658938Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:45:26.1718166Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T16:45:26.1777527Z Entering 'third_party/psimd' 2024-11-01T16:45:26.1841335Z Entering 'third_party/pthreadpool' 2024-11-01T16:45:26.1897479Z Entering 'third_party/pybind11' 2024-11-01T16:45:26.1951875Z Entering 'third_party/python-peachpy' 2024-11-01T16:45:26.2010573Z Entering 'third_party/sleef' 2024-11-01T16:45:26.2066760Z Entering 'third_party/tensorpipe' 2024-11-01T16:45:26.2124617Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:45:26.2179630Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:45:26.2239837Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:45:26.2296545Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:45:26.2359503Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:45:26.2439550Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-11-01T16:45:26.2766258Z Entering 'android/libs/fbjni' 2024-11-01T16:45:26.2818344Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-11-01T16:45:26.2836698Z Entering 'third_party/FP16' 2024-11-01T16:45:26.2891683Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-11-01T16:45:26.2906846Z Entering 'third_party/FXdiv' 2024-11-01T16:45:26.2959556Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-11-01T16:45:26.2978241Z Entering 'third_party/NNPACK' 2024-11-01T16:45:26.3028366Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-11-01T16:45:26.3047167Z Entering 'third_party/NVTX' 2024-11-01T16:45:26.3100244Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2024-11-01T16:45:26.3120403Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T16:45:26.3172025Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-11-01T16:45:26.3188303Z Entering 'third_party/XNNPACK' 2024-11-01T16:45:26.3241272Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-11-01T16:45:26.3277881Z Entering 'third_party/benchmark' 2024-11-01T16:45:26.3330324Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-11-01T16:45:26.3345879Z Entering 'third_party/composable_kernel' 2024-11-01T16:45:26.3400162Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2024-11-01T16:45:26.3423183Z Entering 'third_party/cpp-httplib' 2024-11-01T16:45:26.3476872Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-11-01T16:45:26.3495456Z Entering 'third_party/cpuinfo' 2024-11-01T16:45:26.3544599Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-11-01T16:45:26.3564033Z Entering 'third_party/cudnn_frontend' 2024-11-01T16:45:26.3617229Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-11-01T16:45:26.3636394Z Entering 'third_party/cutlass' 2024-11-01T16:45:26.3686409Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-11-01T16:45:26.3711739Z Entering 'third_party/eigen' 2024-11-01T16:45:26.3766628Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-11-01T16:45:26.3785524Z Entering 'third_party/fbgemm' 2024-11-01T16:45:26.3838972Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-11-01T16:45:26.3856182Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:45:26.3908720Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-11-01T16:45:26.3927371Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:45:26.3979225Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-11-01T16:45:26.3996756Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:45:26.4046988Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-11-01T16:45:26.4070061Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:45:26.4158985Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:45:26.4161638Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:45:26.4192804Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-11-01T16:45:26.4213355Z Entering 'third_party/flatbuffers' 2024-11-01T16:45:26.4264005Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-11-01T16:45:26.4283822Z Entering 'third_party/fmt' 2024-11-01T16:45:26.4336975Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-11-01T16:45:26.4357272Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:45:26.4409580Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-11-01T16:45:26.4425461Z Entering 'third_party/gloo' 2024-11-01T16:45:26.4479205Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-11-01T16:45:26.4498048Z Entering 'third_party/googletest' 2024-11-01T16:45:26.4551337Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:45:26.4570188Z Entering 'third_party/ideep' 2024-11-01T16:45:26.4622416Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-11-01T16:45:26.4639268Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T16:45:26.4692999Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-11-01T16:45:26.4721164Z Entering 'third_party/ittapi' 2024-11-01T16:45:26.4773160Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-11-01T16:45:26.4788840Z Entering 'third_party/kineto' 2024-11-01T16:45:26.4843765Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-11-01T16:45:26.4860970Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:45:26.4917059Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-11-01T16:45:26.4934679Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:45:26.4987585Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-11-01T16:45:26.5007562Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:45:26.5060304Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-11-01T16:45:26.5079100Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:45:26.5133209Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-11-01T16:45:26.5148901Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:45:26.5204145Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-11-01T16:45:26.5219662Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:45:26.5271225Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-11-01T16:45:26.5291256Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:45:26.5343853Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-11-01T16:45:26.5361406Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:45:26.5414542Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:45:26.5430002Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:45:26.5483481Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-11-01T16:45:26.5502615Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:45:26.5557511Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-11-01T16:45:26.5576960Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:45:26.5628974Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-11-01T16:45:26.5646610Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:45:26.5700357Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-11-01T16:45:26.5721201Z Entering 'third_party/mimalloc' 2024-11-01T16:45:26.5774049Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-11-01T16:45:26.5790648Z Entering 'third_party/nccl/nccl' 2024-11-01T16:45:26.5844050Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-11-01T16:45:26.5862091Z Entering 'third_party/nlohmann' 2024-11-01T16:45:26.5917227Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-11-01T16:45:26.5936255Z Entering 'third_party/onnx' 2024-11-01T16:45:26.5988877Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-11-01T16:45:26.6022759Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T16:45:26.6079176Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-11-01T16:45:26.6101386Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T16:45:26.6152517Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-11-01T16:45:26.6173131Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:45:26.6225436Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-11-01T16:45:26.6243858Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:45:26.6296225Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:45:26.6311649Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:45:26.6366482Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-11-01T16:45:26.6381715Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:45:26.6433094Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-11-01T16:45:26.6453700Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:45:26.6504862Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-11-01T16:45:26.6522436Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:45:26.6574201Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-11-01T16:45:26.6589489Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:45:26.6642964Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-11-01T16:45:26.6659265Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:45:26.6712665Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-11-01T16:45:26.6733556Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:45:26.6786240Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-11-01T16:45:26.6807581Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:45:26.6858734Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-11-01T16:45:26.6900338Z Entering 'third_party/pocketfft' 2024-11-01T16:45:26.6950217Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-11-01T16:45:26.6968758Z Entering 'third_party/protobuf' 2024-11-01T16:45:26.7023182Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-11-01T16:45:26.7043097Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:45:26.7096541Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-11-01T16:45:26.7111505Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T16:45:26.7164603Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:45:26.7182604Z Entering 'third_party/psimd' 2024-11-01T16:45:26.7236023Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-11-01T16:45:26.7253131Z Entering 'third_party/pthreadpool' 2024-11-01T16:45:26.7306272Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-11-01T16:45:26.7325252Z Entering 'third_party/pybind11' 2024-11-01T16:45:26.7378132Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-11-01T16:45:26.7396445Z Entering 'third_party/python-peachpy' 2024-11-01T16:45:26.7447785Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-11-01T16:45:26.7463422Z Entering 'third_party/sleef' 2024-11-01T16:45:26.7517977Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-11-01T16:45:26.7534996Z Entering 'third_party/tensorpipe' 2024-11-01T16:45:26.7587588Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-11-01T16:45:26.7605438Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:45:26.7658155Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:45:26.7675900Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:45:26.7730427Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-11-01T16:45:26.7745186Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:45:26.7801094Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-11-01T16:45:26.7819108Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:45:26.7871965Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-11-01T16:45:26.7886948Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:45:26.7940239Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-11-01T16:45:26.8566040Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-11-01T16:45:26.8888941Z Entering 'android/libs/fbjni' 2024-11-01T16:45:26.8935343Z Entering 'third_party/FP16' 2024-11-01T16:45:26.8979293Z Entering 'third_party/FXdiv' 2024-11-01T16:45:26.9026216Z Entering 'third_party/NNPACK' 2024-11-01T16:45:26.9069366Z Entering 'third_party/NVTX' 2024-11-01T16:45:26.9116371Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T16:45:26.9160686Z Entering 'third_party/XNNPACK' 2024-11-01T16:45:26.9222379Z Entering 'third_party/benchmark' 2024-11-01T16:45:26.9264769Z Entering 'third_party/composable_kernel' 2024-11-01T16:45:26.9317470Z Entering 'third_party/cpp-httplib' 2024-11-01T16:45:26.9361281Z Entering 'third_party/cpuinfo' 2024-11-01T16:45:26.9406299Z Entering 'third_party/cudnn_frontend' 2024-11-01T16:45:26.9449050Z Entering 'third_party/cutlass' 2024-11-01T16:45:26.9502681Z Entering 'third_party/eigen' 2024-11-01T16:45:26.9546797Z Entering 'third_party/fbgemm' 2024-11-01T16:45:26.9591391Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:45:26.9635438Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:45:26.9678821Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:45:26.9731244Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:45:26.9774287Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:45:26.9819860Z Entering 'third_party/flatbuffers' 2024-11-01T16:45:26.9864982Z Entering 'third_party/fmt' 2024-11-01T16:45:26.9909718Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:45:26.9952566Z Entering 'third_party/gloo' 2024-11-01T16:45:26.9997563Z Entering 'third_party/googletest' 2024-11-01T16:45:27.0039880Z Entering 'third_party/ideep' 2024-11-01T16:45:27.0080278Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T16:45:27.0134326Z Entering 'third_party/ittapi' 2024-11-01T16:45:27.0178508Z Entering 'third_party/kineto' 2024-11-01T16:45:27.0222868Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:45:27.0265467Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:45:27.0311243Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:45:27.0363709Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:45:27.0407169Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:45:27.0449075Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:45:27.0496908Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:45:27.0540084Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:45:27.0590335Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:45:27.0633698Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:45:27.0687054Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:45:27.0730847Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:45:27.0775930Z Entering 'third_party/mimalloc' 2024-11-01T16:45:27.0819949Z Entering 'third_party/nccl/nccl' 2024-11-01T16:45:27.0863796Z Entering 'third_party/nlohmann' 2024-11-01T16:45:27.0910893Z Entering 'third_party/onnx' 2024-11-01T16:45:27.0971756Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T16:45:27.1019776Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T16:45:27.1064346Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:45:27.1108923Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:45:27.1152038Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:45:27.1196766Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:45:27.1244893Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:45:27.1287471Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:45:27.1331380Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:45:27.1375313Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:45:27.1421093Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:45:27.1464532Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:45:27.1532682Z Entering 'third_party/pocketfft' 2024-11-01T16:45:27.1578792Z Entering 'third_party/protobuf' 2024-11-01T16:45:27.1624022Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:45:27.1669084Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T16:45:27.1721598Z Entering 'third_party/psimd' 2024-11-01T16:45:27.1763692Z Entering 'third_party/pthreadpool' 2024-11-01T16:45:27.1808206Z Entering 'third_party/pybind11' 2024-11-01T16:45:27.1853400Z Entering 'third_party/python-peachpy' 2024-11-01T16:45:27.1897552Z Entering 'third_party/sleef' 2024-11-01T16:45:27.1941326Z Entering 'third_party/tensorpipe' 2024-11-01T16:45:27.1988824Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:45:27.2029424Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:45:27.2072465Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:45:27.2118936Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:45:27.2160824Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:45:27.2226453Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-11-01T16:45:27.2546333Z Entering 'android/libs/fbjni' 2024-11-01T16:45:27.2589953Z Entering 'third_party/FP16' 2024-11-01T16:45:27.2637775Z Entering 'third_party/FXdiv' 2024-11-01T16:45:27.2679809Z Entering 'third_party/NNPACK' 2024-11-01T16:45:27.2722848Z Entering 'third_party/NVTX' 2024-11-01T16:45:27.2766669Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T16:45:27.2818610Z Entering 'third_party/XNNPACK' 2024-11-01T16:45:27.2878870Z Entering 'third_party/benchmark' 2024-11-01T16:45:27.2923920Z Entering 'third_party/composable_kernel' 2024-11-01T16:45:27.2974567Z Entering 'third_party/cpp-httplib' 2024-11-01T16:45:27.3018430Z Entering 'third_party/cpuinfo' 2024-11-01T16:45:27.3062876Z Entering 'third_party/cudnn_frontend' 2024-11-01T16:45:27.3107668Z Entering 'third_party/cutlass' 2024-11-01T16:45:27.3167206Z Entering 'third_party/eigen' 2024-11-01T16:45:27.3216483Z Entering 'third_party/fbgemm' 2024-11-01T16:45:27.3260417Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:45:27.3303356Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:45:27.3345039Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:45:27.3398142Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:45:27.3442032Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:45:27.3484245Z Entering 'third_party/flatbuffers' 2024-11-01T16:45:27.3532150Z Entering 'third_party/fmt' 2024-11-01T16:45:27.3576264Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:45:27.3621908Z Entering 'third_party/gloo' 2024-11-01T16:45:27.3663808Z Entering 'third_party/googletest' 2024-11-01T16:45:27.3709269Z Entering 'third_party/ideep' 2024-11-01T16:45:27.3751147Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T16:45:27.3804276Z Entering 'third_party/ittapi' 2024-11-01T16:45:27.3846738Z Entering 'third_party/kineto' 2024-11-01T16:45:27.3888918Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:45:27.3934348Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:45:27.3979089Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:45:27.4027955Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:45:27.4070929Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:45:27.4116589Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:45:27.4162419Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:45:27.4207014Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:45:27.4250538Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:45:27.4296705Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:45:27.4347390Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:45:27.4398609Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:45:27.4442623Z Entering 'third_party/mimalloc' 2024-11-01T16:45:27.4484819Z Entering 'third_party/nccl/nccl' 2024-11-01T16:45:27.4529499Z Entering 'third_party/nlohmann' 2024-11-01T16:45:27.4575623Z Entering 'third_party/onnx' 2024-11-01T16:45:27.4636372Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T16:45:27.4684614Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T16:45:27.4731264Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:45:27.4779474Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:45:27.4825334Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:45:27.4867083Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:45:27.4922922Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:45:27.4965185Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:45:27.5007764Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:45:27.5050227Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:45:27.5099991Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:45:27.5143374Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:45:27.5209184Z Entering 'third_party/pocketfft' 2024-11-01T16:45:27.5254018Z Entering 'third_party/protobuf' 2024-11-01T16:45:27.5302049Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:45:27.5345832Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T16:45:27.5395565Z Entering 'third_party/psimd' 2024-11-01T16:45:27.5440342Z Entering 'third_party/pthreadpool' 2024-11-01T16:45:27.5481688Z Entering 'third_party/pybind11' 2024-11-01T16:45:27.5525614Z Entering 'third_party/python-peachpy' 2024-11-01T16:45:27.5568937Z Entering 'third_party/sleef' 2024-11-01T16:45:27.5613495Z Entering 'third_party/tensorpipe' 2024-11-01T16:45:27.5657806Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:45:27.5701956Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:45:27.5746416Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:45:27.5789553Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:45:27.5833326Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:45:27.5896533Z ##[endgroup] 2024-11-01T16:45:27.5935922Z [command]/usr/bin/git log -1 --format='%H' 2024-11-01T16:45:27.5965905Z 'd1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea' 2024-11-01T16:45:27.6175681Z Prepare all required actions 2024-11-01T16:45:27.6176190Z Getting action download info 2024-11-01T16:45:27.7934776Z ##[group]Run ./.github/actions/setup-linux 2024-11-01T16:45:27.7935207Z env: 2024-11-01T16:45:27.7935475Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:27.7935821Z ##[endgroup] 2024-11-01T16:45:27.8003879Z ##[group]Run set -euo pipefail 2024-11-01T16:45:27.8004313Z set -euo pipefail 2024-11-01T16:45:27.8004711Z function get_ec2_metadata() { 2024-11-01T16:45:27.8005267Z  # Pulled from instance metadata endpoint for EC2 2024-11-01T16:45:27.8006205Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-11-01T16:45:27.8007045Z  category=$1 2024-11-01T16:45:27.8007588Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-11-01T16:45:27.8008249Z  runner_name_str=i-0961bf540f093269b 2024-11-01T16:45:27.8008762Z  if [[ -f /.inarc ]]; then 2024-11-01T16:45:27.8009251Z  echo "ARC Runner, no info on ec2 metadata" 2024-11-01T16:45:27.8009837Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-11-01T16:45:27.8010557Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-11-01T16:45:27.8011192Z  else 2024-11-01T16:45:27.8012507Z  curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-11-01T16:45:27.8013912Z  fi 2024-11-01T16:45:27.8014181Z } 2024-11-01T16:45:27.8014544Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-11-01T16:45:27.8015151Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-11-01T16:45:27.8015836Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-11-01T16:45:27.8016424Z echo "system info $(uname -a)" 2024-11-01T16:45:27.8022887Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:27.8023469Z env: 2024-11-01T16:45:27.8023758Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:27.8024144Z ##[endgroup] 2024-11-01T16:45:27.8169777Z ami-id: ami-0fff1b9a61dec8a5f 2024-11-01T16:45:27.8280964Z instance-id: i-0961bf540f093269b 2024-11-01T16:45:27.8399780Z instance-type: g4dn.12xlarge 2024-11-01T16:45:27.8410515Z system info Linux ip-10-0-37-113.ec2.internal 6.1.109-118.189.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 10 08:59:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-11-01T16:45:27.8438352Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:45:27.8439747Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:45:27.8446147Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:27.8446662Z env: 2024-11-01T16:45:27.8446944Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:27.8447298Z ##[endgroup] 2024-11-01T16:45:27.8505968Z ##[group]Run if systemctl is-active --quiet docker; then 2024-11-01T16:45:27.8506656Z if systemctl is-active --quiet docker; then 2024-11-01T16:45:27.8507284Z  echo "Docker daemon is running..."; 2024-11-01T16:45:27.8507795Z else 2024-11-01T16:45:27.8508359Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-11-01T16:45:27.8509036Z fi 2024-11-01T16:45:27.8514987Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:27.8515633Z env: 2024-11-01T16:45:27.8515902Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:27.8516249Z ##[endgroup] 2024-11-01T16:45:27.8587898Z Docker daemon is running... 2024-11-01T16:45:27.8661805Z ##[group]Run nick-fields/retry@v3.0.0 2024-11-01T16:45:27.8662361Z with: 2024-11-01T16:45:27.8662630Z shell: bash 2024-11-01T16:45:27.8663200Z timeout_minutes: 5 2024-11-01T16:45:27.8663629Z max_attempts: 3 2024-11-01T16:45:27.8663977Z retry_wait_seconds: 30 2024-11-01T16:45:27.8668020Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-11-01T16:45:27.8671988Z polling_interval_seconds: 1 2024-11-01T16:45:27.8672417Z warning_on_retry: true 2024-11-01T16:45:27.8672860Z continue_on_error: false 2024-11-01T16:45:27.8673292Z env: 2024-11-01T16:45:27.8673698Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:27.8674135Z AWS_RETRY_MODE: standard 2024-11-01T16:45:27.8674517Z AWS_MAX_ATTEMPTS: 5 2024-11-01T16:45:27.8674950Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:45:27.8675357Z ##[endgroup] 2024-11-01T16:45:29.2121116Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-11-01T16:45:29.2122129Z Configure a credential helper to remove this warning. See 2024-11-01T16:45:29.2123166Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-11-01T16:45:29.2123792Z 2024-11-01T16:45:29.2123913Z Login Succeeded 2024-11-01T16:45:30.0162078Z Command completed after 1 attempt(s). 2024-11-01T16:45:30.0243901Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-11-01T16:45:30.0244802Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-11-01T16:45:30.0245502Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-11-01T16:45:30.0253526Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:30.0254091Z env: 2024-11-01T16:45:30.0254396Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:30.0254779Z ##[endgroup] 2024-11-01T16:45:30.0355639Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-11-01T16:45:30.0356519Z # ignore expansion of "docker ps -q" since it could be empty 2024-11-01T16:45:30.0357194Z # shellcheck disable=SC2046 2024-11-01T16:45:30.0357716Z docker stop $(docker ps -q) || true 2024-11-01T16:45:30.0358262Z # Prune all of the docker images 2024-11-01T16:45:30.0358770Z docker system prune -af 2024-11-01T16:45:30.0365056Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:30.0365585Z env: 2024-11-01T16:45:30.0365875Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:30.0366241Z ##[endgroup] 2024-11-01T16:45:30.0687021Z "docker stop" requires at least 1 argument. 2024-11-01T16:45:30.0687852Z See 'docker stop --help'. 2024-11-01T16:45:30.0688148Z 2024-11-01T16:45:30.0688420Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-11-01T16:45:30.0688861Z 2024-11-01T16:45:30.0689041Z Stop one or more running containers 2024-11-01T16:45:30.0880275Z Total reclaimed space: 0B 2024-11-01T16:45:30.0932617Z ##[group]Run set +e 2024-11-01T16:45:30.0933008Z set +e 2024-11-01T16:45:30.0933308Z set -x 2024-11-01T16:45:30.0933613Z  2024-11-01T16:45:30.0933956Z PT_DOMAIN=download.pytorch.org 2024-11-01T16:45:30.0934843Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-11-01T16:45:30.0936075Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-11-01T16:45:30.0937256Z # one is returned at random 2024-11-01T16:45:30.0937870Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-11-01T16:45:30.0938438Z  2024-11-01T16:45:30.0938926Z if [ -z "${RESOLVED_IP}" ]; then 2024-11-01T16:45:30.0939638Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-11-01T16:45:30.0940512Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-11-01T16:45:30.0941258Z  2024-11-01T16:45:30.0941765Z  if [ -z "${RESOLVED_IP}" ]; then 2024-11-01T16:45:30.0942502Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-11-01T16:45:30.0943094Z  exit 1 2024-11-01T16:45:30.0943443Z  fi 2024-11-01T16:45:30.0943759Z fi 2024-11-01T16:45:30.0944062Z  2024-11-01T16:45:30.0944445Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-11-01T16:45:30.0945055Z  # Clean up any old records first 2024-11-01T16:45:30.0945643Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-11-01T16:45:30.0946158Z fi 2024-11-01T16:45:30.0946457Z  2024-11-01T16:45:30.0946935Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-11-01T16:45:30.0947571Z cat /etc/hosts 2024-11-01T16:45:30.0954340Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:30.0954890Z env: 2024-11-01T16:45:30.0955181Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:30.0955555Z ##[endgroup] 2024-11-01T16:45:30.0981879Z + PT_DOMAIN=download.pytorch.org 2024-11-01T16:45:30.0987754Z ++ dig -4 +short download.pytorch.org 2024-11-01T16:45:30.0988553Z ++ tail -n1 2024-11-01T16:45:30.1214492Z + RESOLVED_IP=18.160.10.36 2024-11-01T16:45:30.1215184Z + '[' -z 18.160.10.36 ']' 2024-11-01T16:45:30.1215716Z + grep -r download.pytorch.org /etc/hosts 2024-11-01T16:45:30.1228748Z 18.160.10.36 download.pytorch.org 2024-11-01T16:45:30.1230078Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-11-01T16:45:30.2564609Z + echo '18.160.10.36 download.pytorch.org' 2024-11-01T16:45:30.2565200Z + sudo tee -a /etc/hosts 2024-11-01T16:45:30.2992062Z 18.160.10.36 download.pytorch.org 2024-11-01T16:45:30.3008857Z + cat /etc/hosts 2024-11-01T16:45:30.3016498Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-11-01T16:45:30.3028900Z ::1 localhost6 localhost6.localdomain6 2024-11-01T16:45:30.3029561Z 18.160.10.36 download.pytorch.org 2024-11-01T16:45:30.3245494Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-11-01T16:45:30.3246131Z with: 2024-11-01T16:45:30.3247180Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:30.3248377Z docker-build-dir: .ci/docker 2024-11-01T16:45:30.3248761Z working-directory: . 2024-11-01T16:45:30.3249247Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:30.3249809Z force-push: false 2024-11-01T16:45:30.3250109Z env: 2024-11-01T16:45:30.3250378Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:30.3250726Z ##[endgroup] 2024-11-01T16:45:30.3275446Z ##[group]Run set -ex 2024-11-01T16:45:30.3275809Z set -ex 2024-11-01T16:45:30.3276097Z  2024-11-01T16:45:30.3276832Z # If the docker build directory or the build script doesn't exist, the action will 2024-11-01T16:45:30.3278394Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-11-01T16:45:30.3279351Z # job could then download the pre-built image as usual 2024-11-01T16:45:30.3280223Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-11-01T16:45:30.3281021Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:30.3281764Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:30.3282566Z  2024-11-01T16:45:30.3283155Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-11-01T16:45:30.3283891Z  exit 0 2024-11-01T16:45:30.3284207Z else 2024-11-01T16:45:30.3284615Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:30.3285131Z fi 2024-11-01T16:45:30.3285426Z  2024-11-01T16:45:30.3285957Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-11-01T16:45:30.3286983Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-11-01T16:45:30.3287893Z  # use it as it is, but first let's extract the tag 2024-11-01T16:45:30.3288722Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-11-01T16:45:30.3289670Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:30.3290520Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:30.3291098Z else 2024-11-01T16:45:30.3291536Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-11-01T16:45:30.3292223Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:30.3293160Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:30.3293958Z fi 2024-11-01T16:45:30.3300182Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:30.3300685Z env: 2024-11-01T16:45:30.3301046Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:30.3301579Z REPO_NAME: pytorch 2024-11-01T16:45:30.3302799Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:30.3304113Z DOCKER_BUILD_DIR: .ci/docker 2024-11-01T16:45:30.3304695Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:30.3305293Z ##[endgroup] 2024-11-01T16:45:30.3330504Z + [[ ! -d .ci/docker ]] 2024-11-01T16:45:30.3330994Z + [[ ! -f .ci/docker/build.sh ]] 2024-11-01T16:45:30.3331420Z + echo skip=false 2024-11-01T16:45:30.3333399Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-11-01T16:45:30.3337584Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:30.3338848Z ++ awk -F '[:,]' '{print $2}' 2024-11-01T16:45:30.3360505Z + DOCKER_TAG=bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:30.3361440Z + echo docker-tag=bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:30.3363196Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:30.3394919Z ##[group]Run set +e 2024-11-01T16:45:30.3395285Z set +e 2024-11-01T16:45:30.3395581Z set -x 2024-11-01T16:45:30.3395866Z  2024-11-01T16:45:30.3396134Z login() { 2024-11-01T16:45:30.3396807Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-11-01T16:45:30.3397568Z } 2024-11-01T16:45:30.3397835Z  2024-11-01T16:45:30.3398113Z retry () { 2024-11-01T16:45:30.3398505Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-11-01T16:45:30.3398962Z } 2024-11-01T16:45:30.3399214Z  2024-11-01T16:45:30.3399519Z retry login "${DOCKER_REGISTRY}" 2024-11-01T16:45:30.3399938Z  2024-11-01T16:45:30.3400413Z # Check if image already exists, if it does then skip building it 2024-11-01T16:45:30.3401288Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-11-01T16:45:30.3401792Z  exit 0 2024-11-01T16:45:30.3402087Z fi 2024-11-01T16:45:30.3402354Z  2024-11-01T16:45:30.3402868Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-11-01T16:45:30.3403740Z # be empty. The default action would be to continue rebuild the image 2024-11-01T16:45:30.3404515Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-11-01T16:45:30.3405210Z  # if we're on the base branch then use the parent commit 2024-11-01T16:45:30.3405822Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-11-01T16:45:30.3406256Z else 2024-11-01T16:45:30.3406729Z  # otherwise we're on a PR, so use the most recent base commit 2024-11-01T16:45:30.3407446Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-11-01T16:45:30.3407963Z fi 2024-11-01T16:45:30.3408217Z  2024-11-01T16:45:30.3408529Z if [[ -z "${MERGE_BASE}" ]]; then 2024-11-01T16:45:30.3409043Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:30.3409506Z  2024-11-01T16:45:30.3410189Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-11-01T16:45:30.3410993Z  exit 0 2024-11-01T16:45:30.3411274Z fi 2024-11-01T16:45:30.3411539Z  2024-11-01T16:45:30.3411977Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-11-01T16:45:30.3413013Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-11-01T16:45:30.3413866Z  exit 1 2024-11-01T16:45:30.3414156Z fi 2024-11-01T16:45:30.3414407Z  2024-11-01T16:45:30.3414898Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-11-01T16:45:30.3415913Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-11-01T16:45:30.3416834Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-11-01T16:45:30.3417869Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-11-01T16:45:30.3419043Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-11-01T16:45:30.3419721Z fi 2024-11-01T16:45:30.3419976Z  2024-11-01T16:45:30.3420493Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:30.3427288Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:30.3427862Z env: 2024-11-01T16:45:30.3428160Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:30.3428555Z DOCKER_BUILD_DIR: .ci/docker 2024-11-01T16:45:30.3429090Z BASE_REVISION: 2055d3c8dc8b93c52eda6b9c07d716448d9e28c9 2024-11-01T16:45:30.3430482Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:30.3431858Z DOCKER_TAG: bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:30.3432557Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:30.3433153Z ##[endgroup] 2024-11-01T16:45:30.3457678Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:30.3458443Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:30.3460692Z + aws ecr get-login-password --region us-east-1 2024-11-01T16:45:30.3462358Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:30.9317060Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-11-01T16:45:30.9318027Z Configure a credential helper to remove this warning. See 2024-11-01T16:45:30.9319000Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-11-01T16:45:30.9319920Z 2024-11-01T16:45:30.9320047Z Login Succeeded 2024-11-01T16:45:30.9335354Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:31.1504417Z { 2024-11-01T16:45:31.1528069Z + exit 0 2024-11-01T16:45:31.1528573Z "schemaVersion": 2, 2024-11-01T16:45:31.1529380Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-11-01T16:45:31.1530094Z "config": { 2024-11-01T16:45:31.1530633Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-11-01T16:45:31.1531259Z "size": 49259, 2024-11-01T16:45:31.1531931Z "digest": "sha256:f4475432de42c25077a2ffe0f2f56efb42124dedca9504f01daf084f9103394b" 2024-11-01T16:45:31.1532674Z }, 2024-11-01T16:45:31.1532963Z "layers": [ 2024-11-01T16:45:31.1533278Z { 2024-11-01T16:45:31.1533780Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1534418Z "size": 28580681, 2024-11-01T16:45:31.1535206Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-11-01T16:45:31.1536131Z }, 2024-11-01T16:45:31.1536593Z { 2024-11-01T16:45:31.1537487Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1538204Z "size": 7943353, 2024-11-01T16:45:31.1538984Z "digest": "sha256:09d415c238d76b32a7ea4a6e6add9542db9a5641f7f183af70aae185d0709e58" 2024-11-01T16:45:31.1540278Z }, 2024-11-01T16:45:31.1540856Z { 2024-11-01T16:45:31.1541657Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1542804Z "size": 56226389, 2024-11-01T16:45:31.1543711Z "digest": "sha256:9fe6e2e61518cba6844870c03b285737daec35e62baf25ae7744629ed3a7b470" 2024-11-01T16:45:31.1544733Z }, 2024-11-01T16:45:31.1545188Z { 2024-11-01T16:45:31.1545929Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1546902Z "size": 187, 2024-11-01T16:45:31.1547786Z "digest": "sha256:41f16248e682693ff20b3032c1d5e5541cc87c5af898ae2ff9b24d2940e59100" 2024-11-01T16:45:31.1548809Z }, 2024-11-01T16:45:31.1549208Z { 2024-11-01T16:45:31.1549942Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1551111Z "size": 6885, 2024-11-01T16:45:31.1551885Z "digest": "sha256:95d7b781703928cf3c4eece39d800cccb76728c375fedf51ecd83833fb25e458" 2024-11-01T16:45:31.1552614Z }, 2024-11-01T16:45:31.1552894Z { 2024-11-01T16:45:31.1553682Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1554490Z "size": 1383662475, 2024-11-01T16:45:31.1555127Z "digest": "sha256:8f6c9048534734f4c873935293b7296225846ceb31c1a158400a67ea170dde7f" 2024-11-01T16:45:31.1555809Z }, 2024-11-01T16:45:31.1556091Z { 2024-11-01T16:45:31.1556593Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1557247Z "size": 62282, 2024-11-01T16:45:31.1557863Z "digest": "sha256:ab17245097e491b9368790714f9d90ed447bf0973bd677cfe6f2456d62b72a13" 2024-11-01T16:45:31.1558559Z }, 2024-11-01T16:45:31.1558802Z { 2024-11-01T16:45:31.1559296Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1559948Z "size": 1684, 2024-11-01T16:45:31.1560597Z "digest": "sha256:dfecd7e9912b76ed460b8edd5a85f1943666e38a973ab5458177cf2c7c3110e3" 2024-11-01T16:45:31.1561331Z }, 2024-11-01T16:45:31.1561584Z { 2024-11-01T16:45:31.1562087Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1562738Z "size": 1522, 2024-11-01T16:45:31.1563349Z "digest": "sha256:464a8f74544589bf7b57f9a4cadcb6681e5ed00758f6c35025e691df4e88e890" 2024-11-01T16:45:31.1564049Z }, 2024-11-01T16:45:31.1564296Z { 2024-11-01T16:45:31.1564800Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1565453Z "size": 2445884909, 2024-11-01T16:45:31.1566100Z "digest": "sha256:d67c111fa58886d8904335b5122039816c8e51af5844cb6fe4e14154d9a7c016" 2024-11-01T16:45:31.1566952Z }, 2024-11-01T16:45:31.1567208Z { 2024-11-01T16:45:31.1567705Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1568345Z "size": 85882, 2024-11-01T16:45:31.1568983Z "digest": "sha256:0723abde19c09431db80b5d0f0e003541f6232c2da5c4f35c635c861875937ae" 2024-11-01T16:45:31.1569705Z }, 2024-11-01T16:45:31.1569959Z { 2024-11-01T16:45:31.1570457Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1571102Z "size": 1822, 2024-11-01T16:45:31.1571904Z "digest": "sha256:353d86cedcdc414207029061854fbc806c438ad1f5c00ac8aebf6daa9c2e9e9f" 2024-11-01T16:45:31.1572641Z }, 2024-11-01T16:45:31.1572913Z { 2024-11-01T16:45:31.1573397Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1574051Z "size": 655141106, 2024-11-01T16:45:31.1574704Z "digest": "sha256:0299f4e4d5284c3c947696ef5f6ca6f7b5fab7881f4c4a6096399ca3e64f2c44" 2024-11-01T16:45:31.1575439Z }, 2024-11-01T16:45:31.1575717Z { 2024-11-01T16:45:31.1576199Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1577112Z "size": 703, 2024-11-01T16:45:31.1577959Z "digest": "sha256:86c353c3bc581a31d9abdc8486224b29531a0b199d09e0ea8c5dadce351325b9" 2024-11-01T16:45:31.1578848Z }, 2024-11-01T16:45:31.1579133Z { 2024-11-01T16:45:31.1579630Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1580297Z "size": 1259, 2024-11-01T16:45:31.1581047Z "digest": "sha256:4f10af237965d5a943f19bb5409eef94fae79f531a7e7137ea8a8815aff476b1" 2024-11-01T16:45:31.1581799Z }, 2024-11-01T16:45:31.1582084Z { 2024-11-01T16:45:31.1582602Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1583255Z "size": 484, 2024-11-01T16:45:31.1583919Z "digest": "sha256:b7b00fcb07ca8c1ce9232c36e9c4bcd528d3167479c20a9e0b05a4c318417a21" 2024-11-01T16:45:31.1584680Z }, 2024-11-01T16:45:31.1584967Z { 2024-11-01T16:45:31.1585480Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1586136Z "size": 91712745, 2024-11-01T16:45:31.1586849Z "digest": "sha256:db97392a2fce6d95ddfdcafbdee9fc4102fa976535ce9687d8f32dde3e188f61" 2024-11-01T16:45:31.1587630Z }, 2024-11-01T16:45:31.1587911Z { 2024-11-01T16:45:31.1588422Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1589069Z "size": 3678, 2024-11-01T16:45:31.1589912Z "digest": "sha256:27e2d54f402a8b72082e53cf941bbc159517167ef364e2f9bee99de378effd71" 2024-11-01T16:45:31.1590662Z }, 2024-11-01T16:45:31.1590948Z { 2024-11-01T16:45:31.1591466Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1592113Z "size": 1801, 2024-11-01T16:45:31.1592866Z "digest": "sha256:8269c56845c337b349db7a6d46a81a462af0d1183f2761ab3489909f0663638c" 2024-11-01T16:45:31.1593687Z }, 2024-11-01T16:45:31.1593956Z { 2024-11-01T16:45:31.1594450Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1595081Z "size": 702, 2024-11-01T16:45:31.1595671Z "digest": "sha256:6c7e3cfd6320c1a2e3323c5b6ed269549b5102449ba19199811783455948b5cd" 2024-11-01T16:45:31.1596366Z }, 2024-11-01T16:45:31.1596634Z { 2024-11-01T16:45:31.1597119Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1597756Z "size": 2921113290, 2024-11-01T16:45:31.1598378Z "digest": "sha256:82c15385d36d076d32b291ce1d513b5a7a94f02b56c06254edfca098f5e7ecba" 2024-11-01T16:45:31.1599083Z }, 2024-11-01T16:45:31.1599354Z { 2024-11-01T16:45:31.1599845Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1600474Z "size": 380, 2024-11-01T16:45:31.1601061Z "digest": "sha256:5ffae31f204af230644f54b5d1a9178b0313f3591541a866397342ba4177e782" 2024-11-01T16:45:31.1601760Z }, 2024-11-01T16:45:31.1602028Z { 2024-11-01T16:45:31.1602516Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1603269Z "size": 12890, 2024-11-01T16:45:31.1603858Z "digest": "sha256:879b9cee6016a801826c1899830336c41370e72ae7b77305190ac6d2016bc8df" 2024-11-01T16:45:31.1604545Z }, 2024-11-01T16:45:31.1604818Z { 2024-11-01T16:45:31.1605308Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1605938Z "size": 862, 2024-11-01T16:45:31.1606539Z "digest": "sha256:ead67a6283927375ef4a93495b43516a6a077bce28c304063c69a1c09c40cf72" 2024-11-01T16:45:31.1607223Z }, 2024-11-01T16:45:31.1607492Z { 2024-11-01T16:45:31.1607983Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1608617Z "size": 106, 2024-11-01T16:45:31.1609245Z "digest": "sha256:2f25291ccc900ebaceb5c80edef126cf7641178b30cbc9ae518fad289b0beef0" 2024-11-01T16:45:31.1609952Z }, 2024-11-01T16:45:31.1610222Z { 2024-11-01T16:45:31.1610703Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1611336Z "size": 505, 2024-11-01T16:45:31.1612133Z "digest": "sha256:5cdf685c64c3bd498aa6eb1dfce3a7e059b6f631b0c53f0bed7a78dc821391ef" 2024-11-01T16:45:31.1612843Z }, 2024-11-01T16:45:31.1613109Z { 2024-11-01T16:45:31.1613591Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1614227Z "size": 121477626, 2024-11-01T16:45:31.1614909Z "digest": "sha256:4eab1e4b631667fa7e17e7f2631d88a23b6869b53b25f0bce68b294ff2a25753" 2024-11-01T16:45:31.1615616Z }, 2024-11-01T16:45:31.1615864Z { 2024-11-01T16:45:31.1616352Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1616983Z "size": 109, 2024-11-01T16:45:31.1617587Z "digest": "sha256:7caa1481847d89677e0ca3d3e4cfb637968c2e5fa17d24a78d061617629477a0" 2024-11-01T16:45:31.1618295Z }, 2024-11-01T16:45:31.1618540Z { 2024-11-01T16:45:31.1619024Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1619652Z "size": 489, 2024-11-01T16:45:31.1620290Z "digest": "sha256:dafa8b6784dd9f01c5b1aefb90f8c8e21681ad0e63c7f6a4bb43db1e7e1eb0e2" 2024-11-01T16:45:31.1621102Z }, 2024-11-01T16:45:31.1621523Z { 2024-11-01T16:45:31.1622039Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1622710Z "size": 296, 2024-11-01T16:45:31.1623385Z "digest": "sha256:dfccec371c6fccbb4240dd4ebb403f011e78b159e5dce84145a1ea58da02937e" 2024-11-01T16:45:31.1624161Z }, 2024-11-01T16:45:31.1624425Z { 2024-11-01T16:45:31.1625051Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1625719Z "size": 103, 2024-11-01T16:45:31.1626379Z "digest": "sha256:e41e511e406bc1694c2e5da2a5ee58d9450141ae66b41c3e9ca24989dacc957c" 2024-11-01T16:45:31.1627128Z }, 2024-11-01T16:45:31.1627389Z { 2024-11-01T16:45:31.1627905Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1628575Z "size": 1473, 2024-11-01T16:45:31.1629210Z "digest": "sha256:7cd18613846984c3f25803460d6ec45d1d41db66bd4f9527b365889495421604" 2024-11-01T16:45:31.1629946Z }, 2024-11-01T16:45:31.1630227Z { 2024-11-01T16:45:31.1630724Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1631395Z "size": 426740495, 2024-11-01T16:45:31.1632082Z "digest": "sha256:38aadcbec80791616741dde49d32a2b56f6d3079d6cf8e823b4220a02eabcf1c" 2024-11-01T16:45:31.1632845Z }, 2024-11-01T16:45:31.1633128Z { 2024-11-01T16:45:31.1633733Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1634458Z "size": 164, 2024-11-01T16:45:31.1635047Z "digest": "sha256:f0b80c9d9a164550e271aa228fb110aefe145520e814c2fa44aa345ecb40f815" 2024-11-01T16:45:31.1635715Z }, 2024-11-01T16:45:31.1635967Z { 2024-11-01T16:45:31.1636408Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1637008Z "size": 565, 2024-11-01T16:45:31.1637592Z "digest": "sha256:bad6bd964ba483454e0e3f1a2b99160c38b3900658032c6597e8afb9a4c9c9c8" 2024-11-01T16:45:31.1638351Z }, 2024-11-01T16:45:31.1638605Z { 2024-11-01T16:45:31.1639048Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1639650Z "size": 35806344, 2024-11-01T16:45:31.1640241Z "digest": "sha256:a348147f958bd269f1391a5802dc1c9d7c6414005de995ca5e2d1d8b4867d087" 2024-11-01T16:45:31.1640906Z }, 2024-11-01T16:45:31.1641159Z { 2024-11-01T16:45:31.1641617Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1642190Z "size": 104, 2024-11-01T16:45:31.1642784Z "digest": "sha256:a3dfd1341f5cb909b01d7c3816d56e35af65a2728bffc76a810ca40cfc3a6dc3" 2024-11-01T16:45:31.1643461Z }, 2024-11-01T16:45:31.1643717Z { 2024-11-01T16:45:31.1644175Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1644757Z "size": 425, 2024-11-01T16:45:31.1645333Z "digest": "sha256:c6d6b49273d1097ce850bdcaa1a0f5a19ad7a7897388642b190dcbdfb906f7f1" 2024-11-01T16:45:31.1646012Z }, 2024-11-01T16:45:31.1646265Z { 2024-11-01T16:45:31.1646724Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1647305Z "size": 20262172, 2024-11-01T16:45:31.1647896Z "digest": "sha256:9dec96f6a1c93d47374302f17840755ee42274e1a7c0aa05659042c2e3b3c5fb" 2024-11-01T16:45:31.1648556Z }, 2024-11-01T16:45:31.1648809Z { 2024-11-01T16:45:31.1649270Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1649851Z "size": 440, 2024-11-01T16:45:31.1650428Z "digest": "sha256:98a2657c4e6010d698d905ba80b545f25d8103e49e3da9e17c15359af100b9ca" 2024-11-01T16:45:31.1651087Z }, 2024-11-01T16:45:31.1651397Z { 2024-11-01T16:45:31.1651857Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1652456Z "size": 702, 2024-11-01T16:45:31.1653008Z "digest": "sha256:6c7e3cfd6320c1a2e3323c5b6ed269549b5102449ba19199811783455948b5cd" 2024-11-01T16:45:31.1653668Z }, 2024-11-01T16:45:31.1653928Z { 2024-11-01T16:45:31.1654385Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1654976Z "size": 141, 2024-11-01T16:45:31.1655540Z "digest": "sha256:955cea5b7f364a6e63ba32e894866f7a53b915e04275cc94a39cddd565d9649f" 2024-11-01T16:45:31.1656208Z }, 2024-11-01T16:45:31.1656461Z { 2024-11-01T16:45:31.1656921Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1657518Z "size": 135, 2024-11-01T16:45:31.1658172Z "digest": "sha256:f38b3a72f6838f3a8895fa755edba3b26894109b9b4cf953a046145ffead84c7" 2024-11-01T16:45:31.1658842Z }, 2024-11-01T16:45:31.1659096Z { 2024-11-01T16:45:31.1659558Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1660151Z "size": 32, 2024-11-01T16:45:31.1660719Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:45:31.1661668Z }, 2024-11-01T16:45:31.1661949Z { 2024-11-01T16:45:31.1662465Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1663143Z "size": 195, 2024-11-01T16:45:31.1663804Z "digest": "sha256:c5cb1a39bf9af5ffdd89e40e62f8019aae4b2dccfd34bc5972e4ab5ae8300688" 2024-11-01T16:45:31.1664578Z }, 2024-11-01T16:45:31.1664858Z { 2024-11-01T16:45:31.1665371Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1666037Z "size": 565, 2024-11-01T16:45:31.1666693Z "digest": "sha256:6765d3ee658a82d6af47206ebbed1dd650339da0dacbd454d6c3dbb8f635a9ab" 2024-11-01T16:45:31.1667430Z }, 2024-11-01T16:45:31.1667715Z { 2024-11-01T16:45:31.1668227Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1668895Z "size": 43163402, 2024-11-01T16:45:31.1669554Z "digest": "sha256:cf81cf49414b46152b641c670668b110a04ac48412f9c4dc6613311cf3a0942d" 2024-11-01T16:45:31.1670274Z }, 2024-11-01T16:45:31.1670554Z { 2024-11-01T16:45:31.1671068Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1671852Z "size": 106, 2024-11-01T16:45:31.1672495Z "digest": "sha256:303676327e274ec659dcce26b1e2613e6952e538ab3568f619a5e1a8b603ec89" 2024-11-01T16:45:31.1673213Z }, 2024-11-01T16:45:31.1673493Z { 2024-11-01T16:45:31.1674104Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1674954Z "size": 1401, 2024-11-01T16:45:31.1675600Z "digest": "sha256:09aac48ef4835fec34e2ae2ba94b6081c53d8a2d4fd257e6a0b73349800b9864" 2024-11-01T16:45:31.1676343Z }, 2024-11-01T16:45:31.1676777Z { 2024-11-01T16:45:31.1677474Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1678151Z "size": 702, 2024-11-01T16:45:31.1678797Z "digest": "sha256:6c7e3cfd6320c1a2e3323c5b6ed269549b5102449ba19199811783455948b5cd" 2024-11-01T16:45:31.1679537Z }, 2024-11-01T16:45:31.1679803Z { 2024-11-01T16:45:31.1680319Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1680993Z "size": 136, 2024-11-01T16:45:31.1681668Z "digest": "sha256:ce24c94fd6add217e412ea0dd91c6e0d8cee5c7f27d8700aa6fe56f32a46ef85" 2024-11-01T16:45:31.1682446Z }, 2024-11-01T16:45:31.1682710Z { 2024-11-01T16:45:31.1683226Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1683898Z "size": 120, 2024-11-01T16:45:31.1684544Z "digest": "sha256:8fd476fea7ee8bf52b382a9037418721f15c17b35692cc1f728e52380dfa773c" 2024-11-01T16:45:31.1685289Z }, 2024-11-01T16:45:31.1685550Z { 2024-11-01T16:45:31.1686068Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1686740Z "size": 4310238020, 2024-11-01T16:45:31.1687432Z "digest": "sha256:4cf3da11584940e52b04ab8799eb5c048bd93de4b8ae9a7a8cde0f70aad2326e" 2024-11-01T16:45:31.1688191Z }, 2024-11-01T16:45:31.1688475Z { 2024-11-01T16:45:31.1688970Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1689834Z "size": 176, 2024-11-01T16:45:31.1690415Z "digest": "sha256:1de6b248716cf53af496379d6fd310a4a060fd7d801c9398c3cb180e9b1358a6" 2024-11-01T16:45:31.1691084Z }, 2024-11-01T16:45:31.1691343Z { 2024-11-01T16:45:31.1691786Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1692391Z "size": 907, 2024-11-01T16:45:31.1692989Z "digest": "sha256:f4eec39b74d5ab5024e1ee6c0a12e5c875dff461e7f65e95cab5adf5b7b0e558" 2024-11-01T16:45:31.1693672Z }, 2024-11-01T16:45:31.1693928Z { 2024-11-01T16:45:31.1694379Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1695126Z "size": 702, 2024-11-01T16:45:31.1695705Z "digest": "sha256:6c7e3cfd6320c1a2e3323c5b6ed269549b5102449ba19199811783455948b5cd" 2024-11-01T16:45:31.1696365Z }, 2024-11-01T16:45:31.1696619Z { 2024-11-01T16:45:31.1697066Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1697664Z "size": 135, 2024-11-01T16:45:31.1698258Z "digest": "sha256:bc86c61c49d69deaa90f9b4c601c930d1ae073fcac2b1e505aa15210056bfc49" 2024-11-01T16:45:31.1698947Z }, 2024-11-01T16:45:31.1699202Z { 2024-11-01T16:45:31.1699648Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1700253Z "size": 32, 2024-11-01T16:45:31.1700842Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:45:31.1701797Z }, 2024-11-01T16:45:31.1702080Z { 2024-11-01T16:45:31.1702659Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1703311Z "size": 158, 2024-11-01T16:45:31.1703973Z "digest": "sha256:4790041b71a9d94be048d9d17de7eaf5ccc4fc6824ba1988a6c0e8d5d8bb39d8" 2024-11-01T16:45:31.1704723Z }, 2024-11-01T16:45:31.1705008Z { 2024-11-01T16:45:31.1705528Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1706183Z "size": 2047, 2024-11-01T16:45:31.1706824Z "digest": "sha256:9113890e17732f4e737f88fd824bf1d95f4990250788e87d7eafca3ca8d3c145" 2024-11-01T16:45:31.1707675Z }, 2024-11-01T16:45:31.1707957Z { 2024-11-01T16:45:31.1708475Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1709125Z "size": 7529790, 2024-11-01T16:45:31.1709990Z "digest": "sha256:ad6d4c88cacf37e10c86d50f66a2ba3cc404356243a1399b9986c73beeee06c0" 2024-11-01T16:45:31.1710753Z }, 2024-11-01T16:45:31.1711032Z { 2024-11-01T16:45:31.1711545Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1712197Z "size": 165, 2024-11-01T16:45:31.1712851Z "digest": "sha256:760e1aa1b5e32af659fb52bd61597d28609b31878840d1d752e99f3cfd8ce3ec" 2024-11-01T16:45:31.1713598Z }, 2024-11-01T16:45:31.1713986Z { 2024-11-01T16:45:31.1714445Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1715042Z "size": 7943, 2024-11-01T16:45:31.1715625Z "digest": "sha256:3c4483bbdce846f42d57f201dfe75cda1f5046ec2ac3bcade9042e5adb8c5079" 2024-11-01T16:45:31.1716310Z }, 2024-11-01T16:45:31.1716569Z { 2024-11-01T16:45:31.1717026Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1717624Z "size": 8071, 2024-11-01T16:45:31.1718193Z "digest": "sha256:8764bbbef59040f5d6a8293cfe6e35d1e0178ad129615f69281bcb3b4788b0f0" 2024-11-01T16:45:31.1718864Z }, 2024-11-01T16:45:31.1719117Z { 2024-11-01T16:45:31.1719573Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1720166Z "size": 301, 2024-11-01T16:45:31.1720722Z "digest": "sha256:71826d5dd434366bb74c562c6725263f57f398f696093b870a0e1aa0af3834fe" 2024-11-01T16:45:31.1721375Z }, 2024-11-01T16:45:31.1721628Z { 2024-11-01T16:45:31.1722087Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1722687Z "size": 7630059, 2024-11-01T16:45:31.1723267Z "digest": "sha256:7b4c7f0a396ed0af23fe3f9a24cdd2783c8558177cb3702476b8529d23144f4a" 2024-11-01T16:45:31.1723944Z }, 2024-11-01T16:45:31.1724197Z { 2024-11-01T16:45:31.1724662Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1725259Z "size": 108, 2024-11-01T16:45:31.1725839Z "digest": "sha256:1200fc258863ca7884a699c7e393d80eb77d6dc143b3d42cc2fbba1cb84826c6" 2024-11-01T16:45:31.1726489Z }, 2024-11-01T16:45:31.1726743Z { 2024-11-01T16:45:31.1727202Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1727799Z "size": 54145652, 2024-11-01T16:45:31.1728396Z "digest": "sha256:1b6c965722e8b660c32a130dec4fcc09d3f8e1924d3351990c2317af7dae6911" 2024-11-01T16:45:31.1729130Z }, 2024-11-01T16:45:31.1729386Z { 2024-11-01T16:45:31.1729843Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1730438Z "size": 472, 2024-11-01T16:45:31.1731030Z "digest": "sha256:6cf78d9b64805ade5c1afb24be9ca48e9ffa016e6091322be4ede273692a7d82" 2024-11-01T16:45:31.1731696Z }, 2024-11-01T16:45:31.1731952Z { 2024-11-01T16:45:31.1732413Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1733022Z "size": 1373229322, 2024-11-01T16:45:31.1733651Z "digest": "sha256:accad7e3a239e2b010d87ef44b7db0fc6939535831f7ac907de46fd089b84fcb" 2024-11-01T16:45:31.1734329Z }, 2024-11-01T16:45:31.1734567Z { 2024-11-01T16:45:31.1735022Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1735619Z "size": 103, 2024-11-01T16:45:31.1736207Z "digest": "sha256:b0b6f60fa25319ceba8a48c89bae7353160dafff3324f625d2c62b808da4d98e" 2024-11-01T16:45:31.1736885Z }, 2024-11-01T16:45:31.1737131Z { 2024-11-01T16:45:31.1737593Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1738193Z "size": 569, 2024-11-01T16:45:31.1738789Z "digest": "sha256:acbcfa904fd51dbd77c9918823c59fc72b24bc49e9d3e843530877c2120b8db0" 2024-11-01T16:45:31.1739465Z }, 2024-11-01T16:45:31.1739704Z { 2024-11-01T16:45:31.1740167Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1740843Z "size": 36532055, 2024-11-01T16:45:31.1741717Z "digest": "sha256:024bd3c6bf01b2391930e2c9aa2648a121639bc8eadf45eaf325f74eb1a29822" 2024-11-01T16:45:31.1742470Z }, 2024-11-01T16:45:31.1742734Z { 2024-11-01T16:45:31.1743246Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1743917Z "size": 111, 2024-11-01T16:45:31.1744573Z "digest": "sha256:9cc11594aa2a6ba9e369c7fdf6f7a27181e53af46336a4e825fb3fa888c2cdbc" 2024-11-01T16:45:31.1745336Z }, 2024-11-01T16:45:31.1745597Z { 2024-11-01T16:45:31.1746113Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1746781Z "size": 531, 2024-11-01T16:45:31.1747425Z "digest": "sha256:01fd6f2e9fa39599a7a515a787a896943a1e32b00f54701d01633e1ab2d42449" 2024-11-01T16:45:31.1748157Z }, 2024-11-01T16:45:31.1748439Z { 2024-11-01T16:45:31.1748929Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1749596Z "size": 25817, 2024-11-01T16:45:31.1750260Z "digest": "sha256:d06fa63dc0b7603ba664281c70a33befe32c8b1d0fa355cf9a520049165b88ea" 2024-11-01T16:45:31.1751012Z }, 2024-11-01T16:45:31.1751289Z { 2024-11-01T16:45:31.1751783Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1752446Z "size": 106, 2024-11-01T16:45:31.1753090Z "digest": "sha256:72612b798350e7abf0b29d04df95850b30c9b7cb9b0b15787b9090a3b18ee673" 2024-11-01T16:45:31.1753939Z }, 2024-11-01T16:45:31.1754190Z { 2024-11-01T16:45:31.1754635Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1755231Z "size": 32, 2024-11-01T16:45:31.1755810Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:45:31.1756484Z }, 2024-11-01T16:45:31.1756740Z { 2024-11-01T16:45:31.1757178Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1757772Z "size": 32, 2024-11-01T16:45:31.1758350Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:45:31.1759029Z }, 2024-11-01T16:45:31.1759283Z { 2024-11-01T16:45:31.1759725Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1760318Z "size": 32, 2024-11-01T16:45:31.1760903Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:45:31.1761572Z }, 2024-11-01T16:45:31.1761826Z { 2024-11-01T16:45:31.1762285Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:31.1762952Z "size": 32, 2024-11-01T16:45:31.1763539Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:45:31.1764212Z } 2024-11-01T16:45:31.1764466Z ] 2024-11-01T16:45:31.1764714Z } 2024-11-01T16:45:31.1898424Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-11-01T16:45:31.1898918Z tag=${ECR_DOCKER_IMAGE##*/} 2024-11-01T16:45:31.1899488Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-11-01T16:45:31.1906526Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:31.1907113Z env: 2024-11-01T16:45:31.1907416Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:31.1908679Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:31.1910140Z ##[endgroup] 2024-11-01T16:45:31.1946906Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9-bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:31.2006176Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-11-01T16:45:31.2006762Z with: 2024-11-01T16:45:31.2007783Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:31.2009089Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:31.2009765Z env: 2024-11-01T16:45:31.2010051Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:31.2010409Z ##[endgroup] 2024-11-01T16:45:31.2031456Z ##[group]Run set -x 2024-11-01T16:45:31.2031852Z set -x 2024-11-01T16:45:31.2032196Z set +e 2024-11-01T16:45:31.2032523Z  2024-11-01T16:45:31.2032832Z login() { 2024-11-01T16:45:31.2033614Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-11-01T16:45:31.2034492Z } 2024-11-01T16:45:31.2034769Z  2024-11-01T16:45:31.2035089Z retry () { 2024-11-01T16:45:31.2035487Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-11-01T16:45:31.2035940Z } 2024-11-01T16:45:31.2036212Z  2024-11-01T16:45:31.2036525Z retry login "${DOCKER_REGISTRY}" 2024-11-01T16:45:31.2036960Z  2024-11-01T16:45:31.2037234Z set -e 2024-11-01T16:45:31.2037706Z # ignore output since only exit code is used for conditional 2024-11-01T16:45:31.2038455Z # only pull docker image if it's not available locally 2024-11-01T16:45:31.2039267Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-11-01T16:45:31.2040004Z  retry docker pull "${DOCKER_IMAGE}" 2024-11-01T16:45:31.2040460Z fi 2024-11-01T16:45:31.2046032Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:31.2046555Z env: 2024-11-01T16:45:31.2046821Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:31.2047926Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:31.2049206Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:31.2049749Z ##[endgroup] 2024-11-01T16:45:31.2073985Z + set +e 2024-11-01T16:45:31.2074918Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:31.2075698Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:31.2078873Z + aws ecr get-login-password --region us-east-1 2024-11-01T16:45:31.2080139Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:31.7914753Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-11-01T16:45:31.7917043Z Configure a credential helper to remove this warning. See 2024-11-01T16:45:31.7918034Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-11-01T16:45:31.7918656Z 2024-11-01T16:45:31.7918792Z Login Succeeded 2024-11-01T16:45:31.7932582Z + set -e 2024-11-01T16:45:31.7934017Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:31.8080056Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:31.8082536Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:32.0410857Z bdd298b12da59246147f016e0693ffd722419941: Pulling from pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9 2024-11-01T16:45:32.0412059Z 7a2c55901189: Pulling fs layer 2024-11-01T16:45:32.0412518Z 09d415c238d7: Pulling fs layer 2024-11-01T16:45:32.0412955Z 9fe6e2e61518: Pulling fs layer 2024-11-01T16:45:32.0413383Z 41f16248e682: Pulling fs layer 2024-11-01T16:45:32.0414184Z 95d7b7817039: Pulling fs layer 2024-11-01T16:45:32.0414671Z 8f6c90485347: Pulling fs layer 2024-11-01T16:45:32.0415091Z ab17245097e4: Pulling fs layer 2024-11-01T16:45:32.0415506Z 41f16248e682: Waiting 2024-11-01T16:45:32.0415883Z dfecd7e9912b: Pulling fs layer 2024-11-01T16:45:32.0416295Z 464a8f745445: Pulling fs layer 2024-11-01T16:45:32.0416701Z 95d7b7817039: Waiting 2024-11-01T16:45:32.0417048Z 8f6c90485347: Waiting 2024-11-01T16:45:32.0417560Z d67c111fa588: Pulling fs layer 2024-11-01T16:45:32.0417990Z 0723abde19c0: Pulling fs layer 2024-11-01T16:45:32.0418402Z 353d86cedcdc: Pulling fs layer 2024-11-01T16:45:32.0418830Z 0299f4e4d528: Pulling fs layer 2024-11-01T16:45:32.0419255Z 86c353c3bc58: Pulling fs layer 2024-11-01T16:45:32.0419677Z 4f10af237965: Pulling fs layer 2024-11-01T16:45:32.0420101Z b7b00fcb07ca: Pulling fs layer 2024-11-01T16:45:32.0420513Z db97392a2fce: Pulling fs layer 2024-11-01T16:45:32.0421050Z 27e2d54f402a: Pulling fs layer 2024-11-01T16:45:32.0421520Z 8269c56845c3: Pulling fs layer 2024-11-01T16:45:32.0421945Z 6c7e3cfd6320: Pulling fs layer 2024-11-01T16:45:32.0422375Z 82c15385d36d: Pulling fs layer 2024-11-01T16:45:32.0422782Z ab17245097e4: Waiting 2024-11-01T16:45:32.0423120Z d67c111fa588: Waiting 2024-11-01T16:45:32.0423490Z 5ffae31f204a: Pulling fs layer 2024-11-01T16:45:32.0423903Z 0723abde19c0: Waiting 2024-11-01T16:45:32.0424274Z 879b9cee6016: Pulling fs layer 2024-11-01T16:45:32.0424699Z ead67a628392: Pulling fs layer 2024-11-01T16:45:32.0425102Z dfecd7e9912b: Waiting 2024-11-01T16:45:32.0425476Z 2f25291ccc90: Pulling fs layer 2024-11-01T16:45:32.0425887Z b7b00fcb07ca: Waiting 2024-11-01T16:45:32.0426263Z 5cdf685c64c3: Pulling fs layer 2024-11-01T16:45:32.0426669Z 464a8f745445: Waiting 2024-11-01T16:45:32.0427023Z 4eab1e4b6316: Pulling fs layer 2024-11-01T16:45:32.0427434Z db97392a2fce: Waiting 2024-11-01T16:45:32.0427805Z 7caa1481847d: Pulling fs layer 2024-11-01T16:45:32.0428232Z dafa8b6784dd: Pulling fs layer 2024-11-01T16:45:32.0428640Z 0299f4e4d528: Waiting 2024-11-01T16:45:32.0428984Z 6c7e3cfd6320: Waiting 2024-11-01T16:45:32.0429340Z 4f10af237965: Waiting 2024-11-01T16:45:32.0429711Z dfccec371c6f: Pulling fs layer 2024-11-01T16:45:32.0430138Z e41e511e406b: Pulling fs layer 2024-11-01T16:45:32.0430543Z 5ffae31f204a: Waiting 2024-11-01T16:45:32.0430899Z 7cd186138469: Pulling fs layer 2024-11-01T16:45:32.0431328Z 38aadcbec807: Pulling fs layer 2024-11-01T16:45:32.0431757Z f0b80c9d9a16: Pulling fs layer 2024-11-01T16:45:32.0432217Z 86c353c3bc58: Waiting 2024-11-01T16:45:32.0432589Z bad6bd964ba4: Pulling fs layer 2024-11-01T16:45:32.0433134Z a348147f958b: Pulling fs layer 2024-11-01T16:45:32.0433513Z 8269c56845c3: Waiting 2024-11-01T16:45:32.0433876Z a3dfd1341f5c: Pulling fs layer 2024-11-01T16:45:32.0434289Z c6d6b49273d1: Pulling fs layer 2024-11-01T16:45:32.0434684Z 353d86cedcdc: Waiting 2024-11-01T16:45:32.0435028Z 7cd186138469: Waiting 2024-11-01T16:45:32.0435388Z 9dec96f6a1c9: Pulling fs layer 2024-11-01T16:45:32.0435773Z 38aadcbec807: Waiting 2024-11-01T16:45:32.0436142Z 98a2657c4e60: Pulling fs layer 2024-11-01T16:45:32.0436552Z 955cea5b7f36: Pulling fs layer 2024-11-01T16:45:32.0436945Z 7caa1481847d: Waiting 2024-11-01T16:45:32.0437303Z f38b3a72f683: Pulling fs layer 2024-11-01T16:45:32.0437687Z 4eab1e4b6316: Waiting 2024-11-01T16:45:32.0438050Z 4f4fb700ef54: Pulling fs layer 2024-11-01T16:45:32.0438449Z f0b80c9d9a16: Waiting 2024-11-01T16:45:32.0438793Z ead67a628392: Waiting 2024-11-01T16:45:32.0439146Z a3dfd1341f5c: Waiting 2024-11-01T16:45:32.0439494Z c5cb1a39bf9a: Pulling fs layer 2024-11-01T16:45:32.0439896Z 2f25291ccc90: Waiting 2024-11-01T16:45:32.0440241Z 5cdf685c64c3: Waiting 2024-11-01T16:45:32.0440584Z 955cea5b7f36: Waiting 2024-11-01T16:45:32.0440945Z 6765d3ee658a: Pulling fs layer 2024-11-01T16:45:32.0441346Z cf81cf49414b: Pulling fs layer 2024-11-01T16:45:32.0441747Z bad6bd964ba4: Waiting 2024-11-01T16:45:32.0442107Z 303676327e27: Pulling fs layer 2024-11-01T16:45:32.0442514Z 09aac48ef483: Pulling fs layer 2024-11-01T16:45:32.0443057Z ce24c94fd6ad: Pulling fs layer 2024-11-01T16:45:32.0443453Z dafa8b6784dd: Waiting 2024-11-01T16:45:32.0443805Z c6d6b49273d1: Waiting 2024-11-01T16:45:32.0444154Z dfccec371c6f: Waiting 2024-11-01T16:45:32.0444507Z 9dec96f6a1c9: Waiting 2024-11-01T16:45:32.0444849Z e41e511e406b: Waiting 2024-11-01T16:45:32.0445196Z 8fd476fea7ee: Pulling fs layer 2024-11-01T16:45:32.0445591Z 303676327e27: Waiting 2024-11-01T16:45:32.0445925Z 09aac48ef483: Waiting 2024-11-01T16:45:32.0446361Z 4cf3da115849: Pulling fs layer 2024-11-01T16:45:32.0446758Z f38b3a72f683: Waiting 2024-11-01T16:45:32.0447104Z 1de6b248716c: Pulling fs layer 2024-11-01T16:45:32.0447504Z 98a2657c4e60: Waiting 2024-11-01T16:45:32.0447864Z f4eec39b74d5: Pulling fs layer 2024-11-01T16:45:32.0448283Z bc86c61c49d6: Pulling fs layer 2024-11-01T16:45:32.0448679Z 6765d3ee658a: Waiting 2024-11-01T16:45:32.0449006Z 4cf3da115849: Waiting 2024-11-01T16:45:32.0449359Z 4790041b71a9: Pulling fs layer 2024-11-01T16:45:32.0449765Z 9113890e1773: Pulling fs layer 2024-11-01T16:45:32.0450167Z 4f4fb700ef54: Waiting 2024-11-01T16:45:32.0450530Z ad6d4c88cacf: Pulling fs layer 2024-11-01T16:45:32.0450936Z 760e1aa1b5e3: Pulling fs layer 2024-11-01T16:45:32.0451339Z bc86c61c49d6: Waiting 2024-11-01T16:45:32.0451730Z 3c4483bbdce8: Pulling fs layer 2024-11-01T16:45:32.0452130Z 760e1aa1b5e3: Waiting 2024-11-01T16:45:32.0452518Z 8764bbbef590: Pulling fs layer 2024-11-01T16:45:32.0452972Z 3c4483bbdce8: Waiting 2024-11-01T16:45:32.0453348Z ad6d4c88cacf: Waiting 2024-11-01T16:45:32.0453710Z 71826d5dd434: Pulling fs layer 2024-11-01T16:45:32.0454134Z 7b4c7f0a396e: Pulling fs layer 2024-11-01T16:45:32.0454554Z 1200fc258863: Pulling fs layer 2024-11-01T16:45:32.0454949Z 1de6b248716c: Waiting 2024-11-01T16:45:32.0455289Z 71826d5dd434: Waiting 2024-11-01T16:45:32.0455616Z 7b4c7f0a396e: Waiting 2024-11-01T16:45:32.0455974Z 1b6c965722e8: Pulling fs layer 2024-11-01T16:45:32.0456384Z 6cf78d9b6480: Pulling fs layer 2024-11-01T16:45:32.0456798Z accad7e3a239: Pulling fs layer 2024-11-01T16:45:32.0457197Z 1b6c965722e8: Waiting 2024-11-01T16:45:32.0457523Z 6cf78d9b6480: Waiting 2024-11-01T16:45:32.0457880Z b0b6f60fa253: Pulling fs layer 2024-11-01T16:45:32.0458302Z acbcfa904fd5: Pulling fs layer 2024-11-01T16:45:32.0458699Z 9113890e1773: Waiting 2024-11-01T16:45:32.0459042Z b0b6f60fa253: Waiting 2024-11-01T16:45:32.0459388Z 024bd3c6bf01: Pulling fs layer 2024-11-01T16:45:32.0459805Z 9cc11594aa2a: Pulling fs layer 2024-11-01T16:45:32.0460205Z c5cb1a39bf9a: Waiting 2024-11-01T16:45:32.0460576Z 01fd6f2e9fa3: Pulling fs layer 2024-11-01T16:45:32.0461091Z d06fa63dc0b7: Pulling fs layer 2024-11-01T16:45:32.0461684Z 72612b798350: Pulling fs layer 2024-11-01T16:45:32.0462095Z d06fa63dc0b7: Waiting 2024-11-01T16:45:32.0462453Z 8764bbbef590: Waiting 2024-11-01T16:45:32.0462810Z 01fd6f2e9fa3: Waiting 2024-11-01T16:45:32.0463163Z 1200fc258863: Waiting 2024-11-01T16:45:32.0463495Z 72612b798350: Waiting 2024-11-01T16:45:32.0463840Z 024bd3c6bf01: Waiting 2024-11-01T16:45:32.0464191Z 9cc11594aa2a: Waiting 2024-11-01T16:45:32.1762897Z 09d415c238d7: Verifying Checksum 2024-11-01T16:45:32.1763446Z 09d415c238d7: Download complete 2024-11-01T16:45:32.2596276Z 41f16248e682: Verifying Checksum 2024-11-01T16:45:32.2596792Z 41f16248e682: Download complete 2024-11-01T16:45:32.3646007Z 95d7b7817039: Download complete 2024-11-01T16:45:32.3839329Z 7a2c55901189: Download complete 2024-11-01T16:45:32.5050417Z ab17245097e4: Verifying Checksum 2024-11-01T16:45:32.5050928Z ab17245097e4: Download complete 2024-11-01T16:45:32.5800016Z dfecd7e9912b: Verifying Checksum 2024-11-01T16:45:32.5800567Z dfecd7e9912b: Download complete 2024-11-01T16:45:32.6567745Z 464a8f745445: Verifying Checksum 2024-11-01T16:45:32.6568269Z 464a8f745445: Download complete 2024-11-01T16:45:32.6635001Z 9fe6e2e61518: Verifying Checksum 2024-11-01T16:45:32.6635451Z 9fe6e2e61518: Download complete 2024-11-01T16:45:32.7407496Z 0723abde19c0: Download complete 2024-11-01T16:45:32.8121406Z 353d86cedcdc: Download complete 2024-11-01T16:45:33.1225529Z 7a2c55901189: Pull complete 2024-11-01T16:45:33.3269788Z 09d415c238d7: Pull complete 2024-11-01T16:45:34.0813305Z 9fe6e2e61518: Pull complete 2024-11-01T16:45:34.0945084Z 41f16248e682: Pull complete 2024-11-01T16:45:34.1074511Z 95d7b7817039: Pull complete 2024-11-01T16:45:39.4228192Z 0299f4e4d528: Verifying Checksum 2024-11-01T16:45:39.4228736Z 0299f4e4d528: Download complete 2024-11-01T16:45:39.5088848Z 86c353c3bc58: Verifying Checksum 2024-11-01T16:45:39.5089355Z 86c353c3bc58: Download complete 2024-11-01T16:45:39.5909669Z 4f10af237965: Verifying Checksum 2024-11-01T16:45:39.5910194Z 4f10af237965: Download complete 2024-11-01T16:45:39.6686147Z b7b00fcb07ca: Verifying Checksum 2024-11-01T16:45:39.6686684Z b7b00fcb07ca: Download complete 2024-11-01T16:45:40.6408089Z db97392a2fce: Verifying Checksum 2024-11-01T16:45:40.6408621Z db97392a2fce: Download complete 2024-11-01T16:45:40.7235407Z 27e2d54f402a: Download complete 2024-11-01T16:45:40.8065241Z 8269c56845c3: Verifying Checksum 2024-11-01T16:45:40.8065767Z 8269c56845c3: Download complete 2024-11-01T16:45:40.8876917Z 6c7e3cfd6320: Verifying Checksum 2024-11-01T16:45:40.8877438Z 6c7e3cfd6320: Download complete 2024-11-01T16:45:46.2528597Z 8f6c90485347: Verifying Checksum 2024-11-01T16:45:46.2529206Z 8f6c90485347: Download complete 2024-11-01T16:45:46.3246046Z 5ffae31f204a: Verifying Checksum 2024-11-01T16:45:46.3246872Z 5ffae31f204a: Download complete 2024-11-01T16:45:46.4113126Z 879b9cee6016: Download complete 2024-11-01T16:45:46.4898092Z ead67a628392: Verifying Checksum 2024-11-01T16:45:46.4898779Z ead67a628392: Download complete 2024-11-01T16:45:46.6061904Z 2f25291ccc90: Download complete 2024-11-01T16:45:46.6931137Z 5cdf685c64c3: Download complete 2024-11-01T16:45:47.9650882Z 4eab1e4b6316: Verifying Checksum 2024-11-01T16:45:47.9651438Z 4eab1e4b6316: Download complete 2024-11-01T16:45:48.0531775Z 7caa1481847d: Verifying Checksum 2024-11-01T16:45:48.0532329Z 7caa1481847d: Download complete 2024-11-01T16:45:48.1250091Z dafa8b6784dd: Verifying Checksum 2024-11-01T16:45:48.1250836Z dafa8b6784dd: Download complete 2024-11-01T16:45:48.2564545Z e41e511e406b: Download complete 2024-11-01T16:45:48.3354544Z 7cd186138469: Verifying Checksum 2024-11-01T16:45:48.3355208Z 7cd186138469: Download complete 2024-11-01T16:45:52.6577941Z 38aadcbec807: Verifying Checksum 2024-11-01T16:45:52.6578584Z 38aadcbec807: Download complete 2024-11-01T16:45:52.7213276Z f0b80c9d9a16: Download complete 2024-11-01T16:45:52.7870929Z bad6bd964ba4: Verifying Checksum 2024-11-01T16:45:52.7871517Z bad6bd964ba4: Download complete 2024-11-01T16:45:53.2036944Z a348147f958b: Verifying Checksum 2024-11-01T16:45:53.2037481Z a348147f958b: Download complete 2024-11-01T16:45:53.2831756Z a3dfd1341f5c: Verifying Checksum 2024-11-01T16:45:53.2832305Z a3dfd1341f5c: Download complete 2024-11-01T16:45:53.3410135Z c6d6b49273d1: Download complete 2024-11-01T16:45:53.5974342Z 9dec96f6a1c9: Verifying Checksum 2024-11-01T16:45:53.5975193Z 9dec96f6a1c9: Download complete 2024-11-01T16:45:53.6750148Z 98a2657c4e60: Verifying Checksum 2024-11-01T16:45:53.6750698Z 98a2657c4e60: Download complete 2024-11-01T16:45:53.7424536Z 955cea5b7f36: Verifying Checksum 2024-11-01T16:45:53.7425091Z 955cea5b7f36: Download complete 2024-11-01T16:45:53.8237756Z f38b3a72f683: Verifying Checksum 2024-11-01T16:45:53.8238346Z f38b3a72f683: Download complete 2024-11-01T16:45:53.8339784Z 4f4fb700ef54: Verifying Checksum 2024-11-01T16:45:53.8340286Z 4f4fb700ef54: Download complete 2024-11-01T16:45:53.9112059Z c5cb1a39bf9a: Download complete 2024-11-01T16:45:53.9821455Z 6765d3ee658a: Verifying Checksum 2024-11-01T16:45:53.9822174Z 6765d3ee658a: Download complete 2024-11-01T16:45:54.4672666Z cf81cf49414b: Verifying Checksum 2024-11-01T16:45:54.4673290Z cf81cf49414b: Download complete 2024-11-01T16:45:54.5561092Z 303676327e27: Download complete 2024-11-01T16:45:54.6418782Z 09aac48ef483: Verifying Checksum 2024-11-01T16:45:54.6419343Z 09aac48ef483: Download complete 2024-11-01T16:45:54.7087535Z ce24c94fd6ad: Verifying Checksum 2024-11-01T16:45:54.7088206Z ce24c94fd6ad: Download complete 2024-11-01T16:45:54.7938825Z 8fd476fea7ee: Verifying Checksum 2024-11-01T16:45:54.7939386Z 8fd476fea7ee: Download complete 2024-11-01T16:45:57.1640777Z d67c111fa588: Verifying Checksum 2024-11-01T16:45:57.1641376Z d67c111fa588: Download complete 2024-11-01T16:45:57.2378902Z 1de6b248716c: Verifying Checksum 2024-11-01T16:45:57.2379686Z 1de6b248716c: Download complete 2024-11-01T16:45:57.3266595Z f4eec39b74d5: Verifying Checksum 2024-11-01T16:45:57.3267540Z f4eec39b74d5: Download complete 2024-11-01T16:45:57.4089723Z bc86c61c49d6: Verifying Checksum 2024-11-01T16:45:57.4090312Z bc86c61c49d6: Download complete 2024-11-01T16:45:57.4937949Z 4790041b71a9: Download complete 2024-11-01T16:45:57.5740099Z 9113890e1773: Verifying Checksum 2024-11-01T16:45:57.5740619Z 9113890e1773: Download complete 2024-11-01T16:45:57.7057880Z ad6d4c88cacf: Verifying Checksum 2024-11-01T16:45:57.7058446Z ad6d4c88cacf: Download complete 2024-11-01T16:45:57.7785460Z 760e1aa1b5e3: Download complete 2024-11-01T16:45:57.8450586Z 3c4483bbdce8: Download complete 2024-11-01T16:45:57.9240815Z 8764bbbef590: Verifying Checksum 2024-11-01T16:45:57.9241736Z 8764bbbef590: Download complete 2024-11-01T16:45:57.9983137Z 71826d5dd434: Verifying Checksum 2024-11-01T16:45:57.9983705Z 71826d5dd434: Download complete 2024-11-01T16:45:58.1413432Z 7b4c7f0a396e: Verifying Checksum 2024-11-01T16:45:58.1413968Z 7b4c7f0a396e: Download complete 2024-11-01T16:45:58.2267797Z 1200fc258863: Download complete 2024-11-01T16:45:58.8292288Z 1b6c965722e8: Verifying Checksum 2024-11-01T16:45:58.8292922Z 1b6c965722e8: Download complete 2024-11-01T16:45:58.9066178Z 6cf78d9b6480: Verifying Checksum 2024-11-01T16:45:58.9066792Z 6cf78d9b6480: Download complete 2024-11-01T16:46:00.7280760Z 8f6c90485347: Pull complete 2024-11-01T16:46:01.0767900Z ab17245097e4: Pull complete 2024-11-01T16:46:01.4904132Z dfecd7e9912b: Pull complete 2024-11-01T16:46:01.9235819Z 464a8f745445: Pull complete 2024-11-01T16:46:10.1578281Z 82c15385d36d: Verifying Checksum 2024-11-01T16:46:10.1578893Z 82c15385d36d: Download complete 2024-11-01T16:46:10.2414098Z b0b6f60fa253: Download complete 2024-11-01T16:46:10.3577456Z acbcfa904fd5: Verifying Checksum 2024-11-01T16:46:10.3578103Z acbcfa904fd5: Download complete 2024-11-01T16:46:10.7771304Z 024bd3c6bf01: Verifying Checksum 2024-11-01T16:46:10.7771865Z 024bd3c6bf01: Download complete 2024-11-01T16:46:10.8584980Z 9cc11594aa2a: Verifying Checksum 2024-11-01T16:46:10.8585561Z 9cc11594aa2a: Download complete 2024-11-01T16:46:10.9706633Z 01fd6f2e9fa3: Verifying Checksum 2024-11-01T16:46:10.9707202Z 01fd6f2e9fa3: Download complete 2024-11-01T16:46:11.0581304Z d06fa63dc0b7: Download complete 2024-11-01T16:46:11.1367803Z 72612b798350: Verifying Checksum 2024-11-01T16:46:11.1368641Z 72612b798350: Download complete 2024-11-01T16:46:12.6931288Z accad7e3a239: Download complete 2024-11-01T16:46:27.3532980Z d67c111fa588: Pull complete 2024-11-01T16:46:27.8244808Z 0723abde19c0: Pull complete 2024-11-01T16:46:28.2871926Z 353d86cedcdc: Pull complete 2024-11-01T16:46:37.9539071Z 4cf3da115849: Verifying Checksum 2024-11-01T16:46:37.9539731Z 4cf3da115849: Download complete 2024-11-01T16:46:38.1695731Z 0299f4e4d528: Pull complete 2024-11-01T16:46:38.6296386Z 86c353c3bc58: Pull complete 2024-11-01T16:46:39.0568774Z 4f10af237965: Pull complete 2024-11-01T16:46:39.5161248Z b7b00fcb07ca: Pull complete 2024-11-01T16:46:41.8513563Z db97392a2fce: Pull complete 2024-11-01T16:46:42.3002583Z 27e2d54f402a: Pull complete 2024-11-01T16:46:42.7572408Z 8269c56845c3: Pull complete 2024-11-01T16:46:43.2599575Z 6c7e3cfd6320: Pull complete 2024-11-01T16:47:25.2006248Z 82c15385d36d: Pull complete 2024-11-01T16:47:25.6553442Z 5ffae31f204a: Pull complete 2024-11-01T16:47:26.1606235Z 879b9cee6016: Pull complete 2024-11-01T16:47:26.6075742Z ead67a628392: Pull complete 2024-11-01T16:47:27.0855869Z 2f25291ccc90: Pull complete 2024-11-01T16:47:27.5035701Z 5cdf685c64c3: Pull complete 2024-11-01T16:47:30.5426434Z 4eab1e4b6316: Pull complete 2024-11-01T16:47:30.6656988Z 7caa1481847d: Pull complete 2024-11-01T16:47:31.0663557Z dafa8b6784dd: Pull complete 2024-11-01T16:47:31.5210466Z dfccec371c6f: Pull complete 2024-11-01T16:47:32.0134779Z e41e511e406b: Pull complete 2024-11-01T16:47:32.5083999Z 7cd186138469: Pull complete 2024-11-01T16:47:39.7930089Z 38aadcbec807: Pull complete 2024-11-01T16:47:39.9807864Z f0b80c9d9a16: Pull complete 2024-11-01T16:47:40.3557899Z bad6bd964ba4: Pull complete 2024-11-01T16:47:41.3394385Z a348147f958b: Pull complete 2024-11-01T16:47:41.7578710Z a3dfd1341f5c: Pull complete 2024-11-01T16:47:42.1174450Z c6d6b49273d1: Pull complete 2024-11-01T16:47:42.7655814Z 9dec96f6a1c9: Pull complete 2024-11-01T16:47:43.2065999Z 98a2657c4e60: Pull complete 2024-11-01T16:47:44.0648660Z 955cea5b7f36: Pull complete 2024-11-01T16:47:44.4903108Z f38b3a72f683: Pull complete 2024-11-01T16:47:44.9613659Z 4f4fb700ef54: Pull complete 2024-11-01T16:47:45.4303513Z c5cb1a39bf9a: Pull complete 2024-11-01T16:47:45.8973926Z 6765d3ee658a: Pull complete 2024-11-01T16:47:48.0118586Z cf81cf49414b: Pull complete 2024-11-01T16:47:48.4616487Z 303676327e27: Pull complete 2024-11-01T16:47:48.9593366Z 09aac48ef483: Pull complete 2024-11-01T16:47:49.8808017Z ce24c94fd6ad: Pull complete 2024-11-01T16:47:50.3606936Z 8fd476fea7ee: Pull complete 2024-11-01T16:48:55.8977475Z 4cf3da115849: Pull complete 2024-11-01T16:48:56.3884861Z 1de6b248716c: Pull complete 2024-11-01T16:48:56.8353053Z f4eec39b74d5: Pull complete 2024-11-01T16:48:57.7497396Z bc86c61c49d6: Pull complete 2024-11-01T16:48:58.6988815Z 4790041b71a9: Pull complete 2024-11-01T16:48:59.1526051Z 9113890e1773: Pull complete 2024-11-01T16:48:59.7231668Z ad6d4c88cacf: Pull complete 2024-11-01T16:49:00.2137889Z 760e1aa1b5e3: Pull complete 2024-11-01T16:49:00.6407081Z 3c4483bbdce8: Pull complete 2024-11-01T16:49:01.0217099Z 8764bbbef590: Pull complete 2024-11-01T16:49:01.4853058Z 71826d5dd434: Pull complete 2024-11-01T16:49:02.7019539Z 7b4c7f0a396e: Pull complete 2024-11-01T16:49:03.1423656Z 1200fc258863: Pull complete 2024-11-01T16:49:05.0678629Z 1b6c965722e8: Pull complete 2024-11-01T16:49:05.5526941Z 6cf78d9b6480: Pull complete 2024-11-01T16:49:18.5565113Z accad7e3a239: Pull complete 2024-11-01T16:49:18.9894040Z b0b6f60fa253: Pull complete 2024-11-01T16:49:19.4943172Z acbcfa904fd5: Pull complete 2024-11-01T16:49:20.3737942Z 024bd3c6bf01: Pull complete 2024-11-01T16:49:20.8120309Z 9cc11594aa2a: Pull complete 2024-11-01T16:49:21.2247687Z 01fd6f2e9fa3: Pull complete 2024-11-01T16:49:21.6750002Z d06fa63dc0b7: Pull complete 2024-11-01T16:49:22.1181722Z 72612b798350: Pull complete 2024-11-01T16:49:23.5913079Z Digest: sha256:7e55ca35c7b3a515f97404dcaef8e9d5373f6e358ac63c424d2d80dda1e618f7 2024-11-01T16:49:23.6655587Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:49:23.7116033Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:49:23.7212033Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:49:23.7213593Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:49:23.7222302Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:49:23.7222913Z env: 2024-11-01T16:49:23.7223245Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:49:23.7223677Z ##[endgroup] 2024-11-01T16:49:23.7460684Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-11-01T16:49:23.7461555Z with: 2024-11-01T16:49:23.7461875Z driver-version: 550.54.15 2024-11-01T16:49:23.7462296Z env: 2024-11-01T16:49:23.7462605Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:49:23.7462998Z ##[endgroup] 2024-11-01T16:49:23.7671294Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-11-01T16:49:23.7671943Z with: 2024-11-01T16:49:23.7672245Z timeout_minutes: 10 2024-11-01T16:49:23.7672588Z max_attempts: 3 2024-11-01T16:49:23.7713683Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-11-01T16:49:23.7754771Z retry_wait_seconds: 10 2024-11-01T16:49:23.7755173Z polling_interval_seconds: 1 2024-11-01T16:49:23.7755585Z warning_on_retry: true 2024-11-01T16:49:23.7755966Z continue_on_error: false 2024-11-01T16:49:23.7756314Z env: 2024-11-01T16:49:23.7756605Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:49:23.7756994Z DRIVER_VERSION: 550.54.15 2024-11-01T16:49:23.7757368Z ##[endgroup] 2024-11-01T16:49:23.8609315Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-11-01T16:49:23.8609970Z + pre_install_nvidia_driver_amzn2 2024-11-01T16:49:23.8610556Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-11-01T16:49:24.2262500Z No match for argument: nvidia-driver-latest-dkms 2024-11-01T16:49:24.2263113Z No packages marked for removal. 2024-11-01T16:49:24.2335979Z Dependencies resolved. 2024-11-01T16:49:24.2344677Z Nothing to do. 2024-11-01T16:49:24.2345170Z Complete! 2024-11-01T16:49:24.3569830Z + install_nvidia_driver_common 2024-11-01T16:49:24.3570997Z + echo 'Before installing NVIDIA driver' 2024-11-01T16:49:24.3571478Z + lspci 2024-11-01T16:49:24.3572767Z Before installing NVIDIA driver 2024-11-01T16:49:24.3703933Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-11-01T16:49:24.3704849Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-11-01T16:49:24.3705970Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-11-01T16:49:24.3706937Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-11-01T16:49:24.3707908Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-11-01T16:49:24.3708877Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-11-01T16:49:24.3709791Z 00:1b.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-11-01T16:49:24.3710648Z 00:1c.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-11-01T16:49:24.3711725Z 00:1d.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-11-01T16:49:24.3712582Z 00:1e.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-11-01T16:49:24.3713693Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-11-01T16:49:24.3714372Z + lsmod 2024-11-01T16:49:24.3742470Z Module Size Used by 2024-11-01T16:49:24.3742983Z ib_core 454656 0 2024-11-01T16:49:24.3743419Z veth 36864 0 2024-11-01T16:49:24.3743847Z nvidia_modeset 1351680 0 2024-11-01T16:49:24.3744325Z video 65536 1 nvidia_modeset 2024-11-01T16:49:24.3744840Z wmi 36864 1 video 2024-11-01T16:49:24.3745290Z nvidia_uvm 4706304 0 2024-11-01T16:49:24.3745818Z nvidia 54071296 19 nvidia_uvm,nvidia_modeset 2024-11-01T16:49:24.3746374Z drm 602112 1 nvidia 2024-11-01T16:49:24.3746883Z drm_panel_orientation_quirks 32768 1 drm 2024-11-01T16:49:24.3747500Z backlight 24576 3 video,drm,nvidia_modeset 2024-11-01T16:49:24.3748334Z i2c_core 106496 2 nvidia,drm 2024-11-01T16:49:24.3748825Z xt_conntrack 16384 1 2024-11-01T16:49:24.3749241Z nft_chain_nat 16384 3 2024-11-01T16:49:24.3749677Z xt_MASQUERADE 20480 1 2024-11-01T16:49:24.3750202Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-11-01T16:49:24.3750775Z nf_conntrack_netlink 57344 0 2024-11-01T16:49:24.3751465Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-11-01T16:49:24.3752233Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-11-01T16:49:24.3752758Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-11-01T16:49:24.3753255Z xfrm_user 57344 1 2024-11-01T16:49:24.3753817Z xfrm_algo 16384 1 xfrm_user 2024-11-01T16:49:24.3754283Z xt_addrtype 16384 2 2024-11-01T16:49:24.3754698Z nft_compat 20480 4 2024-11-01T16:49:24.3755196Z nf_tables 311296 57 nft_compat,nft_chain_nat 2024-11-01T16:49:24.3755909Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-11-01T16:49:24.3756527Z br_netfilter 36864 0 2024-11-01T16:49:24.3756982Z bridge 323584 1 br_netfilter 2024-11-01T16:49:24.3757472Z stp 16384 1 bridge 2024-11-01T16:49:24.3757925Z llc 16384 2 bridge,stp 2024-11-01T16:49:24.3758407Z overlay 167936 0 2024-11-01T16:49:24.3758813Z tls 114688 0 2024-11-01T16:49:24.3759214Z nls_ascii 16384 1 2024-11-01T16:49:24.3759618Z nls_cp437 20480 1 2024-11-01T16:49:24.3760020Z vfat 24576 1 2024-11-01T16:49:24.3760418Z fat 86016 1 vfat 2024-11-01T16:49:24.3760846Z ena 167936 0 2024-11-01T16:49:24.3761243Z i8042 45056 0 2024-11-01T16:49:24.3761656Z serio 28672 3 i8042 2024-11-01T16:49:24.3762094Z button 24576 0 2024-11-01T16:49:24.3762508Z skx_edac_common 28672 0 2024-11-01T16:49:24.3762930Z ghash_clmulni_intel 16384 0 2024-11-01T16:49:24.3763514Z sunrpc 692224 1 2024-11-01T16:49:24.3763924Z sch_fq_codel 20480 33 2024-11-01T16:49:24.3764332Z dm_mod 188416 0 2024-11-01T16:49:24.3764730Z fuse 163840 1 2024-11-01T16:49:24.3765124Z configfs 57344 1 2024-11-01T16:49:24.3765525Z loop 36864 0 2024-11-01T16:49:24.3765933Z dax 45056 1 dm_mod 2024-11-01T16:49:24.3766372Z dmi_sysfs 20480 0 2024-11-01T16:49:24.3766781Z crc32_pclmul 16384 0 2024-11-01T16:49:24.3767175Z crc32c_intel 24576 0 2024-11-01T16:49:24.3767582Z efivarfs 24576 1 2024-11-01T16:49:24.3768155Z + modinfo nvidia 2024-11-01T16:49:24.3769029Z filename: /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-11-01T16:49:24.3769870Z alias: char-major-195-* 2024-11-01T16:49:24.3770410Z version: 550.54.15 2024-11-01T16:49:24.3770792Z supported: external 2024-11-01T16:49:24.3771177Z license: NVIDIA 2024-11-01T16:49:24.3771605Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-11-01T16:49:24.3772167Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-11-01T16:49:24.3772700Z srcversion: 833721318DA517F0C2FEC97 2024-11-01T16:49:24.3773226Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-11-01T16:49:24.3773812Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-11-01T16:49:24.3774384Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-11-01T16:49:24.3774949Z depends: i2c-core,drm 2024-11-01T16:49:24.3775357Z retpoline: Y 2024-11-01T16:49:24.3775686Z name: nvidia 2024-11-01T16:49:24.3776420Z vermagic: 6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-11-01T16:49:24.3777462Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-11-01T16:49:24.3778372Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-11-01T16:49:24.3779078Z parm: NVreg_ResmanDebugLevel:int 2024-11-01T16:49:24.3779590Z parm: NVreg_RmLogonRC:int 2024-11-01T16:49:24.3780075Z parm: NVreg_ModifyDeviceFiles:int 2024-11-01T16:49:24.3780603Z parm: NVreg_DeviceFileUID:int 2024-11-01T16:49:24.3781180Z parm: NVreg_DeviceFileGID:int 2024-11-01T16:49:24.3781687Z parm: NVreg_DeviceFileMode:int 2024-11-01T16:49:24.3782309Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-11-01T16:49:24.3782976Z parm: NVreg_UsePageAttributeTable:int 2024-11-01T16:49:24.3783518Z parm: NVreg_EnablePCIeGen3:int 2024-11-01T16:49:24.3784016Z parm: NVreg_EnableMSI:int 2024-11-01T16:49:24.3784491Z parm: NVreg_TCEBypassMode:int 2024-11-01T16:49:24.3785020Z parm: NVreg_EnableStreamMemOPs:int 2024-11-01T16:49:24.3785643Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-11-01T16:49:24.3786333Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-11-01T16:49:24.3787001Z parm: NVreg_EnableS0ixPowerManagement:int 2024-11-01T16:49:24.3787722Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-11-01T16:49:24.3788436Z parm: NVreg_DynamicPowerManagement:int 2024-11-01T16:49:24.3789164Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-11-01T16:49:24.3789877Z parm: NVreg_EnableGpuFirmware:int 2024-11-01T16:49:24.3790436Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-11-01T16:49:24.3791059Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-11-01T16:49:24.3791703Z parm: NVreg_EnableUserNUMAManagement:int 2024-11-01T16:49:24.3792285Z parm: NVreg_MemoryPoolSize:int 2024-11-01T16:49:24.3792927Z parm: NVreg_KMallocHeapMaxSize:int 2024-11-01T16:49:24.3793459Z parm: NVreg_VMallocHeapMaxSize:int 2024-11-01T16:49:24.3793985Z parm: NVreg_IgnoreMMIOCheck:int 2024-11-01T16:49:24.3794498Z parm: NVreg_NvLinkDisable:int 2024-11-01T16:49:24.3795070Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-11-01T16:49:24.3795669Z parm: NVreg_RegisterPCIDriver:int 2024-11-01T16:49:24.3796207Z parm: NVreg_EnableResizableBar:int 2024-11-01T16:49:24.3796744Z parm: NVreg_EnableDbgBreakpoint:int 2024-11-01T16:49:24.3797418Z parm: NVreg_EnableNonblockingOpen:int 2024-11-01T16:49:24.3797952Z parm: NVreg_RegistryDwords:charp 2024-11-01T16:49:24.3798499Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-11-01T16:49:24.3799025Z parm: NVreg_RmMsg:charp 2024-11-01T16:49:24.3799624Z parm: NVreg_GpuBlacklist:charp 2024-11-01T16:49:24.3800149Z parm: NVreg_TemporaryFilePath:charp 2024-11-01T16:49:24.3800678Z parm: NVreg_ExcludedGpus:charp 2024-11-01T16:49:24.3801190Z parm: NVreg_DmaRemapPeerMmio:int 2024-11-01T16:49:24.3801732Z parm: NVreg_RmNvlinkBandwidth:charp 2024-11-01T16:49:24.3802378Z parm: NVreg_ImexChannelCount:int 2024-11-01T16:49:24.3802893Z parm: rm_firmware_active:charp 2024-11-01T16:49:24.3803358Z + HAS_NVIDIA_DRIVER=0 2024-11-01T16:49:24.3803795Z ++ command -v nvidia-smi 2024-11-01T16:49:24.3804238Z + '[' -x /usr/bin/nvidia-smi ']' 2024-11-01T16:49:24.3804626Z + set +e 2024-11-01T16:49:24.3805210Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-11-01T16:49:24.4080933Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-11-01T16:49:24.4081473Z + NVIDIA_SMI_STATUS=0 2024-11-01T16:49:24.4081978Z + '[' 0 -ne 0 ']' 2024-11-01T16:49:24.4082390Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-11-01T16:49:24.4082813Z + HAS_NVIDIA_DRIVER=1 2024-11-01T16:49:24.4083685Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-11-01T16:49:24.4084524Z + set -e 2024-11-01T16:49:24.4084855Z + '[' 1 -eq 0 ']' 2024-11-01T16:49:24.4085528Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-11-01T16:49:24.4086579Z + post_install_nvidia_driver_common 2024-11-01T16:49:24.4087028Z + sudo modprobe nvidia 2024-11-01T16:49:24.5259120Z + echo 'After installing NVIDIA driver' 2024-11-01T16:49:24.5259627Z + lspci 2024-11-01T16:49:24.5259947Z After installing NVIDIA driver 2024-11-01T16:49:24.5387185Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-11-01T16:49:24.5388114Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-11-01T16:49:24.5389234Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-11-01T16:49:24.5390196Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-11-01T16:49:24.5391166Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-11-01T16:49:24.5392126Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-11-01T16:49:24.5393177Z 00:1b.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-11-01T16:49:24.5394023Z 00:1c.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-11-01T16:49:24.5394832Z 00:1d.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-11-01T16:49:24.5395649Z 00:1e.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-11-01T16:49:24.5396615Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-11-01T16:49:24.5397294Z + lsmod 2024-11-01T16:49:24.5413773Z Module Size Used by 2024-11-01T16:49:24.5414390Z ib_core 454656 0 2024-11-01T16:49:24.5414930Z veth 36864 0 2024-11-01T16:49:24.5415335Z nvidia_modeset 1351680 0 2024-11-01T16:49:24.5415800Z video 65536 1 nvidia_modeset 2024-11-01T16:49:24.5416478Z wmi 36864 1 video 2024-11-01T16:49:24.5416930Z nvidia_uvm 4706304 0 2024-11-01T16:49:24.5417463Z nvidia 54071296 19 nvidia_uvm,nvidia_modeset 2024-11-01T16:49:24.5418044Z drm 602112 1 nvidia 2024-11-01T16:49:24.5418564Z drm_panel_orientation_quirks 32768 1 drm 2024-11-01T16:49:24.5419183Z backlight 24576 3 video,drm,nvidia_modeset 2024-11-01T16:49:24.5419785Z i2c_core 106496 2 nvidia,drm 2024-11-01T16:49:24.5420289Z xt_conntrack 16384 1 2024-11-01T16:49:24.5420720Z nft_chain_nat 16384 3 2024-11-01T16:49:24.5421272Z xt_MASQUERADE 20480 1 2024-11-01T16:49:24.5421805Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-11-01T16:49:24.5422379Z nf_conntrack_netlink 57344 0 2024-11-01T16:49:24.5423075Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-11-01T16:49:24.5423840Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-11-01T16:49:24.5424380Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-11-01T16:49:24.5424862Z xfrm_user 57344 1 2024-11-01T16:49:24.5425311Z xfrm_algo 16384 1 xfrm_user 2024-11-01T16:49:24.5426024Z xt_addrtype 16384 2 2024-11-01T16:49:24.5426463Z nft_compat 20480 4 2024-11-01T16:49:24.5426995Z nf_tables 311296 57 nft_compat,nft_chain_nat 2024-11-01T16:49:24.5427706Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-11-01T16:49:24.5428354Z br_netfilter 36864 0 2024-11-01T16:49:24.5428830Z bridge 323584 1 br_netfilter 2024-11-01T16:49:24.5429335Z stp 16384 1 bridge 2024-11-01T16:49:24.5429816Z llc 16384 2 bridge,stp 2024-11-01T16:49:24.5430283Z overlay 167936 0 2024-11-01T16:49:24.5430705Z tls 114688 0 2024-11-01T16:49:24.5431127Z nls_ascii 16384 1 2024-11-01T16:49:24.5431545Z nls_cp437 20480 1 2024-11-01T16:49:24.5431956Z vfat 24576 1 2024-11-01T16:49:24.5432376Z fat 86016 1 vfat 2024-11-01T16:49:24.5432805Z ena 167936 0 2024-11-01T16:49:24.5433478Z i8042 45056 0 2024-11-01T16:49:24.5433893Z serio 28672 3 i8042 2024-11-01T16:49:24.5434331Z button 24576 0 2024-11-01T16:49:24.5434745Z skx_edac_common 28672 0 2024-11-01T16:49:24.5435157Z ghash_clmulni_intel 16384 0 2024-11-01T16:49:24.5435576Z sunrpc 692224 1 2024-11-01T16:49:24.5435986Z sch_fq_codel 20480 33 2024-11-01T16:49:24.5436395Z dm_mod 188416 0 2024-11-01T16:49:24.5436794Z fuse 163840 1 2024-11-01T16:49:24.5437182Z configfs 57344 1 2024-11-01T16:49:24.5437591Z loop 36864 0 2024-11-01T16:49:24.5438010Z dax 45056 1 dm_mod 2024-11-01T16:49:24.5438457Z dmi_sysfs 20480 0 2024-11-01T16:49:24.5438870Z crc32_pclmul 16384 0 2024-11-01T16:49:24.5439268Z crc32c_intel 24576 0 2024-11-01T16:49:24.5439680Z efivarfs 24576 1 2024-11-01T16:49:24.5440080Z + modinfo nvidia 2024-11-01T16:49:24.5440931Z filename: /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-11-01T16:49:24.5441753Z alias: char-major-195-* 2024-11-01T16:49:24.5442181Z version: 550.54.15 2024-11-01T16:49:24.5442552Z supported: external 2024-11-01T16:49:24.5442928Z license: NVIDIA 2024-11-01T16:49:24.5443351Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-11-01T16:49:24.5443898Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-11-01T16:49:24.5444415Z srcversion: 833721318DA517F0C2FEC97 2024-11-01T16:49:24.5444940Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-11-01T16:49:24.5445490Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-11-01T16:49:24.5446051Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-11-01T16:49:24.5446714Z depends: i2c-core,drm 2024-11-01T16:49:24.5447103Z retpoline: Y 2024-11-01T16:49:24.5447431Z name: nvidia 2024-11-01T16:49:24.5448110Z vermagic: 6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-11-01T16:49:24.5448925Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-11-01T16:49:24.5449651Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-11-01T16:49:24.5450318Z parm: NVreg_ResmanDebugLevel:int 2024-11-01T16:49:24.5450803Z parm: NVreg_RmLogonRC:int 2024-11-01T16:49:24.5451281Z parm: NVreg_ModifyDeviceFiles:int 2024-11-01T16:49:24.5451768Z parm: NVreg_DeviceFileUID:int 2024-11-01T16:49:24.5452245Z parm: NVreg_DeviceFileGID:int 2024-11-01T16:49:24.5452726Z parm: NVreg_DeviceFileMode:int 2024-11-01T16:49:24.5453315Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-11-01T16:49:24.5453947Z parm: NVreg_UsePageAttributeTable:int 2024-11-01T16:49:24.5454462Z parm: NVreg_EnablePCIeGen3:int 2024-11-01T16:49:24.5454932Z parm: NVreg_EnableMSI:int 2024-11-01T16:49:24.5455386Z parm: NVreg_TCEBypassMode:int 2024-11-01T16:49:24.5455987Z parm: NVreg_EnableStreamMemOPs:int 2024-11-01T16:49:24.5456579Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-11-01T16:49:24.5457218Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-11-01T16:49:24.5457848Z parm: NVreg_EnableS0ixPowerManagement:int 2024-11-01T16:49:24.5458539Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-11-01T16:49:24.5459217Z parm: NVreg_DynamicPowerManagement:int 2024-11-01T16:49:24.5459913Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-11-01T16:49:24.5460583Z parm: NVreg_EnableGpuFirmware:int 2024-11-01T16:49:24.5461196Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-11-01T16:49:24.5461989Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-11-01T16:49:24.5462635Z parm: NVreg_EnableUserNUMAManagement:int 2024-11-01T16:49:24.5463223Z parm: NVreg_MemoryPoolSize:int 2024-11-01T16:49:24.5463761Z parm: NVreg_KMallocHeapMaxSize:int 2024-11-01T16:49:24.5464407Z parm: NVreg_VMallocHeapMaxSize:int 2024-11-01T16:49:24.5464946Z parm: NVreg_IgnoreMMIOCheck:int 2024-11-01T16:49:24.5465466Z parm: NVreg_NvLinkDisable:int 2024-11-01T16:49:24.5466058Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-11-01T16:49:24.5466679Z parm: NVreg_RegisterPCIDriver:int 2024-11-01T16:49:24.5467237Z parm: NVreg_EnableResizableBar:int 2024-11-01T16:49:24.5467788Z parm: NVreg_EnableDbgBreakpoint:int 2024-11-01T16:49:24.5468374Z parm: NVreg_EnableNonblockingOpen:int 2024-11-01T16:49:24.5468943Z parm: NVreg_RegistryDwords:charp 2024-11-01T16:49:24.5469528Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-11-01T16:49:24.5470090Z parm: NVreg_RmMsg:charp 2024-11-01T16:49:24.5470550Z parm: NVreg_GpuBlacklist:charp 2024-11-01T16:49:24.5471092Z parm: NVreg_TemporaryFilePath:charp 2024-11-01T16:49:24.5471646Z parm: NVreg_ExcludedGpus:charp 2024-11-01T16:49:24.5472177Z parm: NVreg_DmaRemapPeerMmio:int 2024-11-01T16:49:24.5472733Z parm: NVreg_RmNvlinkBandwidth:charp 2024-11-01T16:49:24.5473282Z parm: NVreg_ImexChannelCount:int 2024-11-01T16:49:24.5473899Z parm: rm_firmware_active:charp 2024-11-01T16:49:24.5474312Z + set +e 2024-11-01T16:49:24.5474626Z + nvidia-smi 2024-11-01T16:49:24.5931090Z Fri Nov 1 16:49:24 2024 2024-11-01T16:49:24.5931897Z +-----------------------------------------------------------------------------------------+ 2024-11-01T16:49:24.5932872Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-11-01T16:49:24.5933816Z |-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:49:24.5934764Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-11-01T16:49:24.5935821Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-11-01T16:49:24.5936628Z | | | MIG M. | 2024-11-01T16:49:24.5937254Z |=========================================+========================+======================| 2024-11-01T16:49:24.6105343Z | 0 Tesla T4 On | 00000000:00:1B.0 Off | 0 | 2024-11-01T16:49:24.6106219Z | N/A 29C P8 9W / 70W | 0MiB / 15360MiB | 0% Default | 2024-11-01T16:49:24.6106983Z | | | N/A | 2024-11-01T16:49:24.6107857Z +-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:49:24.6262816Z | 1 Tesla T4 On | 00000000:00:1C.0 Off | 0 | 2024-11-01T16:49:24.6263688Z | N/A 29C P8 9W / 70W | 0MiB / 15360MiB | 0% Default | 2024-11-01T16:49:24.6264663Z | | | N/A | 2024-11-01T16:49:24.6265545Z +-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:49:24.6423549Z | 2 Tesla T4 On | 00000000:00:1D.0 Off | 0 | 2024-11-01T16:49:24.6424431Z | N/A 28C P8 9W / 70W | 0MiB / 15360MiB | 0% Default | 2024-11-01T16:49:24.6425195Z | | | N/A | 2024-11-01T16:49:24.6426067Z +-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:49:24.6577989Z | 3 Tesla T4 On | 00000000:00:1E.0 Off | 0 | 2024-11-01T16:49:24.6578856Z | N/A 29C P8 9W / 70W | 0MiB / 15360MiB | 0% Default | 2024-11-01T16:49:24.6579612Z | | | N/A | 2024-11-01T16:49:24.6580650Z +-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:49:24.6597426Z 2024-11-01T16:49:24.6598351Z +-----------------------------------------------------------------------------------------+ 2024-11-01T16:49:24.6599107Z | Processes: | 2024-11-01T16:49:24.6599943Z | GPU GI CI PID Type Process name GPU Memory | 2024-11-01T16:49:24.6600758Z | ID ID Usage | 2024-11-01T16:49:24.6601398Z |=========================================================================================| 2024-11-01T16:49:24.6623501Z | No running processes found | 2024-11-01T16:49:24.6624543Z +-----------------------------------------------------------------------------------------+ 2024-11-01T16:49:25.2681073Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-11-01T16:49:25.2897389Z Tesla T4 2024-11-01T16:49:25.2978256Z + NVIDIA_SMI_STATUS=0 2024-11-01T16:49:25.2979539Z + '[' 0 -eq 0 ']' 2024-11-01T16:49:25.2980676Z + echo 'INFO: Ignoring allowed status 0' 2024-11-01T16:49:25.2982141Z + set -e 2024-11-01T16:49:25.2982936Z INFO: Ignoring allowed status 0 2024-11-01T16:49:25.2989464Z == Installing nvidia container toolkit for amzn2023 == 2024-11-01T16:49:25.2992125Z + sudo yum install -y yum-utils 2024-11-01T16:49:25.8576301Z Last metadata expiration check: 0:21:49 ago on Fri Nov 1 16:27:36 2024. 2024-11-01T16:49:25.8863489Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-11-01T16:49:25.9267065Z Dependencies resolved. 2024-11-01T16:49:25.9454667Z Nothing to do. 2024-11-01T16:49:25.9455012Z Complete! 2024-11-01T16:49:26.0778961Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-11-01T16:49:26.0780208Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-11-01T16:49:26.0782110Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-11-01T16:49:26.3853384Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-11-01T16:49:26.4695923Z + sudo yum install -y nvidia-docker2 2024-11-01T16:49:27.0020407Z nvidia-container-toolkit 10 kB/s | 833 B 00:00 2024-11-01T16:49:27.0306989Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-11-01T16:49:27.0697231Z Dependencies resolved. 2024-11-01T16:49:27.0886391Z Nothing to do. 2024-11-01T16:49:27.0886749Z Complete! 2024-11-01T16:49:27.2040404Z + sudo systemctl restart docker 2024-11-01T16:51:05.2542190Z nvidia-persistenced failed to initialize. Check syslog for more details. 2024-11-01T16:51:05.3073025Z Fri Nov 1 16:51:05 2024 2024-11-01T16:51:05.3074116Z +-----------------------------------------------------------------------------------------+ 2024-11-01T16:51:05.3075109Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-11-01T16:51:05.3076057Z |-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:51:05.3079963Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-11-01T16:51:05.3081110Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-11-01T16:51:05.3081959Z | | | MIG M. | 2024-11-01T16:51:05.3082616Z |=========================================+========================+======================| 2024-11-01T16:51:05.3263684Z | 0 Tesla T4 On | 00000000:00:1B.0 Off | 0 | 2024-11-01T16:51:05.3265889Z | N/A 29C P8 9W / 70W | 0MiB / 15360MiB | 0% Default | 2024-11-01T16:51:05.3266664Z | | | N/A | 2024-11-01T16:51:05.3267552Z +-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:51:05.3418669Z | 1 Tesla T4 On | 00000000:00:1C.0 Off | 0 | 2024-11-01T16:51:05.3419577Z | N/A 29C P8 9W / 70W | 0MiB / 15360MiB | 0% Default | 2024-11-01T16:51:05.3420335Z | | | N/A | 2024-11-01T16:51:05.3421306Z +-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:51:05.3569690Z | 2 Tesla T4 On | 00000000:00:1D.0 Off | 0 | 2024-11-01T16:51:05.3570607Z | N/A 29C P8 8W / 70W | 0MiB / 15360MiB | 0% Default | 2024-11-01T16:51:05.3571365Z | | | N/A | 2024-11-01T16:51:05.3572232Z +-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:51:05.3740270Z | 3 Tesla T4 On | 00000000:00:1E.0 Off | 0 | 2024-11-01T16:51:05.3741237Z | N/A 28C P8 10W / 70W | 0MiB / 15360MiB | 0% Default | 2024-11-01T16:51:05.3742188Z | | | N/A | 2024-11-01T16:51:05.3743062Z +-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:51:05.3757880Z 2024-11-01T16:51:05.3758948Z +-----------------------------------------------------------------------------------------+ 2024-11-01T16:51:05.3759728Z | Processes: | 2024-11-01T16:51:05.3760577Z | GPU GI CI PID Type Process name GPU Memory | 2024-11-01T16:51:05.3761388Z | ID ID Usage | 2024-11-01T16:51:05.3762031Z |=========================================================================================| 2024-11-01T16:51:05.3782938Z | No running processes found | 2024-11-01T16:51:05.3783990Z +-----------------------------------------------------------------------------------------+ 2024-11-01T16:51:06.9651922Z Command completed after 1 attempt(s). 2024-11-01T16:51:06.9765475Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-11-01T16:51:06.9766387Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-11-01T16:51:06.9767350Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-11-01T16:51:06.9768210Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:51:06.9780123Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:06.9780717Z env: 2024-11-01T16:51:06.9781142Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:06.9781854Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:06.9782433Z ##[endgroup] 2024-11-01T16:51:07.2641789Z Defaulting to user installation because normal site-packages is not writeable 2024-11-01T16:51:07.2828512Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.9/site-packages (5.9.1) 2024-11-01T16:51:07.2835501Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.9/site-packages (11.525.84) 2024-11-01T16:51:07.4244879Z Prepare all required actions 2024-11-01T16:51:07.4245405Z Getting action download info 2024-11-01T16:51:07.6227148Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-11-01T16:51:07.8185619Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-11-01T16:51:08.0501691Z ##[group]Run ./.github/actions/download-build-artifacts 2024-11-01T16:51:08.0502262Z with: 2024-11-01T16:51:08.0502614Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-11-01T16:51:08.0503118Z s3-bucket: gha-artifacts 2024-11-01T16:51:08.0503496Z env: 2024-11-01T16:51:08.0503779Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:08.0504295Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:08.0504852Z ##[endgroup] 2024-11-01T16:51:08.0544449Z ##[group]Run seemethere/download-artifact-s3@v4 2024-11-01T16:51:08.0544971Z with: 2024-11-01T16:51:08.0545369Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-11-01T16:51:08.0545877Z s3-bucket: gha-artifacts 2024-11-01T16:51:08.0546262Z region: us-east-1 2024-11-01T16:51:08.0546594Z env: 2024-11-01T16:51:08.0546905Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:08.0547420Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:08.0547981Z ##[endgroup] 2024-11-01T16:51:08.6088866Z (node:117613) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-11-01T16:51:08.6089765Z 2024-11-01T16:51:08.6090072Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-11-01T16:51:08.6090905Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-11-01T16:51:08.6091948Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-11-01T16:51:08.6880552Z Found 1 objects with prefix pytorch/pytorch/11632514903/linux-focal-cuda11.8-py3.10-gcc9/ 2024-11-01T16:51:08.6881828Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-11-01T16:51:21.7020580Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-11-01T16:51:21.7027478Z Artifact download has finished successfully 2024-11-01T16:51:21.7301594Z ##[group]Run unzip -o artifacts.zip 2024-11-01T16:51:21.7302101Z unzip -o artifacts.zip 2024-11-01T16:51:21.7309792Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:21.7310384Z env: 2024-11-01T16:51:21.7310699Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:21.7311227Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:21.7311801Z ##[endgroup] 2024-11-01T16:51:21.7358552Z Archive: artifacts.zip 2024-11-01T16:51:21.7359016Z creating: dist/ 2024-11-01T16:51:23.7706827Z inflating: dist/torch-2.6.0a0+gitd1aa4ef-cp310-cp310-linux_x86_64.whl 2024-11-01T16:51:23.7707637Z creating: build/custom_test_artifacts/ 2024-11-01T16:51:23.7708343Z creating: build/custom_test_artifacts/custom-op-build/ 2024-11-01T16:51:23.7709205Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-11-01T16:51:23.7710223Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-11-01T16:51:23.7715004Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-11-01T16:51:23.7716122Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-11-01T16:51:23.7717250Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-11-01T16:51:23.7718435Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-11-01T16:51:23.7719587Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-11-01T16:51:23.7720899Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-11-01T16:51:23.7722536Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-11-01T16:51:23.7723756Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-11-01T16:51:23.7725104Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-11-01T16:51:23.7726502Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-11-01T16:51:23.7727914Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-11-01T16:51:23.7729301Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-11-01T16:51:23.7730674Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-11-01T16:51:23.7732075Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-11-01T16:51:23.7733498Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-11-01T16:51:23.7734743Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-11-01T16:51:23.7735954Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-11-01T16:51:23.7779987Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-11-01T16:51:23.7782041Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-11-01T16:51:23.7783881Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-11-01T16:51:23.7785758Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-11-01T16:51:23.7787623Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-11-01T16:51:23.7789356Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-11-01T16:51:23.7791125Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-11-01T16:51:23.7792996Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-11-01T16:51:23.7794691Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-11-01T16:51:23.7827049Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-11-01T16:51:23.7869798Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-11-01T16:51:23.7871606Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-11-01T16:51:23.7873257Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-11-01T16:51:23.7874912Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-11-01T16:51:23.7876388Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-11-01T16:51:23.7878086Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-11-01T16:51:23.7879534Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-11-01T16:51:23.7881006Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-11-01T16:51:23.7942596Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-11-01T16:51:23.8007405Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-11-01T16:51:23.8009145Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-11-01T16:51:23.8010352Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-11-01T16:51:23.8011375Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-11-01T16:51:23.8012455Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-11-01T16:51:23.8013562Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-11-01T16:51:23.8014790Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-11-01T16:51:23.8016185Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-11-01T16:51:23.8017540Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-11-01T16:51:23.8018787Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-11-01T16:51:23.8020077Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-11-01T16:51:23.8021650Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-11-01T16:51:23.8023025Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-11-01T16:51:23.8024374Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-11-01T16:51:23.8025706Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-11-01T16:51:23.8034856Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-11-01T16:51:23.8183731Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-11-01T16:51:23.8185010Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-11-01T16:51:23.8186382Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-11-01T16:51:23.8187902Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-11-01T16:51:23.8189359Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-11-01T16:51:23.8190726Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-11-01T16:51:23.8192138Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-11-01T16:51:23.8193671Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-11-01T16:51:23.8195040Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-11-01T16:51:23.8196405Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-11-01T16:51:23.8197757Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-11-01T16:51:23.8210075Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-11-01T16:51:23.8297995Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-11-01T16:51:23.8299491Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-11-01T16:51:23.8300810Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-11-01T16:51:23.8302477Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-11-01T16:51:23.8303625Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-11-01T16:51:23.8304836Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-11-01T16:51:23.8305954Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-11-01T16:51:23.8306997Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-11-01T16:51:23.8307946Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-11-01T16:51:23.8308935Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-11-01T16:51:23.8430580Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-11-01T16:51:23.8499417Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-11-01T16:51:23.8500300Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-11-01T16:51:23.8501231Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-11-01T16:51:23.8502407Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-11-01T16:51:23.8506475Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-11-01T16:51:23.8507627Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-11-01T16:51:23.8508779Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-11-01T16:51:23.8509986Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-11-01T16:51:23.8511151Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-11-01T16:51:23.8512508Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-11-01T16:51:23.8513973Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-11-01T16:51:23.8515156Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-11-01T16:51:23.8516339Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-11-01T16:51:23.8517731Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-11-01T16:51:23.8519118Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-11-01T16:51:23.8520471Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-11-01T16:51:23.8521815Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-11-01T16:51:23.8523208Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-11-01T16:51:23.8524600Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-11-01T16:51:23.8525835Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-11-01T16:51:23.8527027Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-11-01T16:51:23.8571787Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-11-01T16:51:23.8573557Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-11-01T16:51:23.8575305Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-11-01T16:51:23.8577292Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-11-01T16:51:23.8579081Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-11-01T16:51:23.8580953Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-11-01T16:51:23.8583058Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-11-01T16:51:23.8584820Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-11-01T16:51:23.8586562Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-11-01T16:51:23.8618581Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-11-01T16:51:23.8659392Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-11-01T16:51:23.8661246Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-11-01T16:51:23.8663037Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-11-01T16:51:23.8664588Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-11-01T16:51:23.8666108Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-11-01T16:51:23.8667651Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-11-01T16:51:23.8669132Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-11-01T16:51:23.8670644Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-11-01T16:51:23.8731863Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-11-01T16:51:23.8798013Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-11-01T16:51:23.8799474Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-11-01T16:51:23.8800679Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-11-01T16:51:23.8801705Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-11-01T16:51:23.8802767Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-11-01T16:51:23.8803891Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-11-01T16:51:23.8805167Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-11-01T16:51:23.8806606Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-11-01T16:51:23.8807974Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-11-01T16:51:23.8809266Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-11-01T16:51:23.8810606Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-11-01T16:51:23.8811940Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-11-01T16:51:23.8813285Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-11-01T16:51:23.8814608Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-11-01T16:51:23.8815926Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-11-01T16:51:23.8825297Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-11-01T16:51:23.8896205Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-11-01T16:51:23.8897722Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-11-01T16:51:23.8899150Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-11-01T16:51:23.8900317Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-11-01T16:51:23.8901664Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-11-01T16:51:23.8902769Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-11-01T16:51:23.8903876Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-11-01T16:51:23.8904882Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-11-01T16:51:23.8905814Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-11-01T16:51:23.8906791Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-11-01T16:51:23.8958139Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-11-01T16:51:23.8959079Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-11-01T16:51:23.8959985Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-11-01T16:51:23.8961045Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-11-01T16:51:23.8965092Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-11-01T16:51:23.8966278Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-11-01T16:51:23.8967482Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-11-01T16:51:23.8968758Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-11-01T16:51:23.8969997Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-11-01T16:51:23.8971397Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-11-01T16:51:23.8972823Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-11-01T16:51:23.8974089Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-11-01T16:51:23.8975353Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-11-01T16:51:23.8977031Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-11-01T16:51:23.8978521Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-11-01T16:51:23.8979978Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-11-01T16:51:23.8981698Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-11-01T16:51:23.8983243Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-11-01T16:51:23.8984786Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-11-01T16:51:23.8986144Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-11-01T16:51:23.8987466Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-11-01T16:51:23.9029378Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-11-01T16:51:23.9031289Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-11-01T16:51:23.9033520Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-11-01T16:51:23.9035460Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-11-01T16:51:23.9037572Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-11-01T16:51:23.9039350Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-11-01T16:51:23.9041141Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-11-01T16:51:23.9042930Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-11-01T16:51:23.9044723Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-11-01T16:51:23.9076762Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-11-01T16:51:23.9118608Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-11-01T16:51:23.9120430Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-11-01T16:51:23.9122100Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-11-01T16:51:23.9123698Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-11-01T16:51:23.9125263Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-11-01T16:51:23.9126862Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-11-01T16:51:23.9128378Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-11-01T16:51:23.9129948Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-11-01T16:51:23.9188105Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-11-01T16:51:23.9255169Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-11-01T16:51:23.9256877Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-11-01T16:51:23.9258167Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-11-01T16:51:23.9259279Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-11-01T16:51:23.9260467Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-11-01T16:51:23.9261977Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-11-01T16:51:23.9263417Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-11-01T16:51:23.9264993Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-11-01T16:51:23.9266538Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-11-01T16:51:23.9267972Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-11-01T16:51:23.9269451Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-11-01T16:51:23.9271133Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-11-01T16:51:23.9272641Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-11-01T16:51:23.9274315Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-11-01T16:51:23.9275736Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-11-01T16:51:23.9277467Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-11-01T16:51:23.9391891Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-11-01T16:51:23.9393499Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-11-01T16:51:23.9394971Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-11-01T16:51:23.9396597Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-11-01T16:51:23.9398156Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-11-01T16:51:23.9399635Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-11-01T16:51:23.9401160Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-11-01T16:51:23.9402666Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-11-01T16:51:23.9404200Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-11-01T16:51:23.9405723Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-11-01T16:51:23.9407231Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-11-01T16:51:23.9417916Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-11-01T16:51:23.9478936Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-11-01T16:51:23.9480566Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-11-01T16:51:23.9481971Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-11-01T16:51:23.9483216Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-11-01T16:51:23.9484385Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-11-01T16:51:23.9485667Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-11-01T16:51:23.9486847Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-11-01T16:51:23.9487934Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-11-01T16:51:23.9488939Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-11-01T16:51:23.9489975Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-11-01T16:51:23.9595155Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-11-01T16:51:23.9640186Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-11-01T16:51:23.9640967Z creating: build/lib/ 2024-11-01T16:51:23.9649333Z inflating: build/lib/libpthreadpool.a 2024-11-01T16:51:23.9650222Z inflating: build/lib/libclog.a 2024-11-01T16:51:23.9658568Z inflating: build/lib/libcpuinfo_internals.a 2024-11-01T16:51:23.9667757Z inflating: build/lib/libcpuinfo.a 2024-11-01T16:51:23.9670292Z inflating: build/lib/libnnpack_reference_layers.a 2024-11-01T16:51:23.9764469Z inflating: build/lib/libprotobuf-lite.a 2024-11-01T16:51:23.9831958Z inflating: build/lib/libgtest.a 2024-11-01T16:51:23.9912790Z inflating: build/lib/libbenchmark.a 2024-11-01T16:51:23.9920530Z inflating: build/lib/libittnotify.a 2024-11-01T16:51:24.0398928Z inflating: build/lib/libprotobuf.a 2024-11-01T16:51:24.0428023Z inflating: build/lib/libtensorpipe_uv.a 2024-11-01T16:51:24.0498406Z inflating: build/lib/libasmjit.a 2024-11-01T16:51:24.0635436Z inflating: build/lib/libgloo.a 2024-11-01T16:51:24.0657722Z inflating: build/lib/libfmt.a 2024-11-01T16:51:24.0658300Z inflating: build/lib/libcaffe2_nvrtc.so 2024-11-01T16:51:24.0767088Z inflating: build/lib/libc10.so 2024-11-01T16:51:24.0767678Z inflating: build/lib/libtorch_global_deps.so 2024-11-01T16:51:24.0789154Z inflating: build/lib/libpytorch_qnnpack.a 2024-11-01T16:51:24.0810398Z inflating: build/lib/libgmock.a 2024-11-01T16:51:24.1336826Z inflating: build/lib/libprotoc.a 2024-11-01T16:51:24.1337630Z inflating: build/lib/libgtest_main.a 2024-11-01T16:51:24.1338658Z inflating: build/lib/libbenchmark_main.a 2024-11-01T16:51:24.1358575Z inflating: build/lib/libnnpack.a 2024-11-01T16:51:25.1879485Z inflating: build/lib/libdnnl.a 2024-11-01T16:51:25.2466182Z inflating: build/lib/libtensorpipe.a 2024-11-01T16:51:25.2532031Z inflating: build/lib/libc10_cuda.so 2024-11-01T16:51:25.3812460Z inflating: build/lib/libfbgemm.a 2024-11-01T16:51:25.3813061Z inflating: build/lib/libgmock_main.a 2024-11-01T16:51:25.4349338Z inflating: build/lib/libkineto.a 2024-11-01T16:51:25.4614570Z inflating: build/lib/libtensorpipe_cuda.a 2024-11-01T16:51:25.4820091Z inflating: build/lib/libXNNPACK.a 2024-11-01T16:51:25.5210132Z inflating: build/lib/libgloo_cuda.a 2024-11-01T16:51:25.5254418Z inflating: build/lib/libonnx_proto.a 2024-11-01T16:51:25.6014374Z inflating: build/lib/libonnx.a 2024-11-01T16:51:28.2022410Z inflating: build/lib/libtorch_cpu.so 2024-11-01T16:51:28.2025476Z inflating: build/lib/libshm.so 2024-11-01T16:51:28.2030696Z inflating: build/lib/libunbox_lib.a 2024-11-01T16:51:29.9617554Z inflating: build/lib/libtorch_cuda.so 2024-11-01T16:51:29.9618215Z inflating: build/lib/libtorch.so 2024-11-01T16:51:29.9619893Z inflating: build/lib/libc10d_cuda_test.so 2024-11-01T16:51:30.9540743Z inflating: build/lib/libtorch_cuda_linalg.so 2024-11-01T16:51:30.9563089Z inflating: build/lib/libjitbackend_test.so 2024-11-01T16:51:30.9637769Z inflating: build/lib/libtorchbind_test.so 2024-11-01T16:51:30.9662686Z inflating: build/lib/libaoti_custom_ops.so 2024-11-01T16:51:30.9691492Z inflating: build/lib/libbackend_with_compiler.so 2024-11-01T16:51:31.1813222Z inflating: build/lib/libtorch_python.so 2024-11-01T16:51:31.1850750Z inflating: build/lib/libnnapi_backend.so 2024-11-01T16:51:31.1851343Z creating: build/bin/ 2024-11-01T16:51:31.1905433Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-11-01T16:51:31.1961596Z inflating: build/bin/c10_DeviceGuard_test 2024-11-01T16:51:31.2017860Z inflating: build/bin/c10_Device_test 2024-11-01T16:51:31.2080664Z inflating: build/bin/c10_DispatchKeySet_test 2024-11-01T16:51:31.2139191Z inflating: build/bin/c10_Scalar_test 2024-11-01T16:51:31.2191605Z inflating: build/bin/c10_StreamGuard_test 2024-11-01T16:51:31.2248187Z inflating: build/bin/c10_SymInt_test 2024-11-01T16:51:31.2307569Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-11-01T16:51:31.2367894Z inflating: build/bin/c10_InlineStreamGuard_test 2024-11-01T16:51:31.2427207Z inflating: build/bin/c10_SizesAndStrides_test 2024-11-01T16:51:31.2503665Z inflating: build/bin/c10_cow_test 2024-11-01T16:51:31.2564086Z inflating: build/bin/c10_Bitset_test 2024-11-01T16:51:31.2617647Z inflating: build/bin/c10_ConstexprCrc_test 2024-11-01T16:51:31.2670216Z inflating: build/bin/c10_DeadlockDetection_test 2024-11-01T16:51:31.2731772Z inflating: build/bin/c10_LeftRight_test 2024-11-01T16:51:31.2786030Z inflating: build/bin/c10_Half_test 2024-11-01T16:51:31.2845738Z inflating: build/bin/c10_Metaprogramming_test 2024-11-01T16:51:31.2903795Z inflating: build/bin/c10_NetworkFlow_test 2024-11-01T16:51:31.2958915Z inflating: build/bin/c10_Synchronized_test 2024-11-01T16:51:31.3020508Z inflating: build/bin/c10_ThreadLocal_test 2024-11-01T16:51:31.3075996Z inflating: build/bin/c10_TypeIndex_test 2024-11-01T16:51:31.3133056Z inflating: build/bin/c10_TypeList_test 2024-11-01T16:51:31.3185457Z inflating: build/bin/c10_TypeTraits_test 2024-11-01T16:51:31.3243276Z inflating: build/bin/c10_accumulate_test 2024-11-01T16:51:31.3303148Z inflating: build/bin/c10_bfloat16_test 2024-11-01T16:51:31.3358263Z inflating: build/bin/c10_bit_cast_test 2024-11-01T16:51:31.3419293Z inflating: build/bin/c10_complex_math_test 2024-11-01T16:51:31.3478144Z inflating: build/bin/c10_complex_test 2024-11-01T16:51:31.3535638Z inflating: build/bin/c10_exception_test 2024-11-01T16:51:31.3589414Z inflating: build/bin/c10_flags_test 2024-11-01T16:51:31.3645129Z inflating: build/bin/c10_generic_math_test 2024-11-01T16:51:31.3817472Z inflating: build/bin/c10_intrusive_ptr_test 2024-11-01T16:51:31.3871161Z inflating: build/bin/c10_irange_test 2024-11-01T16:51:31.3930188Z inflating: build/bin/c10_lazy_test 2024-11-01T16:51:31.3992040Z inflating: build/bin/c10_logging_test 2024-11-01T16:51:31.4071114Z inflating: build/bin/c10_optional_test 2024-11-01T16:51:31.4138915Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-11-01T16:51:31.4198852Z inflating: build/bin/c10_registry_test 2024-11-01T16:51:31.4359898Z inflating: build/bin/c10_small_vector_test 2024-11-01T16:51:31.4415672Z inflating: build/bin/c10_ssize_test 2024-11-01T16:51:31.4470281Z inflating: build/bin/c10_string_util_test 2024-11-01T16:51:31.4535723Z inflating: build/bin/c10_string_view_test 2024-11-01T16:51:31.4589436Z inflating: build/bin/c10_tempfile_test 2024-11-01T16:51:31.4650818Z inflating: build/bin/c10_typeid_test 2024-11-01T16:51:31.4703901Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-11-01T16:51:31.5181074Z inflating: build/bin/protoc-3.13.0.0 2024-11-01T16:51:31.5657288Z inflating: build/bin/protoc 2024-11-01T16:51:31.5716952Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-11-01T16:51:31.5773838Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-11-01T16:51:31.5830322Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-11-01T16:51:31.5887803Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-11-01T16:51:31.5943450Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-11-01T16:51:31.5997967Z inflating: build/bin/c10_cuda_CUDATest 2024-11-01T16:51:31.6054677Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-11-01T16:51:31.6111884Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-11-01T16:51:31.6506379Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-11-01T16:51:31.6916061Z inflating: build/bin/vec_test_all_types_AVX512 2024-11-01T16:51:31.7338831Z inflating: build/bin/vec_test_all_types_AVX2 2024-11-01T16:51:31.7399587Z inflating: build/bin/TCPStoreTest 2024-11-01T16:51:31.7455758Z inflating: build/bin/BackoffTest 2024-11-01T16:51:31.7513721Z inflating: build/bin/HashStoreTest 2024-11-01T16:51:31.7572134Z inflating: build/bin/FileStoreTest 2024-11-01T16:51:31.7586455Z inflating: build/bin/ProcessGroupMPITest 2024-11-01T16:51:31.7590976Z inflating: build/bin/torch_shm_manager 2024-11-01T16:51:31.7649663Z inflating: build/bin/test_edge_op_registration 2024-11-01T16:51:31.7651525Z inflating: build/bin/example_allreduce 2024-11-01T16:51:31.7709377Z inflating: build/bin/Dimname_test 2024-11-01T16:51:31.7787921Z inflating: build/bin/Dict_test 2024-11-01T16:51:31.7856930Z inflating: build/bin/MaybeOwned_test 2024-11-01T16:51:31.7922187Z inflating: build/bin/apply_utils_test 2024-11-01T16:51:31.7982902Z inflating: build/bin/NamedTensor_test 2024-11-01T16:51:31.8047463Z inflating: build/bin/atest 2024-11-01T16:51:31.8117557Z inflating: build/bin/basic 2024-11-01T16:51:31.8177309Z inflating: build/bin/broadcast_test 2024-11-01T16:51:31.8231484Z inflating: build/bin/cpu_allocator_test 2024-11-01T16:51:31.8295701Z inflating: build/bin/cpu_generator_test 2024-11-01T16:51:31.8351904Z inflating: build/bin/cpu_profiling_allocator_test 2024-11-01T16:51:31.8451485Z inflating: build/bin/cpu_rng_test 2024-11-01T16:51:31.8505709Z inflating: build/bin/dispatch_key_set_test 2024-11-01T16:51:31.8560713Z inflating: build/bin/dlconvertor_test 2024-11-01T16:51:31.8620465Z inflating: build/bin/half_test 2024-11-01T16:51:31.8683301Z inflating: build/bin/extension_backend_test 2024-11-01T16:51:31.8737138Z inflating: build/bin/lazy_tensor_test 2024-11-01T16:51:31.8840876Z inflating: build/bin/ivalue_test 2024-11-01T16:51:31.8900614Z inflating: build/bin/math_kernel_test 2024-11-01T16:51:31.8958703Z inflating: build/bin/memory_format_test 2024-11-01T16:51:31.9015947Z inflating: build/bin/memory_overlapping_test 2024-11-01T16:51:31.9072310Z inflating: build/bin/mobile_memory_cleanup 2024-11-01T16:51:31.9128219Z inflating: build/bin/operator_name_test 2024-11-01T16:51:31.9188755Z inflating: build/bin/native_test 2024-11-01T16:51:31.9246815Z inflating: build/bin/operators_test 2024-11-01T16:51:31.9303170Z inflating: build/bin/packedtensoraccessor_test 2024-11-01T16:51:31.9375251Z inflating: build/bin/pow_test 2024-11-01T16:51:31.9428324Z inflating: build/bin/reduce_ops_test 2024-11-01T16:51:31.9492059Z inflating: build/bin/quantized_test 2024-11-01T16:51:31.9545716Z inflating: build/bin/reportMemoryUsage_test 2024-11-01T16:51:31.9608135Z inflating: build/bin/scalar_tensor_test 2024-11-01T16:51:31.9662863Z inflating: build/bin/StorageUtils_test 2024-11-01T16:51:31.9728152Z inflating: build/bin/scalar_test 2024-11-01T16:51:31.9784252Z inflating: build/bin/stride_properties_test 2024-11-01T16:51:31.9867277Z inflating: build/bin/tensor_iterator_test 2024-11-01T16:51:31.9928085Z inflating: build/bin/test_parallel 2024-11-01T16:51:31.9929710Z inflating: build/bin/thread_init_test 2024-11-01T16:51:31.9990719Z inflating: build/bin/type_ptr_test 2024-11-01T16:51:32.0055646Z inflating: build/bin/type_test 2024-11-01T16:51:32.0113189Z inflating: build/bin/undefined_tensor_test 2024-11-01T16:51:32.0114555Z inflating: build/bin/verify_api_visibility 2024-11-01T16:51:32.0170886Z inflating: build/bin/weakref_test 2024-11-01T16:51:32.0245530Z inflating: build/bin/legacy_vmap_test 2024-11-01T16:51:32.0302579Z inflating: build/bin/wrapdim_test 2024-11-01T16:51:32.0366392Z inflating: build/bin/IListRef_test 2024-11-01T16:51:32.0479091Z inflating: build/bin/List_test 2024-11-01T16:51:32.0534913Z inflating: build/bin/xla_tensor_test 2024-11-01T16:51:32.0662912Z inflating: build/bin/kernel_function_legacy_test 2024-11-01T16:51:32.0768015Z inflating: build/bin/kernel_function_test 2024-11-01T16:51:32.0840011Z inflating: build/bin/KernelFunction_test 2024-11-01T16:51:32.0975435Z inflating: build/bin/kernel_lambda_legacy_test 2024-11-01T16:51:32.1041643Z inflating: build/bin/kernel_stackbased_test 2024-11-01T16:51:32.1150805Z inflating: build/bin/kernel_lambda_test 2024-11-01T16:51:32.1254659Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-11-01T16:51:32.1309672Z inflating: build/bin/CppSignature_test 2024-11-01T16:51:32.1370104Z inflating: build/bin/backend_fallback_test 2024-11-01T16:51:32.1422816Z inflating: build/bin/op_allowlist_test 2024-11-01T16:51:32.1491684Z inflating: build/bin/inline_container_test 2024-11-01T16:51:32.1806195Z inflating: build/bin/op_registration_test 2024-11-01T16:51:32.1862639Z inflating: build/bin/cuda_apply_test 2024-11-01T16:51:32.1920299Z inflating: build/bin/cuda_allocator_test 2024-11-01T16:51:32.1982759Z inflating: build/bin/cuda_atomic_ops_test 2024-11-01T16:51:32.2042213Z inflating: build/bin/cuda_caching_host_allocator_test 2024-11-01T16:51:32.2118826Z inflating: build/bin/cuda_complex_math_test 2024-11-01T16:51:32.2171908Z inflating: build/bin/cuda_device_test 2024-11-01T16:51:32.2237110Z inflating: build/bin/cuda_complex_test 2024-11-01T16:51:32.2300778Z inflating: build/bin/cuda_cub_test 2024-11-01T16:51:32.2355954Z inflating: build/bin/cuda_dlconvertor_test 2024-11-01T16:51:32.2411249Z inflating: build/bin/cuda_integer_divider_test 2024-11-01T16:51:32.2471191Z inflating: build/bin/cuda_generator_test 2024-11-01T16:51:32.2542888Z inflating: build/bin/cuda_distributions_test 2024-11-01T16:51:32.2597594Z inflating: build/bin/cuda_half_test 2024-11-01T16:51:32.2650529Z inflating: build/bin/cuda_optional_test 2024-11-01T16:51:32.2704831Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-11-01T16:51:32.2762028Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-11-01T16:51:32.2826744Z inflating: build/bin/cuda_stream_test 2024-11-01T16:51:32.2883011Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-11-01T16:51:32.2937359Z inflating: build/bin/cuda_cudnn_test 2024-11-01T16:51:32.2994888Z inflating: build/bin/cuda_vectorized_test 2024-11-01T16:51:32.3064451Z inflating: build/bin/ProcessGroupGlooTest 2024-11-01T16:51:32.3080252Z inflating: build/bin/tutorial_tensorexpr 2024-11-01T16:51:32.3149538Z inflating: build/bin/ProcessGroupNCCLTest 2024-11-01T16:51:32.3214368Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-11-01T16:51:32.3282865Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-11-01T16:51:32.3339496Z inflating: build/bin/ProcessGroupUCCTest 2024-11-01T16:51:32.3400117Z inflating: build/bin/test_dist_autograd 2024-11-01T16:51:32.3401498Z inflating: build/bin/parallel_benchmark 2024-11-01T16:51:32.3411083Z inflating: build/bin/aot_model_compiler_test 2024-11-01T16:51:32.3486305Z inflating: build/bin/test_cpp_rpc 2024-11-01T16:51:32.3559558Z inflating: build/bin/test_mobile_nnc 2024-11-01T16:51:32.3921886Z inflating: build/bin/test_lazy 2024-11-01T16:51:32.4801264Z inflating: build/bin/test_tensorexpr 2024-11-01T16:51:32.6044634Z inflating: build/bin/test_api 2024-11-01T16:51:32.6654706Z inflating: build/bin/test_jit 2024-11-01T16:51:32.6655236Z creating: .additional_ci_files/ 2024-11-01T16:51:32.6737903Z inflating: .additional_ci_files/test-times.json 2024-11-01T16:51:32.7061642Z inflating: .additional_ci_files/test-class-times.json 2024-11-01T16:51:32.7095282Z ##[group]Run rm artifacts.zip 2024-11-01T16:51:32.7095705Z rm artifacts.zip 2024-11-01T16:51:32.7103142Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:32.7103747Z env: 2024-11-01T16:51:32.7104054Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:32.7104581Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:32.7105126Z ##[endgroup] 2024-11-01T16:51:32.7719058Z ##[group]Run df -H 2024-11-01T16:51:32.7719373Z df -H 2024-11-01T16:51:32.7725195Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:32.7725745Z env: 2024-11-01T16:51:32.7726146Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:32.7726618Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:32.7727126Z ##[endgroup] 2024-11-01T16:51:32.7770160Z Filesystem Size Used Avail Use% Mounted on 2024-11-01T16:51:32.7771019Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-11-01T16:51:32.7771578Z tmpfs 101G 0 101G 0% /dev/shm 2024-11-01T16:51:32.7772432Z tmpfs 41G 697k 41G 1% /run 2024-11-01T16:51:32.7773087Z /dev/nvme0n1p1 161G 47G 115G 30% / 2024-11-01T16:51:32.7773630Z tmpfs 101G 13k 101G 1% /tmp 2024-11-01T16:51:32.7774329Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-11-01T16:51:32.7774909Z tmpfs 21G 0 21G 0% /run/user/0 2024-11-01T16:51:32.7822081Z Prepare all required actions 2024-11-01T16:51:32.7822698Z Getting action download info 2024-11-01T16:51:32.9289549Z ##[group]Run ./.github/actions/download-td-artifacts 2024-11-01T16:51:32.9290201Z with: 2024-11-01T16:51:32.9290478Z env: 2024-11-01T16:51:32.9290754Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:32.9291258Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:32.9291802Z ##[endgroup] 2024-11-01T16:51:32.9332910Z ##[group]Run seemethere/download-artifact-s3@v4 2024-11-01T16:51:32.9333410Z with: 2024-11-01T16:51:32.9333697Z name: td_results 2024-11-01T16:51:32.9334041Z s3-bucket: gha-artifacts 2024-11-01T16:51:32.9334407Z region: us-east-1 2024-11-01T16:51:32.9334729Z env: 2024-11-01T16:51:32.9335016Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:32.9335520Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:32.9336054Z ##[endgroup] 2024-11-01T16:51:33.4871190Z (node:117634) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-11-01T16:51:33.4872044Z 2024-11-01T16:51:33.4872365Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-11-01T16:51:33.4873226Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-11-01T16:51:33.4874347Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-11-01T16:51:33.5645138Z Found 1 objects with prefix pytorch/pytorch/11632514903/td_results/ 2024-11-01T16:51:33.5646348Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-11-01T16:51:33.6221138Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-11-01T16:51:33.6231039Z Artifact download has finished successfully 2024-11-01T16:51:33.6437761Z ##[group]Run mkdir -p .additional_ci_files 2024-11-01T16:51:33.6438276Z mkdir -p .additional_ci_files 2024-11-01T16:51:33.6438879Z mv td_results.json .additional_ci_files/td_results.json 2024-11-01T16:51:33.6445551Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:33.6446056Z env: 2024-11-01T16:51:33.6446332Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:33.6446794Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:33.6447440Z ##[endgroup] 2024-11-01T16:51:33.6537408Z ##[group]Run .github/scripts/parse_ref.py 2024-11-01T16:51:33.6537895Z .github/scripts/parse_ref.py 2024-11-01T16:51:33.6544108Z shell: /usr/bin/bash -e {0} 2024-11-01T16:51:33.6544508Z env: 2024-11-01T16:51:33.6544792Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:33.6545312Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:33.6545872Z ##[endgroup] 2024-11-01T16:51:33.6851776Z Prepare all required actions 2024-11-01T16:51:33.6916554Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-11-01T16:51:33.6917160Z with: 2024-11-01T16:51:33.6917750Z github-token: *** 2024-11-01T16:51:33.6918055Z env: 2024-11-01T16:51:33.6918310Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:33.6918774Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:33.6919279Z ##[endgroup] 2024-11-01T16:51:33.6938323Z ##[group]Run set -eux 2024-11-01T16:51:33.6938661Z set -eux 2024-11-01T16:51:33.6939276Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-11-01T16:51:33.6945896Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:33.6946478Z env: 2024-11-01T16:51:33.6946778Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:33.6947293Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:33.6948066Z GITHUB_TOKEN: *** 2024-11-01T16:51:33.6948396Z ##[endgroup] 2024-11-01T16:51:33.6973403Z + python3 .github/scripts/get_workflow_job_id.py 11632514903 i-0961bf540f093269b 2024-11-01T16:51:35.7983738Z setting job-id=32397035132 2024-11-01T16:51:35.7984827Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-11-01T16:51:35.8232839Z Prepare all required actions 2024-11-01T16:51:35.8233461Z Getting action download info 2024-11-01T16:51:35.9505737Z ##[group]Run ./.github/actions/filter-test-configs 2024-11-01T16:51:35.9506267Z with: 2024-11-01T16:51:35.9506779Z github-token: *** 2024-11-01T16:51:35.9508658Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}]} 2024-11-01T16:51:35.9511086Z job-name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-11-01T16:51:35.9511983Z env: 2024-11-01T16:51:35.9512295Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:35.9512795Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:35.9513352Z ##[endgroup] 2024-11-01T16:51:35.9560789Z ##[group]Run nick-fields/retry@v3.0.0 2024-11-01T16:51:35.9561368Z with: 2024-11-01T16:51:35.9561641Z shell: bash 2024-11-01T16:51:35.9561951Z timeout_minutes: 10 2024-11-01T16:51:35.9562293Z max_attempts: 5 2024-11-01T16:51:35.9562605Z retry_wait_seconds: 30 2024-11-01T16:51:35.9563889Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-11-01T16:51:35.9565294Z polling_interval_seconds: 1 2024-11-01T16:51:35.9565670Z warning_on_retry: true 2024-11-01T16:51:35.9566021Z continue_on_error: false 2024-11-01T16:51:35.9566356Z env: 2024-11-01T16:51:35.9566607Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:35.9567066Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:35.9567757Z GITHUB_TOKEN: *** 2024-11-01T16:51:35.9568066Z ##[endgroup] 2024-11-01T16:51:36.0631470Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-11-01T16:51:36.3387964Z Defaulting to user installation because normal site-packages is not writeable 2024-11-01T16:51:36.3569315Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.9/site-packages (2.27.1) 2024-11-01T16:51:36.3572168Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.9/site-packages (6.0.1) 2024-11-01T16:51:36.3725753Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-11-01T16:51:36.3734503Z Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2.0.12) 2024-11-01T16:51:36.3739339Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-11-01T16:51:36.3746643Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2024.8.30) 2024-11-01T16:51:37.0443325Z Command completed after 1 attempt(s). 2024-11-01T16:51:37.0514801Z ##[group]Run set -x 2024-11-01T16:51:37.0515168Z set -x 2024-11-01T16:51:37.0515497Z  2024-11-01T16:51:37.0516112Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-11-01T16:51:37.0516893Z # in runner workspace 2024-11-01T16:51:37.0517602Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-11-01T16:51:37.0524947Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:37.0525464Z env: 2024-11-01T16:51:37.0525756Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:37.0526236Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:37.0526752Z ##[endgroup] 2024-11-01T16:51:37.0556394Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-11-01T16:51:37.0849465Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-11-01T16:51:37.0850299Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-11-01T16:51:37.0850834Z echo "Job name: ${JOB_NAME}" 2024-11-01T16:51:37.0851269Z  2024-11-01T16:51:37.0852009Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-11-01T16:51:37.0852901Z # in runner workspace 2024-11-01T16:51:37.0853523Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-11-01T16:51:37.0854219Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-11-01T16:51:37.0854707Z  --job-name "${JOB_NAME}" \ 2024-11-01T16:51:37.0856562Z  --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}]}" \ 2024-11-01T16:51:37.0858444Z  --selected-test-configs "" \ 2024-11-01T16:51:37.0858912Z  --pr-number "${PR_NUMBER}" \ 2024-11-01T16:51:37.0859364Z  --tag "${TAG}" \ 2024-11-01T16:51:37.0859779Z  --event-name "${EVENT_NAME}" \ 2024-11-01T16:51:37.0860254Z  --schedule "${SCHEDULE}" \ 2024-11-01T16:51:37.0860704Z  --branch "${HEAD_BRANCH}" 2024-11-01T16:51:37.0867611Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:37.0868190Z env: 2024-11-01T16:51:37.0868517Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:37.0869248Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:37.0870056Z GITHUB_TOKEN: *** 2024-11-01T16:51:37.0870875Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-11-01T16:51:37.0871807Z PR_NUMBER: 138766 2024-11-01T16:51:37.0872142Z TAG: 2024-11-01T16:51:37.0872459Z EVENT_NAME: pull_request 2024-11-01T16:51:37.0872863Z SCHEDULE: 2024-11-01T16:51:37.0873204Z HEAD_BRANCH: 2024-11-01T16:51:37.0873539Z ##[endgroup] 2024-11-01T16:51:37.0899850Z Workflow: pull 2024-11-01T16:51:37.0900917Z Job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-11-01T16:51:37.3363307Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-11-01T16:51:37.5161288Z ##[group]Run echo "Filtered matrix:" 2024-11-01T16:51:37.5161787Z echo "Filtered matrix:" 2024-11-01T16:51:37.5163712Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}]}" 2024-11-01T16:51:37.5165628Z  2024-11-01T16:51:37.5165920Z echo 2024-11-01T16:51:37.5166332Z echo "Is the current job unstable? False" 2024-11-01T16:51:37.5166844Z  2024-11-01T16:51:37.5167195Z echo 2024-11-01T16:51:37.5167584Z echo "Is keep-going label set? False" 2024-11-01T16:51:37.5168075Z  2024-11-01T16:51:37.5168358Z echo 2024-11-01T16:51:37.5168681Z echo "Renabled issues? " 2024-11-01T16:51:37.5174999Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:37.5175554Z env: 2024-11-01T16:51:37.5175845Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:37.5176333Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:37.5177332Z ##[endgroup] 2024-11-01T16:51:37.5203216Z Filtered matrix: 2024-11-01T16:51:37.5205030Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: linux.g4dn.12xlarge.nvidia.gpu}, {config: distributed, shard: 2, num_shards: 3, runner: linux.g4dn.12xlarge.nvidia.gpu}, {config: distributed, shard: 3, num_shards: 3, runner: linux.g4dn.12xlarge.nvidia.gpu}]} 2024-11-01T16:51:37.5206985Z 2024-11-01T16:51:37.5207288Z Is the current job unstable? False 2024-11-01T16:51:37.5207630Z 2024-11-01T16:51:37.5207949Z Is keep-going label set? False 2024-11-01T16:51:37.5208251Z 2024-11-01T16:51:37.5208375Z Renabled issues? 2024-11-01T16:51:37.5272475Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-11-01T16:51:37.5273419Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-11-01T16:51:37.5280055Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:37.5280607Z env: 2024-11-01T16:51:37.5280904Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:37.5281407Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:37.5281954Z JOB_TIMEOUT: 360 2024-11-01T16:51:37.5282285Z ##[endgroup] 2024-11-01T16:51:37.5365947Z ##[group]Run # Fetch aws credential from IMDs 2024-11-01T16:51:37.5366556Z # Fetch aws credential from IMDs 2024-11-01T16:51:37.5367175Z eval "$(python3 .github/scripts/get_aws_session_tokens.py)" 2024-11-01T16:51:37.5367742Z set -x 2024-11-01T16:51:37.5368020Z  2024-11-01T16:51:37.5368372Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-11-01T16:51:37.5368934Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-11-01T16:51:37.5369516Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-11-01T16:51:37.5370042Z  TEST_COMMAND=.ci/onnx/test.sh 2024-11-01T16:51:37.5370462Z else 2024-11-01T16:51:37.5370788Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-11-01T16:51:37.5371216Z fi 2024-11-01T16:51:37.5371484Z  2024-11-01T16:51:37.5371965Z # detached container should get cleaned up by teardown_ec2_linux 2024-11-01T16:51:37.5372798Z # TODO: Stop building test binaries as part of the build phase 2024-11-01T16:51:37.5373506Z # Used for GPU_FLAG since that doesn't play nice 2024-11-01T16:51:37.5374078Z # shellcheck disable=SC2086,SC2090 2024-11-01T16:51:37.5374537Z container_name=$(docker run \ 2024-11-01T16:51:37.5374961Z  ${GPU_FLAG:-} \ 2024-11-01T16:51:37.5375388Z  ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \ 2024-11-01T16:51:37.5375881Z  -e BUILD_ENVIRONMENT \ 2024-11-01T16:51:37.5376285Z  -e PR_NUMBER \ 2024-11-01T16:51:37.5376819Z  -e GITHUB_ACTIONS \ 2024-11-01T16:51:37.5377485Z  -e GITHUB_REPOSITORY \ 2024-11-01T16:51:37.5377969Z  -e GITHUB_WORKFLOW \ 2024-11-01T16:51:37.5378393Z  -e GITHUB_JOB \ 2024-11-01T16:51:37.5378788Z  -e GITHUB_RUN_ID \ 2024-11-01T16:51:37.5379207Z  -e GITHUB_RUN_NUMBER \ 2024-11-01T16:51:37.5379639Z  -e GITHUB_RUN_ATTEMPT \ 2024-11-01T16:51:37.5380081Z  -e JOB_ID \ 2024-11-01T16:51:37.5380449Z  -e JOB_NAME \ 2024-11-01T16:51:37.5380827Z  -e BASE_SHA \ 2024-11-01T16:51:37.5381277Z  -e BRANCH \ 2024-11-01T16:51:37.5381812Z  -e SHA1 \ 2024-11-01T16:51:37.5382194Z  -e AWS_DEFAULT_REGION \ 2024-11-01T16:51:37.5382662Z  -e AWS_ACCESS_KEY_ID \ 2024-11-01T16:51:37.5383128Z  -e AWS_SECRET_ACCESS_KEY \ 2024-11-01T16:51:37.5383608Z  -e AWS_SESSION_TOKEN \ 2024-11-01T16:51:37.5384044Z  -e IN_WHEEL_TEST \ 2024-11-01T16:51:37.5384471Z  -e SHARD_NUMBER \ 2024-11-01T16:51:37.5384884Z  -e TEST_CONFIG \ 2024-11-01T16:51:37.5385297Z  -e NUM_TEST_SHARDS \ 2024-11-01T16:51:37.5385738Z  -e REENABLED_ISSUES \ 2024-11-01T16:51:37.5386204Z  -e CONTINUE_THROUGH_ERROR \ 2024-11-01T16:51:37.5386674Z  -e VERBOSE_TEST_LOGS \ 2024-11-01T16:51:37.5387126Z  -e TEST_SHOWLOCALS \ 2024-11-01T16:51:37.5387571Z  -e NO_TEST_TIMEOUT \ 2024-11-01T16:51:37.5387989Z  -e NO_TD \ 2024-11-01T16:51:37.5388539Z  -e TD_DISTRIBUTED \ 2024-11-01T16:51:37.5388960Z  -e PR_LABELS \ 2024-11-01T16:51:37.5389426Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-11-01T16:51:37.5389953Z  -e SCCACHE_BUCKET \ 2024-11-01T16:51:37.5390392Z  -e SCCACHE_REGION \ 2024-11-01T16:51:37.5390839Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-11-01T16:51:37.5391303Z  -e XLA_CUDA \ 2024-11-01T16:51:37.5391739Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-11-01T16:51:37.5392320Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-11-01T16:51:37.5393023Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-11-01T16:51:37.5393686Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-11-01T16:51:37.5394170Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-11-01T16:51:37.5394634Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-11-01T16:51:37.5395195Z  -e DASHBOARD_TAG \ 2024-11-01T16:51:37.5395589Z  -e IS_A100_RUNNER \ 2024-11-01T16:51:37.5395994Z  -e ARTIFACTS_FILE_SUFFIX \ 2024-11-01T16:51:37.5396529Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-11-01T16:51:37.5397110Z  --security-opt seccomp=unconfined \ 2024-11-01T16:51:37.5397580Z  --cap-add=SYS_PTRACE \ 2024-11-01T16:51:37.5397980Z  --ipc=host \ 2024-11-01T16:51:37.5398338Z  --shm-size="${SHM_SIZE}" \ 2024-11-01T16:51:37.5398740Z  --tty \ 2024-11-01T16:51:37.5399051Z  --detach \ 2024-11-01T16:51:37.5399412Z  --name="${container_name}" \ 2024-11-01T16:51:37.5399828Z  --user jenkins \ 2024-11-01T16:51:37.5400333Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-11-01T16:51:37.5400917Z  -w /var/lib/jenkins/workspace \ 2024-11-01T16:51:37.5401361Z  "${DOCKER_IMAGE}" 2024-11-01T16:51:37.5401719Z ) 2024-11-01T16:51:37.5402120Z # Propagate download.pytorch.org IP to container 2024-11-01T16:51:37.5403116Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-11-01T16:51:37.5404165Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-11-01T16:51:37.5405143Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-11-01T16:51:37.5410881Z shell: /usr/bin/bash -e {0} 2024-11-01T16:51:37.5411237Z env: 2024-11-01T16:51:37.5411504Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:37.5411953Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:37.5412562Z BUILD_ENVIRONMENT: linux-focal-cuda11.8-py3.10-gcc9 2024-11-01T16:51:37.5413062Z PR_NUMBER: 138766 2024-11-01T16:51:37.5413401Z GITHUB_REPOSITORY: pytorch/pytorch 2024-11-01T16:51:37.5413819Z GITHUB_WORKFLOW: pull 2024-11-01T16:51:37.5414155Z GITHUB_JOB: test 2024-11-01T16:51:37.5414452Z GITHUB_RUN_ID: 11632514903 2024-11-01T16:51:37.5414818Z GITHUB_RUN_NUMBER: 261361 2024-11-01T16:51:37.5415175Z GITHUB_RUN_ATTEMPT: 1 2024-11-01T16:51:37.5415503Z JOB_ID: 32397035132 2024-11-01T16:51:37.5416217Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-11-01T16:51:37.5417002Z BRANCH: pull/138766 2024-11-01T16:51:37.5417394Z SHA1: d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:51:37.5417940Z BASE_SHA: 2055d3c8dc8b93c52eda6b9c07d716448d9e28c9 2024-11-01T16:51:37.5418425Z TEST_CONFIG: distributed 2024-11-01T16:51:37.5418769Z SHARD_NUMBER: 3 2024-11-01T16:51:37.5419069Z NUM_TEST_SHARDS: 3 2024-11-01T16:51:37.5419376Z REENABLED_ISSUES: 2024-11-01T16:51:37.5419708Z CONTINUE_THROUGH_ERROR: False 2024-11-01T16:51:37.5420097Z VERBOSE_TEST_LOGS: False 2024-11-01T16:51:37.5420451Z TEST_SHOWLOCALS: False 2024-11-01T16:51:37.5420793Z NO_TEST_TIMEOUT: False 2024-11-01T16:51:37.5421208Z NO_TD: False 2024-11-01T16:51:37.5421691Z TD_DISTRIBUTED: False 2024-11-01T16:51:37.5422273Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-11-01T16:51:37.5422841Z SCCACHE_REGION: us-east-1 2024-11-01T16:51:37.5423260Z SCCACHE_S3_KEY_PREFIX: pull 2024-11-01T16:51:37.5423651Z SHM_SIZE: 2g 2024-11-01T16:51:37.5424802Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:51:37.5426067Z XLA_CUDA: 2024-11-01T16:51:37.5426627Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-11-01T16:51:37.5427348Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-11-01T16:51:37.5427838Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-11-01T16:51:37.5428287Z DASHBOARD_TAG: 2024-11-01T16:51:37.5428873Z HUGGING_FACE_HUB_TOKEN: *** 2024-11-01T16:51:37.5429510Z SCRIBE_GRAPHQL_ACCESS_TOKEN: *** 2024-11-01T16:51:37.5429959Z IS_A100_RUNNER: 0 2024-11-01T16:51:37.5430768Z ARTIFACTS_FILE_SUFFIX: test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_32397035132 2024-11-01T16:51:37.5431592Z ##[endgroup] 2024-11-01T16:51:37.5659947Z Traceback (most recent call last): 2024-11-01T16:51:37.5661671Z File "/home/ec2-user/actions-runner/_work/pytorch/pytorch/.github/scripts/get_aws_session_tokens.py", line 2, in 2024-11-01T16:51:37.5662741Z import boto3 # type: ignore[import] 2024-11-01T16:51:37.5663367Z ModuleNotFoundError: No module named 'boto3' 2024-11-01T16:51:37.5691671Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-11-01T16:51:37.5692413Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *onnx* ]] 2024-11-01T16:51:37.5692965Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-11-01T16:51:37.5699312Z +++ nproc --ignore=2 2024-11-01T16:51:37.5718052Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=46 -e SCCACHE_BUCKET -e SCCACHE_REGION -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e IS_A100_RUNNER -e ARTIFACTS_FILE_SUFFIX --env-file=/tmp/github_env_11632514903 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:51:48.7211982Z + container_name=1b47f5de842509be1b3efe91f0254870f158ff4faecdf9723b7904a3b4486cbe 2024-11-01T16:51:48.7216333Z + grep download.pytorch.org /etc/hosts 2024-11-01T16:51:48.7219862Z + docker exec -i 1b47f5de842509be1b3efe91f0254870f158ff4faecdf9723b7904a3b4486cbe sudo bash -c '/bin/cat >> /etc/hosts' 2024-11-01T16:51:48.8816107Z + echo DOCKER_CONTAINER_ID=1b47f5de842509be1b3efe91f0254870f158ff4faecdf9723b7904a3b4486cbe 2024-11-01T16:51:48.8820585Z ++ echo dist/torch-2.6.0a0+gitd1aa4ef-cp310-cp310-linux_x86_64.whl 2024-11-01T16:51:48.8823650Z + docker exec -t 1b47f5de842509be1b3efe91f0254870f158ff4faecdf9723b7904a3b4486cbe sh -c 'pip install dist/torch-2.6.0a0+gitd1aa4ef-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-11-01T16:51:49.3144533Z Processing ./dist/torch-2.6.0a0+gitd1aa4ef-cp310-cp310-linux_x86_64.whl (from torch==2.6.0a0+gitd1aa4ef) 2024-11-01T16:51:50.2145530Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (3.16.1) 2024-11-01T16:51:50.2148462Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (4.12.2) 2024-11-01T16:51:50.2150852Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (2.8.8) 2024-11-01T16:51:50.2154380Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (3.1.4) 2024-11-01T16:51:50.2158102Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (2024.10.0) 2024-11-01T16:51:50.2164683Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (1.13.1) 2024-11-01T16:51:50.2186581Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (1.3.0) 2024-11-01T16:51:50.2207321Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (3.3.0) 2024-11-01T16:51:50.2227385Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (1.22.4) 2024-11-01T16:51:50.2705311Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (3.0.2) 2024-11-01T16:51:50.6740462Z Installing collected packages: torch 2024-11-01T16:52:02.1761716Z Successfully installed torch-2.6.0a0+gitd1aa4ef 2024-11-01T16:52:02.2536185Z + export TERM=vt100 2024-11-01T16:52:02.2536626Z + TERM=vt100 2024-11-01T16:52:02.2540018Z ++ dirname .ci/pytorch/test.sh 2024-11-01T16:52:02.2543158Z + source .ci/pytorch/common.sh 2024-11-01T16:52:02.2546463Z +++ dirname .ci/pytorch/common.sh 2024-11-01T16:52:02.2554660Z ++ source .ci/pytorch/common_utils.sh 2024-11-01T16:52:02.2556571Z +++ declare -f -t trap_add 2024-11-01T16:52:02.2562663Z ++ set -ex 2024-11-01T16:52:02.2563498Z ++ [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-11-01T16:52:02.2564076Z ++ BUILD_TEST_LIBTORCH=0 2024-11-01T16:52:02.2564633Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-11-01T16:52:02.2567576Z ++ stat -c %u /var/lib/jenkins/workspace 2024-11-01T16:52:02.2582941Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-11-01T16:52:02.2583445Z + trap_add cleanup_workspace EXIT 2024-11-01T16:52:02.2583916Z + trap_add_cmd=cleanup_workspace 2024-11-01T16:52:02.2584333Z + shift 2024-11-01T16:52:02.2584657Z + for trap_add_name in "$@" 2024-11-01T16:52:02.2590338Z +++ trap -p EXIT 2024-11-01T16:52:02.2592937Z ++ eval 'extract_trap_cmd ' 2024-11-01T16:52:02.2593332Z +++ extract_trap_cmd 2024-11-01T16:52:02.2593726Z +++ printf '%s\n' '' 2024-11-01T16:52:02.2594151Z ++ printf '%s\n' cleanup_workspace 2024-11-01T16:52:02.2595313Z + trap -- ' 2024-11-01T16:52:02.2595650Z cleanup_workspace' EXIT 2024-11-01T16:52:02.2596167Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-11-01T16:52:02.7855412Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-11-01T16:52:02.7870155Z + echo 'Environment variables:' 2024-11-01T16:52:02.7870624Z Environment variables: 2024-11-01T16:52:02.7870987Z + env 2024-11-01T16:52:02.7879552Z INSTALLED_DB=yes 2024-11-01T16:52:02.7879978Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-11-01T16:52:02.7880409Z NVIDIA_VISIBLE_DEVICES=all 2024-11-01T16:52:02.7880847Z NV_NVML_DEV_VERSION=11.8.86-1 2024-11-01T16:52:02.7881570Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-11-01T16:52:02.7882257Z CONTINUE_THROUGH_ERROR=False 2024-11-01T16:52:02.7883079Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-11-01T16:52:02.7883689Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-11-01T16:52:02.7884321Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-11-01T16:52:02.7884877Z HOSTNAME=1b47f5de8425 2024-11-01T16:52:02.7885935Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_cbdfb947-5e2a-42b4-94f0-98d283b49869 2024-11-01T16:52:02.7886928Z GITHUB_ACTION=__self 2024-11-01T16:52:02.7887315Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-11-01T16:52:02.7890070Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-11-01T16:52:02.7892955Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-11-01T16:52:02.7893553Z NV_NVTX_VERSION=11.8.86-1 2024-11-01T16:52:02.7893930Z GITHUB_RUN_NUMBER=261361 2024-11-01T16:52:02.7894318Z TEST_CONFIG=distributed 2024-11-01T16:52:02.7894717Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-11-01T16:52:02.7895315Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-11-01T16:52:02.7895780Z IS_A100_RUNNER=0 2024-11-01T16:52:02.7896161Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-11-01T16:52:02.7896662Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-11-01T16:52:02.7897473Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2024-11-01T16:52:02.7897943Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-11-01T16:52:02.7898369Z GITHUB_TRIGGERING_ACTOR=c00w 2024-11-01T16:52:02.7898869Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-11-01T16:52:02.7899412Z GITHUB_REF_TYPE=branch 2024-11-01T16:52:02.7899791Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-11-01T16:52:02.7900202Z NCCL_VERSION=2.16.2-1 2024-11-01T16:52:02.7900653Z BASE_SHA=2055d3c8dc8b93c52eda6b9c07d716448d9e28c9 2024-11-01T16:52:02.7901270Z XLA_CUDA= 2024-11-01T16:52:02.7901771Z HUGGING_FACE_HUB_TOKEN=*** 2024-11-01T16:52:02.7903764Z *** 2024-11-01T16:52:02.7904084Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-11-01T16:52:02.7904528Z GITHUB_REPOSITORY_ID=65600975 2024-11-01T16:52:02.7904942Z GITHUB_ACTIONS=true 2024-11-01T16:52:02.7905308Z NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:52:02.7905913Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-11-01T16:52:02.7906542Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-11-01T16:52:02.7907082Z SHA1=d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:52:02.7907688Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-11-01T16:52:02.7908243Z GITHUB_SHA=368da7510f5d755034681777a4e5ae6d33c07b38 2024-11-01T16:52:02.7909084Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/138766/merge 2024-11-01T16:52:02.7909832Z UCC_HOME=/usr 2024-11-01T16:52:02.7910204Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-11-01T16:52:02.7910662Z VERBOSE_TEST_LOGS=False 2024-11-01T16:52:02.7911042Z NVIDIA_PRODUCT_NAME=CUDA 2024-11-01T16:52:02.7911580Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-11-01T16:52:02.7912123Z GITHUB_REF=refs/pull/138766/merge 2024-11-01T16:52:02.7912593Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-11-01T16:52:02.7913109Z SHARD_NUMBER=3 2024-11-01T16:52:02.7913442Z GITHUB_REF_PROTECTED=false 2024-11-01T16:52:02.7913822Z HOME=/var/lib/jenkins 2024-11-01T16:52:02.7914215Z GITHUB_API_URL=https://api.github.com 2024-11-01T16:52:02.7914700Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-11-01T16:52:02.7915215Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-11-01T16:52:02.7915754Z SCCACHE_S3_KEY_PREFIX=pull 2024-11-01T16:52:02.7916132Z CUDA_VERSION=11.8.0 2024-11-01T16:52:02.7916608Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-11-01T16:52:02.7917106Z NUM_TEST_SHARDS=3 2024-11-01T16:52:02.7917418Z UCX_HOME=/usr 2024-11-01T16:52:02.7918014Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-11-01T16:52:02.7919324Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_cbdfb947-5e2a-42b4-94f0-98d283b49869 2024-11-01T16:52:02.7920959Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-11-01T16:52:02.7922470Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_cbdfb947-5e2a-42b4-94f0-98d283b49869 2024-11-01T16:52:02.7923822Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-11-01T16:52:02.7924585Z GITHUB_EVENT_NAME=pull_request 2024-11-01T16:52:02.7924992Z DASHBOARD_TAG= 2024-11-01T16:52:02.7925321Z GITHUB_RUN_ID=11632514903 2024-11-01T16:52:02.7925836Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-11-01T16:52:02.7926445Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-11-01T16:52:02.7927747Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_cbdfb947-5e2a-42b4-94f0-98d283b49869 2024-11-01T16:52:02.7928758Z GITHUB_ACTOR=c00w 2024-11-01T16:52:02.7929145Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-11-01T16:52:02.7929565Z PR_NUMBER=138766 2024-11-01T16:52:02.7929896Z GITHUB_RUN_ATTEMPT=1 2024-11-01T16:52:02.7930258Z ANACONDA_PYTHON_VERSION=3.10 2024-11-01T16:52:02.7930737Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-11-01T16:52:02.7931251Z TERM=vt100 2024-11-01T16:52:02.7931604Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-11-01T16:52:02.7932055Z INSTALLED_VISION=yes 2024-11-01T16:52:02.7932398Z BRANCH=pull/138766 2024-11-01T16:52:02.7932767Z SCCACHE_REGION=us-east-1 2024-11-01T16:52:02.7933263Z OPENSSL_ROOT_DIR=/opt/openssl 2024-11-01T16:52:02.7933695Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-11-01T16:52:02.7934150Z CUDA_PATH=/usr/local/cuda 2024-11-01T16:52:02.7935005Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-11-01T16:52:02.7936028Z GITHUB_SERVER_URL=https://github.com 2024-11-01T16:52:02.7936575Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-11-01T16:52:02.7937102Z REENABLED_ISSUES= 2024-11-01T16:52:02.7937418Z SHLVL=1 2024-11-01T16:52:02.7937690Z MAX_JOBS=46 2024-11-01T16:52:02.7938013Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-11-01T16:52:02.7938393Z NVARCH=x86_64 2024-11-01T16:52:02.7938703Z GITHUB_ACTOR_ID=486199 2024-11-01T16:52:02.7939192Z GITHUB_WORKFLOW_SHA=368da7510f5d755034681777a4e5ae6d33c07b38 2024-11-01T16:52:02.7939766Z GITHUB_REF_NAME=138766/merge 2024-11-01T16:52:02.7940244Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-11-01T16:52:02.7941098Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-11-01T16:52:02.7941933Z GITHUB_JOB=test 2024-11-01T16:52:02.7942394Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-11-01T16:52:02.7943046Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-11-01T16:52:02.7943645Z NO_TEST_TIMEOUT=False 2024-11-01T16:52:02.7943996Z TD_DISTRIBUTED=False 2024-11-01T16:52:02.7944425Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-11-01T16:52:02.7944926Z GITHUB_REPOSITORY=pytorch/pytorch 2024-11-01T16:52:02.7945404Z NV_NVPROF_VERSION=11.8.87-1 2024-11-01T16:52:02.7945809Z GITHUB_RETENTION_DAYS=90 2024-11-01T16:52:02.7946185Z OPENSSL_DIR=/opt/openssl 2024-11-01T16:52:02.7946575Z GITHUB_ACTION_REPOSITORY= 2024-11-01T16:52:02.7947840Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-11-01T16:52:02.7949170Z GITHUB_BASE_REF=gh/c00w/2/base 2024-11-01T16:52:02.7950057Z ARTIFACTS_FILE_SUFFIX=test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_32397035132 2024-11-01T16:52:02.7950887Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-11-01T16:52:02.7951291Z CI=true 2024-11-01T16:52:02.7951641Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-11-01T16:52:02.7952109Z GITHUB_REPOSITORY_OWNER=pytorch 2024-11-01T16:52:02.7952528Z JOB_ID=32397035132 2024-11-01T16:52:02.7952873Z INSTALLED_PROTOBUF=yes 2024-11-01T16:52:02.7953249Z GITHUB_HEAD_REF=gh/c00w/2/head 2024-11-01T16:52:02.7953766Z GITHUB_ACTION_REF= 2024-11-01T16:52:02.7954355Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-11-01T16:52:02.7954880Z TEST_SHOWLOCALS=False 2024-11-01T16:52:02.7955237Z GITHUB_WORKFLOW=pull 2024-11-01T16:52:02.7955596Z DEBIAN_FRONTEND=noninteractive 2024-11-01T16:52:02.7956678Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_cbdfb947-5e2a-42b4-94f0-98d283b49869 2024-11-01T16:52:02.7957640Z NO_TD=False 2024-11-01T16:52:02.7957957Z SKIP_SCCACHE_INITIALIZATION=1 2024-11-01T16:52:02.7958352Z _=/usr/bin/env 2024-11-01T16:52:02.7958972Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-11-01T16:52:02.8143572Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-11-01T16:52:02.8144667Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-11-01T16:52:02.8145945Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-11-01T16:52:02.8147059Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-11-01T16:52:02.8147789Z + BUILD_DIR=build 2024-11-01T16:52:02.8148155Z + BUILD_RENAMED_DIR=build_renamed 2024-11-01T16:52:02.8148583Z + BUILD_BIN_DIR=build/bin 2024-11-01T16:52:02.8148963Z + SHARD_NUMBER=3 2024-11-01T16:52:02.8149298Z + NUM_TEST_SHARDS=3 2024-11-01T16:52:02.8149645Z + export VALGRIND=ON 2024-11-01T16:52:02.8149993Z + VALGRIND=ON 2024-11-01T16:52:02.8150475Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *clang9* ]] 2024-11-01T16:52:02.8151021Z + [[ 0 == \1 ]] 2024-11-01T16:52:02.8151344Z + [[ False == \1 ]] 2024-11-01T16:52:02.8151848Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *bazel* ]] 2024-11-01T16:52:02.8152431Z ++ realpath build/custom_test_artifacts 2024-11-01T16:52:02.8156093Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-11-01T16:52:02.8156878Z + [[ -n '' ]] 2024-11-01T16:52:02.8157263Z + echo 'Environment variables' 2024-11-01T16:52:02.8157690Z Environment variables 2024-11-01T16:52:02.8158030Z + env 2024-11-01T16:52:02.8164920Z INSTALLED_DB=yes 2024-11-01T16:52:02.8165289Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-11-01T16:52:02.8165764Z NVIDIA_VISIBLE_DEVICES=all 2024-11-01T16:52:02.8166186Z NV_NVML_DEV_VERSION=11.8.86-1 2024-11-01T16:52:02.8167060Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-11-01T16:52:02.8167738Z CONTINUE_THROUGH_ERROR=False 2024-11-01T16:52:02.8168312Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-11-01T16:52:02.8168902Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-11-01T16:52:02.8169547Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-11-01T16:52:02.8170100Z HOSTNAME=1b47f5de8425 2024-11-01T16:52:02.8171135Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_cbdfb947-5e2a-42b4-94f0-98d283b49869 2024-11-01T16:52:02.8172128Z GITHUB_ACTION=__self 2024-11-01T16:52:02.8172516Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-11-01T16:52:02.8175134Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-11-01T16:52:02.8178141Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-11-01T16:52:02.8178745Z NV_NVTX_VERSION=11.8.86-1 2024-11-01T16:52:02.8179141Z GITHUB_RUN_NUMBER=261361 2024-11-01T16:52:02.8179530Z TEST_CONFIG=distributed 2024-11-01T16:52:02.8179928Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-11-01T16:52:02.8180459Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-11-01T16:52:02.8180922Z IS_A100_RUNNER=0 2024-11-01T16:52:02.8181393Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-11-01T16:52:02.8181901Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-11-01T16:52:02.8182712Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2024-11-01T16:52:02.8183184Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-11-01T16:52:02.8183600Z GITHUB_TRIGGERING_ACTOR=c00w 2024-11-01T16:52:02.8184280Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-11-01T16:52:02.8184819Z GITHUB_REF_TYPE=branch 2024-11-01T16:52:02.8185202Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-11-01T16:52:02.8185632Z NCCL_VERSION=2.16.2-1 2024-11-01T16:52:02.8186063Z BASE_SHA=2055d3c8dc8b93c52eda6b9c07d716448d9e28c9 2024-11-01T16:52:02.8186573Z XLA_CUDA= 2024-11-01T16:52:02.8187090Z HUGGING_FACE_HUB_TOKEN=*** 2024-11-01T16:52:02.8187524Z *** 2024-11-01T16:52:02.8187826Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-11-01T16:52:02.8188277Z GITHUB_REPOSITORY_ID=65600975 2024-11-01T16:52:02.8188670Z GITHUB_ACTIONS=true 2024-11-01T16:52:02.8189037Z NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:52:02.8189601Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-11-01T16:52:02.8190268Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-11-01T16:52:02.8190815Z SHA1=d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:52:02.8191519Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-11-01T16:52:02.8192085Z GITHUB_SHA=368da7510f5d755034681777a4e5ae6d33c07b38 2024-11-01T16:52:02.8193018Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/138766/merge 2024-11-01T16:52:02.8193749Z UCC_HOME=/usr 2024-11-01T16:52:02.8194115Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-11-01T16:52:02.8194563Z VERBOSE_TEST_LOGS=False 2024-11-01T16:52:02.8194935Z NVIDIA_PRODUCT_NAME=CUDA 2024-11-01T16:52:02.8195431Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-11-01T16:52:02.8195957Z GITHUB_REF=refs/pull/138766/merge 2024-11-01T16:52:02.8196414Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-11-01T16:52:02.8196819Z SHARD_NUMBER=3 2024-11-01T16:52:02.8197147Z GITHUB_REF_PROTECTED=false 2024-11-01T16:52:02.8197512Z HOME=/var/lib/jenkins 2024-11-01T16:52:02.8197905Z GITHUB_API_URL=https://api.github.com 2024-11-01T16:52:02.8198388Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-11-01T16:52:02.8198923Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-11-01T16:52:02.8199458Z SCCACHE_S3_KEY_PREFIX=pull 2024-11-01T16:52:02.8199845Z CUDA_VERSION=11.8.0 2024-11-01T16:52:02.8200300Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-11-01T16:52:02.8200789Z NUM_TEST_SHARDS=3 2024-11-01T16:52:02.8201106Z UCX_HOME=/usr 2024-11-01T16:52:02.8201687Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-11-01T16:52:02.8202979Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_cbdfb947-5e2a-42b4-94f0-98d283b49869 2024-11-01T16:52:02.8204466Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-11-01T16:52:02.8205956Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_cbdfb947-5e2a-42b4-94f0-98d283b49869 2024-11-01T16:52:02.8207296Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-11-01T16:52:02.8208061Z GITHUB_EVENT_NAME=pull_request 2024-11-01T16:52:02.8208466Z DASHBOARD_TAG= 2024-11-01T16:52:02.8208789Z GITHUB_RUN_ID=11632514903 2024-11-01T16:52:02.8209293Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-11-01T16:52:02.8209900Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-11-01T16:52:02.8211094Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_cbdfb947-5e2a-42b4-94f0-98d283b49869 2024-11-01T16:52:02.8212114Z GITHUB_ACTOR=c00w 2024-11-01T16:52:02.8212486Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-11-01T16:52:02.8212906Z PR_NUMBER=138766 2024-11-01T16:52:02.8213222Z GITHUB_RUN_ATTEMPT=1 2024-11-01T16:52:02.8213559Z VALGRIND=ON 2024-11-01T16:52:02.8213873Z ANACONDA_PYTHON_VERSION=3.10 2024-11-01T16:52:02.8214363Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-11-01T16:52:02.8214877Z TERM=vt100 2024-11-01T16:52:02.8215217Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-11-01T16:52:02.8215670Z INSTALLED_VISION=yes 2024-11-01T16:52:02.8216013Z BRANCH=pull/138766 2024-11-01T16:52:02.8216398Z SCCACHE_REGION=us-east-1 2024-11-01T16:52:02.8216784Z OPENSSL_ROOT_DIR=/opt/openssl 2024-11-01T16:52:02.8217301Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-11-01T16:52:02.8217775Z CUDA_PATH=/usr/local/cuda 2024-11-01T16:52:02.8218658Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-11-01T16:52:02.8219536Z GITHUB_SERVER_URL=https://github.com 2024-11-01T16:52:02.8220088Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-11-01T16:52:02.8220605Z REENABLED_ISSUES= 2024-11-01T16:52:02.8220927Z SHLVL=1 2024-11-01T16:52:02.8221316Z MAX_JOBS=46 2024-11-01T16:52:02.8221840Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-11-01T16:52:02.8222239Z NVARCH=x86_64 2024-11-01T16:52:02.8222551Z GITHUB_ACTOR_ID=486199 2024-11-01T16:52:02.8223063Z GITHUB_WORKFLOW_SHA=368da7510f5d755034681777a4e5ae6d33c07b38 2024-11-01T16:52:02.8223660Z GITHUB_REF_NAME=138766/merge 2024-11-01T16:52:02.8224160Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-11-01T16:52:02.8225045Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-11-01T16:52:02.8225731Z GITHUB_JOB=test 2024-11-01T16:52:02.8226180Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-11-01T16:52:02.8226832Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-11-01T16:52:02.8227436Z NO_TEST_TIMEOUT=False 2024-11-01T16:52:02.8227799Z TD_DISTRIBUTED=False 2024-11-01T16:52:02.8228223Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-11-01T16:52:02.8228709Z GITHUB_REPOSITORY=pytorch/pytorch 2024-11-01T16:52:02.8229193Z NV_NVPROF_VERSION=11.8.87-1 2024-11-01T16:52:02.8229603Z GITHUB_RETENTION_DAYS=90 2024-11-01T16:52:02.8229996Z OPENSSL_DIR=/opt/openssl 2024-11-01T16:52:02.8230391Z GITHUB_ACTION_REPOSITORY= 2024-11-01T16:52:02.8231644Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-11-01T16:52:02.8233077Z GITHUB_BASE_REF=gh/c00w/2/base 2024-11-01T16:52:02.8234095Z ARTIFACTS_FILE_SUFFIX=test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_32397035132 2024-11-01T16:52:02.8235022Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-11-01T16:52:02.8235417Z CI=true 2024-11-01T16:52:02.8235743Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-11-01T16:52:02.8236174Z GITHUB_REPOSITORY_OWNER=pytorch 2024-11-01T16:52:02.8236567Z JOB_ID=32397035132 2024-11-01T16:52:02.8236892Z INSTALLED_PROTOBUF=yes 2024-11-01T16:52:02.8237250Z GITHUB_HEAD_REF=gh/c00w/2/head 2024-11-01T16:52:02.8237635Z GITHUB_ACTION_REF= 2024-11-01T16:52:02.8238093Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-11-01T16:52:02.8238600Z TEST_SHOWLOCALS=False 2024-11-01T16:52:02.8238944Z GITHUB_WORKFLOW=pull 2024-11-01T16:52:02.8239300Z DEBIAN_FRONTEND=noninteractive 2024-11-01T16:52:02.8240345Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_cbdfb947-5e2a-42b4-94f0-98d283b49869 2024-11-01T16:52:02.8241264Z NO_TD=False 2024-11-01T16:52:02.8241573Z SKIP_SCCACHE_INITIALIZATION=1 2024-11-01T16:52:02.8241955Z _=/usr/bin/env 2024-11-01T16:52:02.8242315Z + echo 'Testing pytorch' 2024-11-01T16:52:02.8242673Z Testing pytorch 2024-11-01T16:52:02.8243001Z + export LANG=C.UTF-8 2024-11-01T16:52:02.8243352Z + LANG=C.UTF-8 2024-11-01T16:52:02.8243656Z + PR_NUMBER=138766 2024-11-01T16:52:02.8244008Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-11-01T16:52:02.8244493Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-11-01T16:52:02.8245100Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-11-01T16:52:02.8245622Z + [[ distributed == \s\l\o\w ]] 2024-11-01T16:52:02.8246239Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *slow-gradcheck* ]] 2024-11-01T16:52:02.8247028Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-11-01T16:52:02.8247568Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-11-01T16:52:02.8248054Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-11-01T16:52:02.8248473Z + [[ distributed == *crossref* ]] 2024-11-01T16:52:02.8248995Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-11-01T16:52:02.8249609Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *xpu* ]] 2024-11-01T16:52:02.8250232Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-11-01T16:52:02.8250887Z + pip_install --user ninja==1.10.2 2024-11-01T16:52:02.8251869Z + pip install --progress-bar off --user ninja==1.10.2 2024-11-01T16:52:03.6997637Z Collecting ninja==1.10.2 2024-11-01T16:52:03.7203989Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-11-01T16:52:03.7305143Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-11-01T16:52:04.1206945Z Installing collected packages: ninja 2024-11-01T16:52:04.1288479Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-11-01T16:52:04.1290018Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-11-01T16:52:04.1337570Z Successfully installed ninja-1.10.2 2024-11-01T16:52:04.2012214Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-11-01T16:52:04.2014887Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-11-01T16:52:04.2016684Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-11-01T16:52:04.2017228Z + install_tlparse 2024-11-01T16:52:04.2017648Z + pip_install --user tlparse==0.3.25 2024-11-01T16:52:04.2018271Z + pip install --progress-bar off --user tlparse==0.3.25 2024-11-01T16:52:04.6372443Z Collecting tlparse==0.3.25 2024-11-01T16:52:04.6563585Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-11-01T16:52:04.6660364Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-11-01T16:52:05.0833153Z Installing collected packages: tlparse 2024-11-01T16:52:05.1214348Z Successfully installed tlparse-0.3.25 2024-11-01T16:52:05.1904543Z ++ python -m site --user-base 2024-11-01T16:52:05.2174970Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-11-01T16:52:05.2177275Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *asan* ]] 2024-11-01T16:52:05.2178003Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-debug* ]] 2024-11-01T16:52:05.2178724Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-11-01T16:52:05.2179788Z + echo 'We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass' 2024-11-01T16:52:05.2181189Z We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass 2024-11-01T16:52:05.2181995Z + cd test 2024-11-01T16:52:05.2182643Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-11-01T16:52:07.0404425Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-11-01T16:52:07.0405098Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-11-01T16:52:07.0406113Z + DYNAMO_BENCHMARK_FLAGS=() 2024-11-01T16:52:07.0407456Z + [[ distributed == *pr_time_benchmarks* ]] 2024-11-01T16:52:07.0408032Z + [[ distributed == *dynamo_eager* ]] 2024-11-01T16:52:07.0408515Z + [[ distributed == *aot_eager* ]] 2024-11-01T16:52:07.0408973Z + [[ distributed == *aot_inductor* ]] 2024-11-01T16:52:07.0409433Z + [[ distributed == *inductor* ]] 2024-11-01T16:52:07.0409870Z + [[ distributed == *dynamic* ]] 2024-11-01T16:52:07.0410282Z + [[ distributed == *cpu* ]] 2024-11-01T16:52:07.0410937Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-11-01T16:52:07.0448354Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-11-01T16:52:07.0449227Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-bazel-* ]] 2024-11-01T16:52:07.0449821Z + cd test 2024-11-01T16:52:07.0450337Z + python -c 'import torch; print(torch.__config__.show())' 2024-11-01T16:52:09.0643833Z PyTorch built with: 2024-11-01T16:52:09.0644994Z - GCC 9.4 2024-11-01T16:52:09.0645391Z - C++ Version: 201703 2024-11-01T16:52:09.0646422Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-11-01T16:52:09.0647716Z - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-11-01T16:52:09.0648510Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-11-01T16:52:09.0649102Z - LAPACK is enabled (usually provided by MKL) 2024-11-01T16:52:09.0649658Z - NNPACK is enabled 2024-11-01T16:52:09.0650102Z - CPU capability usage: AVX512 2024-11-01T16:52:09.0650580Z - CUDA Runtime 11.8 2024-11-01T16:52:09.0651167Z - NVCC architecture flags: -gencode;arch=compute_75,code=sm_75 2024-11-01T16:52:09.0651801Z - CuDNN 90.1 2024-11-01T16:52:09.0652156Z - Magma 2.6.1 2024-11-01T16:52:09.0661136Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.6.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-11-01T16:52:09.0669677Z 2024-11-01T16:52:09.4942469Z + cd test 2024-11-01T16:52:09.4943329Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-11-01T16:52:11.0284280Z ATen/Parallel: 2024-11-01T16:52:11.0284781Z at::get_num_threads() : 24 2024-11-01T16:52:11.0285431Z at::get_num_interop_threads() : 24 2024-11-01T16:52:11.0285950Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-11-01T16:52:11.0286404Z omp_get_max_threads() : 24 2024-11-01T16:52:11.0287620Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-11-01T16:52:11.0288596Z mkl_get_max_threads() : 24 2024-11-01T16:52:11.0289351Z Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-11-01T16:52:11.0290110Z std::thread::hardware_concurrency() : 48 2024-11-01T16:52:11.0290626Z Environment variables: 2024-11-01T16:52:11.0291024Z OMP_NUM_THREADS : [not set] 2024-11-01T16:52:11.0291442Z MKL_NUM_THREADS : [not set] 2024-11-01T16:52:11.0291868Z ATen parallel backend: OpenMP 2024-11-01T16:52:11.0292171Z 2024-11-01T16:52:11.2810152Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-11-01T16:52:11.2811384Z + [[ distributed == *backward* ]] 2024-11-01T16:52:11.2811854Z + [[ distributed == *xla* ]] 2024-11-01T16:52:11.2812287Z + [[ distributed == *executorch* ]] 2024-11-01T16:52:11.2812772Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-11-01T16:52:11.2813499Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-11-01T16:52:11.2814096Z + [[ distributed == distributed ]] 2024-11-01T16:52:11.2814540Z + test_distributed 2024-11-01T16:52:11.2814998Z + echo 'Testing distributed python tests' 2024-11-01T16:52:11.2815501Z Testing distributed python tests 2024-11-01T16:52:11.2816222Z + python test/run_test.py --distributed-tests --shard 3 3 --verbose 2024-11-01T16:52:11.3953150Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-11-01T16:52:11.3977447Z import pkg_resources 2024-11-01T16:52:17.0937178Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-11-01T16:52:17.1377223Z Ignoring disabled issues: [''] 2024-11-01T16:52:17.1522233Z Found test times from artifacts 2024-11-01T16:52:17.2092951Z Found test times from artifacts 2024-11-01T16:52:17.2107401Z Running 25% of tests based on TD 2024-11-01T16:52:17.2257862Z Running parallel tests on 2 processes 2024-11-01T16:52:17.2258421Z Name: tests to run (est. time: 7.88min) 2024-11-01T16:52:17.2258889Z Serial tests (19): 2024-11-01T16:52:17.2259470Z distributed/test_inductor_collectives 1/1 2024-11-01T16:52:17.2260031Z distributed/_tensor/test_attention 1/1 2024-11-01T16:52:17.2260896Z distributed/checkpoint/test_save_load_api 1/1 2024-11-01T16:52:17.2261643Z distributed/checkpoint/e2e/test_fsdp_ep 1/1 2024-11-01T16:52:17.2262250Z distributed/test_c10d_functional_native 1/1 2024-11-01T16:52:17.2262927Z distributed/_tensor/experimental/test_register_sharding 1/1 2024-11-01T16:52:17.2263584Z distributed/_tools/test_memory_tracker 1/1 2024-11-01T16:52:17.2264197Z distributed/_tensor/test_xla_integration 1/1 2024-11-01T16:52:17.2264849Z distributed/_composable/fsdp/test_fully_shard_state 1/1 2024-11-01T16:52:17.2265471Z distributed/pipelining/test_stage 1/1 2024-11-01T16:52:17.2266018Z distributed/pipelining/test_unflatten 1/1 2024-11-01T16:52:17.2266596Z distributed/_composable/test_checkpoint 1/1 2024-11-01T16:52:17.2267173Z distributed/optim/test_named_optimizer 1/1 2024-11-01T16:52:17.2267712Z distributed/checkpoint/test_fsspec 1/1 2024-11-01T16:52:17.2268372Z distributed/_composable/fully_shard/test_fully_shard_util 1/1 2024-11-01T16:52:17.2269048Z distributed/checkpoint/test_traverse 1/1 2024-11-01T16:52:17.2269628Z distributed/test_compute_comm_reordering 1/1 2024-11-01T16:52:17.2270264Z distributed/checkpoint/test_dtensor_resharding 1/1 2024-11-01T16:52:17.2270929Z distributed/_tensor/debug/test_comm_mode_features 1/1 2024-11-01T16:52:17.2271497Z Parallel tests (0): 2024-11-01T16:52:17.2271894Z Name: excluded (est. time: 102.54min) 2024-11-01T16:52:17.2272350Z Serial tests (56): 2024-11-01T16:52:17.2272758Z distributed/test_distributed_spawn 1/10 2024-11-01T16:52:17.2273401Z distributed/test_distributed_spawn 2/10 2024-11-01T16:52:17.2273913Z distributed/test_distributed_spawn 5/10 2024-11-01T16:52:17.2274433Z distributed/test_distributed_spawn 8/10 2024-11-01T16:52:17.2274971Z distributed/fsdp/test_fsdp_state_dict 1/2 2024-11-01T16:52:17.2275491Z distributed/fsdp/test_fsdp_core 2/2 2024-11-01T16:52:17.2275968Z distributed/test_c10d_nccl 3/3 2024-11-01T16:52:17.2276463Z distributed/rpc/cuda/test_tensorpipe_agent 2/2 2024-11-01T16:52:17.2277464Z distributed/optim/test_zero_redundancy_optimizer 1/1 2024-11-01T16:52:17.2278186Z distributed/_composable/fsdp/test_fully_shard_compile 1/1 2024-11-01T16:52:17.2278836Z distributed/fsdp/test_fsdp_comm_hooks 1/1 2024-11-01T16:52:17.2279356Z distributed/fsdp/test_wrap 1/1 2024-11-01T16:52:17.2279834Z distributed/test_functional_api 1/1 2024-11-01T16:52:17.2280320Z distributed/test_c10d_common 1/1 2024-11-01T16:52:17.2280799Z distributed/test_pg_wrapper 1/1 2024-11-01T16:52:17.2281413Z distributed/algorithms/quantization/test_quantization 1/1 2024-11-01T16:52:17.2282047Z distributed/fsdp/test_fsdp_comm 1/1 2024-11-01T16:52:17.2282584Z distributed/test_c10d_object_collectives 1/1 2024-11-01T16:52:17.2283208Z distributed/tensor/parallel/test_parallelize_api 1/1 2024-11-01T16:52:17.2283829Z distributed/fsdp/test_fsdp_exec_order 1/1 2024-11-01T16:52:17.2284386Z distributed/_composable/test_compose 1/1 2024-11-01T16:52:17.2284923Z distributed/_tensor/test_utils 1/1 2024-11-01T16:52:17.2285506Z distributed/checkpoint/e2e/test_e2e_save_and_load 1/1 2024-11-01T16:52:17.2286315Z distributed/tensor/parallel/test_tp_style 1/1 2024-11-01T16:52:17.2286997Z distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 2024-11-01T16:52:17.2287750Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1 2024-11-01T16:52:17.2288631Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 2024-11-01T16:52:17.2289359Z distributed/elastic/multiprocessing/api_test 1/1 2024-11-01T16:52:17.2289998Z distributed/_shard/sharded_tensor/ops/test_init 1/1 2024-11-01T16:52:17.2290590Z distributed/_tools/test_fsdp2_mem_tracker 1/1 2024-11-01T16:52:17.2291187Z distributed/checkpoint/test_state_dict_utils 1/1 2024-11-01T16:52:17.2291844Z distributed/_shard/sharding_spec/test_sharding_spec 1/1 2024-11-01T16:52:17.2292427Z distributed/test_c10d_logger 1/1 2024-11-01T16:52:17.2293141Z distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 2024-11-01T16:52:17.2293802Z distributed/fsdp/test_fsdp_apply 1/1 2024-11-01T16:52:17.2294437Z distributed/_composable/fully_shard/test_fully_shard_runtime 1/1 2024-11-01T16:52:17.2295205Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 1/1 2024-11-01T16:52:17.2295927Z distributed/_shard/sharded_optim/test_sharded_optim 1/1 2024-11-01T16:52:17.2296722Z distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 1/1 2024-11-01T16:52:17.2297425Z distributed/fsdp/test_fsdp_input 1/1 2024-11-01T16:52:17.2297982Z distributed/checkpoint/test_fsdp_model_state 1/1 2024-11-01T16:52:17.2298633Z distributed/_composable/fsdp/test_fully_shard_memory 1/1 2024-11-01T16:52:17.2299284Z distributed/fsdp/test_fsdp_multiple_forward 1/1 2024-11-01T16:52:17.2299945Z distributed/_composable/fsdp/test_fully_shard_overlap 1/1 2024-11-01T16:52:17.2300745Z distributed/_composable/fully_shard/test_fully_shard_mixed_precision 1/1 2024-11-01T16:52:17.2301742Z distributed/elastic/timer/local_timer_test 1/1 2024-11-01T16:52:17.2302382Z distributed/checkpoint/test_dtensor_checkpoint 1/1 2024-11-01T16:52:17.2302961Z distributed/test_launcher 1/1 2024-11-01T16:52:17.2303476Z distributed/fsdp/test_checkpoint_wrapper 1/1 2024-11-01T16:52:17.2304042Z distributed/test_multi_threaded_pg 1/1 2024-11-01T16:52:17.2304578Z distributed/elastic/utils/util_test 1/1 2024-11-01T16:52:17.2305137Z distributed/_tensor/debug/test_comm_mode 1/1 2024-11-01T16:52:17.2305690Z distributed/checkpoint/test_planner 1/1 2024-11-01T16:52:17.2306245Z distributed/nn/jit/test_instantiator 1/1 2024-11-01T16:52:17.2306787Z distributed/checkpoint/test_utils 1/1 2024-11-01T16:52:17.2307368Z distributed/fsdp/test_distributed_checkpoint 1/1 2024-11-01T16:52:17.2307917Z Parallel tests (0): 2024-11-01T16:52:17.2325748Z Running distributed/test_inductor_collectives 1/1 ... [2024-11-01 16:52:17.232205] 2024-11-01T16:52:17.2326523Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:52:17.2328588Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_inductor_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:52:17.232581] 2024-11-01T16:54:53.6562427Z 2024-11-01T16:54:53.6564265Z distributed/test_inductor_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_inductor_collectives_1.1_a3cfb12b4b36c712_.log 2024-11-01T16:54:53.6602584Z Running 48 items in this shard: test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor_split_sizes_none, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_contiguous_input, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_into_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_output_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor_cudagraph_trees, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_input_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_broadcast_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_c10d_functional_tagged_pt2_compliant, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_eager_allreduce_inductor_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_eager_async_allreduce_inductor_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_inductor_allreduce_eager_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_permute_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_reduce_scatter_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_backwards, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source_GroupMember_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source__get_default_group, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source_group_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_graphbreaks_unsupported_async_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_pg_var, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_args_match, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_list, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_to_all_single, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_kwargs, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_kwargs_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_positional, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_positional_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_unspecified, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op0, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op1, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op2, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op3, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op4, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_reduce_scatter, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_support_collective_op_with_async_op_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor_pg, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allgather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allreduce, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_reduce_scatter_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_all_gather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_doesnt_mutate_shared, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_reduce_scatter_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_single_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_steal_buffer, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_meta 2024-11-01T16:54:53.6636489Z 2024-11-01T16:54:53.6637175Z Running distributed/_tensor/test_attention 1/1 ... [2024-11-01 16:54:53.656575] 2024-11-01T16:54:53.6637902Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:54:53.6639876Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_attention.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:54:53.656994] 2024-11-01T16:55:08.8586750Z 2024-11-01T16:55:08.8588608Z distributed/_tensor/test_attention 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_attention_1.1_49ee7afd2ab75a63_.log 2024-11-01T16:55:08.8592283Z Running 4 items in this shard: test/distributed/_tensor/test_attention.py::RingAttentionTest::test_is_causal_behavior, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_custom_transformer, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_native_transformer, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa 2024-11-01T16:55:08.8594856Z 2024-11-01T16:55:08.8595813Z Running distributed/checkpoint/test_save_load_api 1/1 ... [2024-11-01 16:55:08.859075] 2024-11-01T16:55:08.8596594Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:55:08.8598679Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_save_load_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:55:08.859471] 2024-11-01T16:55:17.9960810Z 2024-11-01T16:55:17.9962782Z distributed/checkpoint/test_save_load_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_save_load_api_1.1_ed8fb7800807d70b_.log 2024-11-01T16:55:17.9964951Z Running 1 items in this shard: test/distributed/checkpoint/test_save_load_api.py::TestSaveAndLoadAPI::test_auto_detect 2024-11-01T16:55:17.9965825Z 2024-11-01T16:55:17.9969810Z Running distributed/checkpoint/e2e/test_fsdp_ep 1/1 ... [2024-11-01 16:55:17.996196] 2024-11-01T16:55:17.9970593Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:55:17.9972668Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/e2e/test_fsdp_ep.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:55:17.996593] 2024-11-01T16:55:27.5356475Z 2024-11-01T16:55:27.5359658Z distributed/checkpoint/e2e/test_fsdp_ep 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.e2e.test_fsdp_ep_1.1_e818bccad8133b1e_.log 2024-11-01T16:55:27.5363100Z Running 1 items in this shard: test/distributed/checkpoint/e2e/test_fsdp_ep.py::TestFSDPWithEP::test_e2e 2024-11-01T16:55:27.5364335Z 2024-11-01T16:55:27.5365508Z Running distributed/test_c10d_functional_native 1/1 ... [2024-11-01 16:55:27.535963] 2024-11-01T16:55:27.5366676Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:55:27.5370425Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_functional_native.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:55:27.536531] 2024-11-01T16:58:28.5464743Z 2024-11-01T16:58:28.5466662Z distributed/test_c10d_functional_native 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_functional_native_1.1_f95324289c67b833_.log 2024-11-01T16:58:28.5484625Z Running 28 items in this shard: test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_to_all_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_broadcast, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_inductor_dtypeview_memory_leak, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_py_work, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_threading, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_unwaited, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_wait_tensor, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_non_contig_input, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_to_all_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_broadcast, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_inplace_op_on_view, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reuse_buffer_after_inplace_collective, test/distributed/test_c10d_functional_native.py::CompileTest::test_ranks_and_tag, test/distributed/test_c10d_functional_native.py::CompileTest::test_wait_tensor 2024-11-01T16:58:28.5500764Z 2024-11-01T16:58:28.5501703Z Running distributed/_tensor/experimental/test_register_sharding 1/1 ... [2024-11-01 16:58:28.546813] 2024-11-01T16:58:28.5502586Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:58:28.5504844Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/experimental/test_register_sharding.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:58:28.547223] 2024-11-01T16:58:43.1003070Z 2024-11-01T16:58:43.1005123Z distributed/_tensor/experimental/test_register_sharding 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.experimental.test_register_sharding_1.1_e904403f0a12ce2e_.log 2024-11-01T16:58:43.1008539Z Running 2 items in this shard: test/distributed/_tensor/experimental/test_register_sharding.py::TestRegisterSharding::test_argmax, test/distributed/_tensor/experimental/test_register_sharding.py::TestRegisterSharding::test_softmax_fwd 2024-11-01T16:58:43.1010127Z 2024-11-01T16:58:43.1010671Z Running distributed/_tools/test_memory_tracker 1/1 ... [2024-11-01 16:58:43.100513] 2024-11-01T16:58:43.1011423Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:58:43.1013477Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_memory_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:58:43.100930] 2024-11-01T16:58:48.2781838Z 2024-11-01T16:58:48.2784065Z distributed/_tools/test_memory_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_memory_tracker_1.1_943b6cb2e7d5331c_.log 2024-11-01T16:58:48.2786180Z Running 1 items in this shard: test/distributed/_tools/test_memory_tracker.py::TestMemoryTracker::test_local_model 2024-11-01T16:58:48.2787022Z 2024-11-01T16:58:48.2787689Z Running distributed/_tensor/test_xla_integration 1/1 ... [2024-11-01 16:58:48.278352] 2024-11-01T16:58:48.2788451Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:58:48.2790520Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_xla_integration.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:58:48.278780] 2024-11-01T16:58:52.8049315Z 2024-11-01T16:58:52.8051269Z distributed/_tensor/test_xla_integration 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_xla_integration_1.1_f983f07322ed6422_.log 2024-11-01T16:58:52.8054883Z Running 3 items in this shard: test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_1d_replicate, test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_1d_shard, test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_2d 2024-11-01T16:58:52.8057389Z 2024-11-01T16:58:52.8058029Z Running distributed/_composable/fsdp/test_fully_shard_state 1/1 ... [2024-11-01 16:58:52.804949] 2024-11-01T16:58:52.8058862Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:58:52.8061124Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:58:52.805368] 2024-11-01T16:58:57.7824310Z 2024-11-01T16:58:57.7826385Z distributed/_composable/fsdp/test_fully_shard_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_state_1.1_34e6c22ca6fd8831_.log 2024-11-01T16:58:57.7831533Z Running 5 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_cls, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_deepcopy, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_reapply, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_state, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_unsupported_module_cls 2024-11-01T16:58:57.7835257Z 2024-11-01T16:58:57.7835763Z Running distributed/pipelining/test_stage 1/1 ... [2024-11-01 16:58:57.782869] 2024-11-01T16:58:57.7836503Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:58:57.7838533Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_stage.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:58:57.783275] 2024-11-01T16:59:10.3302060Z 2024-11-01T16:59:10.3303946Z distributed/pipelining/test_stage 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_stage_1.1_2afecb60530381b7_.log 2024-11-01T16:59:10.3308949Z Running 6 items in this shard: test/distributed/pipelining/test_stage.py::StageTest::test_custom_dw_errors, test/distributed/pipelining/test_stage.py::StageTest::test_custom_dw_with_fb_schedule, test/distributed/pipelining/test_stage.py::StageTest::test_manual, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_ModelClass0, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_ModelClass1, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_kwargs_ModelClass0 2024-11-01T16:59:10.3315333Z Running 6 items in this shard: test/distributed/pipelining/test_stage.py::StageTest::test_custom_dw_errors, test/distributed/pipelining/test_stage.py::StageTest::test_custom_dw_with_fb_schedule, test/distributed/pipelining/test_stage.py::StageTest::test_manual, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_ModelClass0, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_ModelClass1, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_kwargs_ModelClass0 2024-11-01T16:59:10.3318420Z 2024-11-01T16:59:10.3318943Z Running distributed/pipelining/test_unflatten 1/1 ... [2024-11-01 16:59:10.330272] 2024-11-01T16:59:10.3319677Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:59:10.3321687Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_unflatten.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:59:10.330688] 2024-11-01T16:59:15.3079038Z 2024-11-01T16:59:15.3080921Z distributed/pipelining/test_unflatten 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_unflatten_1.1_083efdd7681f6ca7_.log 2024-11-01T16:59:15.3083092Z Running 1 items in this shard: test/distributed/pipelining/test_unflatten.py::UnflattenTests::test_unflatten 2024-11-01T16:59:15.3083877Z 2024-11-01T16:59:15.3084405Z Running distributed/_composable/test_checkpoint 1/1 ... [2024-11-01 16:59:15.308134] 2024-11-01T16:59:15.3085155Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:59:15.3088122Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:59:15.308556] 2024-11-01T16:59:20.4368280Z 2024-11-01T16:59:20.4370194Z distributed/_composable/test_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_checkpoint_1.1_7d52b01c21c5f994_.log 2024-11-01T16:59:20.4374908Z Running 6 items in this shard: test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_checkpoint_kwargs, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_clears_state_on_error_in_forward, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_multi_args, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_random_cpu, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_tensor_only_cpu, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_tensor_only_gpu 2024-11-01T16:59:20.4378602Z 2024-11-01T16:59:20.4379152Z Running distributed/optim/test_named_optimizer 1/1 ... [2024-11-01 16:59:20.436752] 2024-11-01T16:59:20.4379924Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:59:20.4382270Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/optim/test_named_optimizer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:59:20.437157] 2024-11-01T16:59:23.5750545Z 2024-11-01T16:59:23.5752442Z distributed/optim/test_named_optimizer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_named_optimizer_1.1_5534ef7c52c2ec98_.log 2024-11-01T16:59:23.5753892Z 2024-11-01T16:59:23.5756711Z Running distributed/checkpoint/test_fsspec 1/1 ... [2024-11-01 16:59:23.575500] 2024-11-01T16:59:23.5757447Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:59:23.5761874Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_fsspec.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:59:23.575899] 2024-11-01T16:59:40.4346551Z 2024-11-01T16:59:40.4348563Z distributed/checkpoint/test_fsspec 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsspec_1.1_4b231f0cd78f3d67_.log 2024-11-01T16:59:40.4350874Z Running 2 items in this shard: test/distributed/checkpoint/test_fsspec.py::TestFSSpec::test_fsspec, test/distributed/checkpoint/test_fsspec.py::TestFSSpec::test_overwrite 2024-11-01T16:59:40.4352046Z 2024-11-01T16:59:40.4356528Z Running distributed/_composable/fully_shard/test_fully_shard_util 1/1 ... [2024-11-01 16:59:40.435058] 2024-11-01T16:59:40.4357377Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:59:40.4359588Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fully_shard/test_fully_shard_util.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:59:40.435488] 2024-11-01T16:59:51.1779741Z 2024-11-01T16:59:51.1782026Z distributed/_composable/fully_shard/test_fully_shard_util 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fully_shard.test_fully_shard_util_1.1_40cb507885c5823e_.log 2024-11-01T16:59:51.1785392Z Running 2 items in this shard: test/distributed/_composable/fully_shard/test_fully_shard_util.py::TestUtils::test_get_sharded_module_tree_with_module_name_to_fqns, test/distributed/_composable/fully_shard/test_fully_shard_util.py::TestUtilsSingleDevice::test_foreach_copy_float8 2024-11-01T16:59:51.1787289Z 2024-11-01T16:59:51.1787903Z Running distributed/checkpoint/test_traverse 1/1 ... [2024-11-01 16:59:51.178293] 2024-11-01T16:59:51.1788721Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:59:51.1790820Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_traverse.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:59:51.178720] 2024-11-01T16:59:55.7049672Z 2024-11-01T16:59:55.7051627Z distributed/checkpoint/test_traverse 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_traverse_1.1_d176f2e4d16432ec_.log 2024-11-01T16:59:55.7057091Z Running 7 items in this shard: test/distributed/checkpoint/test_traverse.py::TestTraverse::test_get_element, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_set_element, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_doesnt_ignore_intermediate_collections, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_nested_dict, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_nested_list, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_shallow, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_with_ordered_dict 2024-11-01T16:59:55.7062109Z 2024-11-01T16:59:55.7062690Z Running distributed/test_compute_comm_reordering 1/1 ... [2024-11-01 16:59:55.704960] 2024-11-01T16:59:55.7063470Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:59:55.7065613Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_compute_comm_reordering.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:59:55.705414] 2024-11-01T17:01:03.4424132Z 2024-11-01T17:01:03.4426035Z distributed/test_compute_comm_reordering 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_compute_comm_reordering_1.1_4f05c643b1ecfb40_.log 2024-11-01T17:01:03.4434274Z Running 7 items in this shard: test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_grouped_scheduler_node, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_nccl_heuristics, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_raise_comms, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap_custom_runtime_estimation, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits_raise_comms 2024-11-01T17:01:03.4439642Z 2024-11-01T17:01:03.4440346Z Running distributed/checkpoint/test_dtensor_resharding 1/1 ... [2024-11-01 17:01:03.442825] 2024-11-01T17:01:03.4441152Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:01:03.4443278Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_dtensor_resharding.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:01:03.443232] 2024-11-01T17:01:33.0822269Z 2024-11-01T17:01:33.0824052Z distributed/checkpoint/test_dtensor_resharding 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dtensor_resharding_1.1_07cad78ee7c8c37b_.log 2024-11-01T17:01:33.0829836Z Running 5 items in this shard: test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardPlacementChange::test_1d_to_1d_reshard_placement_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardPlacementChange::test_2d_to_2d_reshard_placement_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_1d_to_2d_reshard_mesh_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_2d_to_1d_reshard_mesh_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_dtensor_checkpoint_resharding_with_empty_shard 2024-11-01T17:01:33.0834112Z 2024-11-01T17:01:33.0834693Z Running distributed/_tensor/debug/test_comm_mode_features 1/1 ... [2024-11-01 17:01:33.082373] 2024-11-01T17:01:33.0835500Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:01:33.0837639Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/debug/test_comm_mode_features.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:01:33.082797] 2024-11-01T17:01:57.7122611Z 2024-11-01T17:01:57.7124695Z distributed/_tensor/debug/test_comm_mode_features 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.debug.test_comm_mode_features_1.1_12e8d9c78fead8ad_.log 2024-11-01T17:01:57.7130933Z Running 4 items in this shard: test/distributed/_tensor/debug/test_comm_mode_features.py::TestCommModeFeatures::test_MLPStacked_distributed_sharding_display, test/distributed/_tensor/debug/test_comm_mode_features.py::TestCommModeFeatures::test_MLP_distributed_sharding_display, test/distributed/_tensor/debug/test_comm_mode_features.py::TestCommModeFeatures::test_MLP_module_tracing, test/distributed/_tensor/debug/test_comm_mode_features.py::TestCommModeFeatures::test_transformer_module_tracing 2024-11-01T17:01:57.7135994Z 2024-11-01T17:01:58.4414112Z Running test batch 'tests to run' cost 581.21 seconds 2024-11-01T17:01:59.2264715Z 2024-11-01T17:01:59.2265222Z real 9m47.946s 2024-11-01T17:01:59.2265637Z user 16m46.539s 2024-11-01T17:01:59.2266743Z sys 6m57.821s 2024-11-01T17:01:59.2267119Z + assert_git_not_dirty 2024-11-01T17:01:59.2267846Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-11-01T17:01:59.2268550Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *xla* ]] 2024-11-01T17:01:59.2269439Z ++ git status --porcelain 2024-11-01T17:01:59.2270136Z ++ grep -v '?? third_party' 2024-11-01T17:02:01.8104799Z ++ true 2024-11-01T17:02:01.8105160Z + git_status= 2024-11-01T17:02:01.8105695Z + [[ -n '' ]] 2024-11-01T17:02:01.8106218Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-11-01T17:02:01.8106765Z + [[ 3 == 1 ]] 2024-11-01T17:02:01.8107080Z + [[ 3 == 1 ]] 2024-11-01T17:02:01.8107404Z + cleanup_workspace 2024-11-01T17:02:01.8108347Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-11-01T17:02:01.8109668Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-11-01T17:02:01.8110779Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-11-01T17:02:01.8111534Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-11-01T17:02:01.8112493Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-11-01T17:02:01.8113580Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-11-01T17:02:01.8114522Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-11-01T17:02:02.3173087Z ##[group]Run cat test/**/*_toprint.log || true 2024-11-01T17:02:02.3173664Z cat test/**/*_toprint.log || true 2024-11-01T17:02:02.3184879Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:02:02.3185465Z env: 2024-11-01T17:02:02.3185769Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:02:02.3186284Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:02:02.3187165Z DOCKER_CONTAINER_ID: 1b47f5de842509be1b3efe91f0254870f158ff4faecdf9723b7904a3b4486cbe 2024-11-01T17:02:02.3187950Z ##[endgroup] 2024-11-01T17:02:02.3267271Z cat: 'test/**/*_toprint.log': No such file or directory 2024-11-01T17:02:02.3301673Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-11-01T17:02:02.3302281Z kill "$MONITOR_SCRIPT_PID" 2024-11-01T17:02:02.3308618Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:02:02.3309120Z env: 2024-11-01T17:02:02.3309389Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:02:02.3309861Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:02:02.3310648Z DOCKER_CONTAINER_ID: 1b47f5de842509be1b3efe91f0254870f158ff4faecdf9723b7904a3b4486cbe 2024-11-01T17:02:02.3311361Z MONITOR_SCRIPT_PID: 117571 2024-11-01T17:02:02.3311704Z ##[endgroup] 2024-11-01T17:02:02.3538886Z Prepare all required actions 2024-11-01T17:02:02.3539358Z Getting action download info 2024-11-01T17:02:02.5738212Z Download action repository 'actions/upload-artifact@v3' (SHA:ff15f0306b3f739f7b6fd43fb5d26cd321bd4de5) 2024-11-01T17:02:02.7252211Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-11-01T17:02:02.7252710Z with: 2024-11-01T17:02:02.7253282Z file-suffix: test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_32397035132 2024-11-01T17:02:02.7254017Z s3-bucket: gha-artifacts 2024-11-01T17:02:02.7254374Z env: 2024-11-01T17:02:02.7254651Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:02:02.7255130Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:02:02.7256111Z DOCKER_CONTAINER_ID: 1b47f5de842509be1b3efe91f0254870f158ff4faecdf9723b7904a3b4486cbe 2024-11-01T17:02:02.7256843Z ##[endgroup] 2024-11-01T17:02:02.7291380Z ##[group]Run # Remove any previous test jsons if they exist 2024-11-01T17:02:02.7292114Z # Remove any previous test jsons if they exist 2024-11-01T17:02:02.7292674Z rm -f test-jsons-*.zip 2024-11-01T17:02:02.7293299Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json' 2024-11-01T17:02:02.7299940Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:02:02.7300502Z env: 2024-11-01T17:02:02.7300797Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:02:02.7301396Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:02:02.7302453Z DOCKER_CONTAINER_ID: 1b47f5de842509be1b3efe91f0254870f158ff4faecdf9723b7904a3b4486cbe 2024-11-01T17:02:02.7303546Z FILE_SUFFIX: test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_32397035132 2024-11-01T17:02:02.7304281Z ##[endgroup] 2024-11-01T17:02:02.7358527Z adding: test/test-reports/td_exclusions-e2cb857a6436aecf756f.json (deflated 86%) 2024-11-01T17:02:02.7390566Z ##[group]Run # Remove any previous test reports if they exist 2024-11-01T17:02:02.7391320Z # Remove any previous test reports if they exist 2024-11-01T17:02:02.7391938Z rm -f test-reports-*.zip 2024-11-01T17:02:02.7392806Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv' 2024-11-01T17:02:02.7399033Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:02:02.7399560Z env: 2024-11-01T17:02:02.7399843Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:02:02.7400332Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:02:02.7401164Z DOCKER_CONTAINER_ID: 1b47f5de842509be1b3efe91f0254870f158ff4faecdf9723b7904a3b4486cbe 2024-11-01T17:02:02.7402193Z FILE_SUFFIX: test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_32397035132 2024-11-01T17:02:02.7402984Z ##[endgroup] 2024-11-01T17:02:02.7454002Z adding: test/test-reports/python-pytest/distributed.test_inductor_collectives/distributed.test_inductor_collectives-b973b50420b13ab4.xml (deflated 87%) 2024-11-01T17:02:02.7456104Z adding: test/test-reports/python-pytest/distributed._tensor.test_attention/distributed._tensor.test_attention-52585a6bdef737c6.xml (deflated 69%) 2024-11-01T17:02:02.7458234Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_save_load_api/distributed.checkpoint.test_save_load_api-ef57249e4321d81f.xml (deflated 36%) 2024-11-01T17:02:02.7460449Z adding: test/test-reports/python-pytest/distributed.checkpoint.e2e.test_fsdp_ep/distributed.checkpoint.e2e.test_fsdp_ep-5812d4c6d32fba10.xml (deflated 36%) 2024-11-01T17:02:02.7462943Z adding: test/test-reports/python-pytest/distributed.test_c10d_functional_native/distributed.test_c10d_functional_native-8d6c356f6e2b97ce.xml (deflated 85%) 2024-11-01T17:02:02.7465398Z adding: test/test-reports/python-pytest/distributed._tensor.experimental.test_register_sharding/distributed._tensor.experimental.test_register_sharding-817584b880520d94.xml (deflated 50%) 2024-11-01T17:02:02.7467795Z adding: test/test-reports/python-pytest/distributed._tools.test_memory_tracker/distributed._tools.test_memory_tracker-86f5daabcf4e575a.xml (deflated 63%) 2024-11-01T17:02:02.7470237Z adding: test/test-reports/python-pytest/distributed._tensor.test_xla_integration/distributed._tensor.test_xla_integration-1ce037758a6b0e6f.xml (deflated 74%) 2024-11-01T17:02:02.7472629Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_state/distributed._composable.fsdp.test_fully_shard_state-bfc03c0986dc8ebc.xml (deflated 73%) 2024-11-01T17:02:02.7475014Z adding: test/test-reports/python-pytest/distributed.pipelining.test_stage/distributed.pipelining.test_stage-b52c6d2af24b7de2.xml (deflated 60%) 2024-11-01T17:02:02.7477202Z adding: test/test-reports/python-pytest/distributed.pipelining.test_stage/distributed.pipelining.test_stage-15737a501fadb4a9.xml (deflated 69%) 2024-11-01T17:02:02.7479688Z adding: test/test-reports/python-pytest/distributed.pipelining.test_unflatten/distributed.pipelining.test_unflatten-fd08b038ed3f5bef.xml (deflated 41%) 2024-11-01T17:02:02.7481909Z adding: test/test-reports/python-pytest/distributed._composable.test_checkpoint/distributed._composable.test_checkpoint-f6b5a72a337bbb18.xml (deflated 72%) 2024-11-01T17:02:02.7484074Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_fsspec/distributed.checkpoint.test_fsspec-5c85037287d622ab.xml (deflated 48%) 2024-11-01T17:02:02.7486468Z adding: test/test-reports/python-pytest/distributed._composable.fully_shard.test_fully_shard_util/distributed._composable.fully_shard.test_fully_shard_util-0d8ed275d235758e.xml (deflated 53%) 2024-11-01T17:02:02.7488958Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_traverse/distributed.checkpoint.test_traverse-cb2b61f7ed730485.xml (deflated 76%) 2024-11-01T17:02:02.7491151Z adding: test/test-reports/python-pytest/distributed.test_compute_comm_reordering/distributed.test_compute_comm_reordering-8e71ae5f89112bde.xml (deflated 73%) 2024-11-01T17:02:02.7493355Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_dtensor_resharding/distributed.checkpoint.test_dtensor_resharding-ed93588daf0c7e41.xml (deflated 70%) 2024-11-01T17:02:02.7495648Z adding: test/test-reports/python-pytest/distributed._tensor.debug.test_comm_mode_features/distributed._tensor.debug.test_comm_mode_features-cf8c6074fb72df95.xml (deflated 66%) 2024-11-01T17:02:02.7523861Z ##[group]Run # Remove any previous usage logs if they exist 2024-11-01T17:02:02.7524561Z # Remove any previous usage logs if they exist 2024-11-01T17:02:02.7525093Z rm -f logs-*.zip 2024-11-01T17:02:02.7525840Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-11-01T17:02:02.7526723Z # so check to see if the file exists first 2024-11-01T17:02:02.7527265Z if [ -f 'usage_log.txt' ]; then 2024-11-01T17:02:02.7527843Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-11-01T17:02:02.7528370Z fi 2024-11-01T17:02:02.7528903Z if find "test/test-reports" -name "*.log" 2>/dev/null | grep -q .; then 2024-11-01T17:02:02.7529785Z  zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' 2024-11-01T17:02:02.7530393Z fi 2024-11-01T17:02:02.7535916Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:02:02.7536447Z env: 2024-11-01T17:02:02.7536731Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:02:02.7537215Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:02:02.7538117Z DOCKER_CONTAINER_ID: 1b47f5de842509be1b3efe91f0254870f158ff4faecdf9723b7904a3b4486cbe 2024-11-01T17:02:02.7539075Z FILE_SUFFIX: test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_32397035132 2024-11-01T17:02:02.7539743Z ##[endgroup] 2024-11-01T17:02:02.7591015Z adding: usage_log.txt (deflated 92%) 2024-11-01T17:02:02.7629748Z adding: test/test-reports/distributed.test_inductor_collectives_1.1_a3cfb12b4b36c712_.log (deflated 87%) 2024-11-01T17:02:02.7631220Z adding: test/test-reports/distributed._tensor.test_attention_1.1_49ee7afd2ab75a63_.log (deflated 68%) 2024-11-01T17:02:02.7632943Z adding: test/test-reports/distributed.checkpoint.test_save_load_api_1.1_ed8fb7800807d70b_.log (deflated 54%) 2024-11-01T17:02:02.7634582Z adding: test/test-reports/distributed.checkpoint.e2e.test_fsdp_ep_1.1_e818bccad8133b1e_.log (deflated 74%) 2024-11-01T17:02:02.7636039Z adding: test/test-reports/distributed.test_c10d_functional_native_1.1_f95324289c67b833_.log (deflated 87%) 2024-11-01T17:02:02.7637589Z adding: test/test-reports/distributed._tensor.experimental.test_register_sharding_1.1_e904403f0a12ce2e_.log (deflated 63%) 2024-11-01T17:02:02.7639116Z adding: test/test-reports/distributed._tools.test_memory_tracker_1.1_943b6cb2e7d5331c_.log (deflated 54%) 2024-11-01T17:02:02.7640537Z adding: test/test-reports/distributed._tensor.test_xla_integration_1.1_f983f07322ed6422_.log (deflated 66%) 2024-11-01T17:02:02.7642175Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_state_1.1_34e6c22ca6fd8831_.log (deflated 71%) 2024-11-01T17:02:02.7643671Z adding: test/test-reports/distributed.pipelining.test_stage_1.1_2afecb60530381b7_.log (deflated 81%) 2024-11-01T17:02:02.7645096Z adding: test/test-reports/distributed.pipelining.test_unflatten_1.1_083efdd7681f6ca7_.log (deflated 54%) 2024-11-01T17:02:02.7646530Z adding: test/test-reports/distributed._composable.test_checkpoint_1.1_7d52b01c21c5f994_.log (deflated 69%) 2024-11-01T17:02:02.7648057Z adding: test/test-reports/distributed.optim.test_named_optimizer_1.1_5534ef7c52c2ec98_.log (deflated 35%) 2024-11-01T17:02:02.7649510Z adding: test/test-reports/distributed.checkpoint.test_fsspec_1.1_4b231f0cd78f3d67_.log (deflated 83%) 2024-11-01T17:02:02.7651043Z adding: test/test-reports/distributed._composable.fully_shard.test_fully_shard_util_1.1_40cb507885c5823e_.log (deflated 66%) 2024-11-01T17:02:02.7652570Z adding: test/test-reports/distributed.checkpoint.test_traverse_1.1_d176f2e4d16432ec_.log (deflated 71%) 2024-11-01T17:02:02.7654001Z adding: test/test-reports/distributed.test_compute_comm_reordering_1.1_4f05c643b1ecfb40_.log (deflated 83%) 2024-11-01T17:02:02.7655488Z adding: test/test-reports/distributed.checkpoint.test_dtensor_resharding_1.1_07cad78ee7c8c37b_.log (deflated 75%) 2024-11-01T17:02:02.7657084Z adding: test/test-reports/distributed._tensor.debug.test_comm_mode_features_1.1_12e8d9c78fead8ad_.log (deflated 72%) 2024-11-01T17:02:02.7686133Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-11-01T17:02:02.7686972Z # Remove any previous debugging artifacts if they exist 2024-11-01T17:02:02.7687607Z rm -f debug-*.zip 2024-11-01T17:02:02.7688034Z if [ -d 'test/debug' ]; then 2024-11-01T17:02:02.7688606Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-11-01T17:02:02.7689258Z fi 2024-11-01T17:02:02.7694840Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:02:02.7695367Z env: 2024-11-01T17:02:02.7695648Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:02:02.7696138Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:02:02.7696972Z DOCKER_CONTAINER_ID: 1b47f5de842509be1b3efe91f0254870f158ff4faecdf9723b7904a3b4486cbe 2024-11-01T17:02:02.7698079Z FILE_SUFFIX: test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_32397035132 2024-11-01T17:02:02.7698743Z ##[endgroup] 2024-11-01T17:02:02.7813998Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-11-01T17:02:02.7814450Z with: 2024-11-01T17:02:02.7814721Z s3-bucket: gha-artifacts 2024-11-01T17:02:02.7815158Z s3-prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T17:02:02.7815629Z retention-days: 14 2024-11-01T17:02:02.7815957Z if-no-files-found: warn 2024-11-01T17:02:02.7816307Z path: test-jsons-*.zip 2024-11-01T17:02:02.7816639Z name: artifact 2024-11-01T17:02:02.7816937Z region: us-east-1 2024-11-01T17:02:02.7817217Z env: 2024-11-01T17:02:02.7817478Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:02:02.7817930Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:02:02.7818856Z DOCKER_CONTAINER_ID: 1b47f5de842509be1b3efe91f0254870f158ff4faecdf9723b7904a3b4486cbe 2024-11-01T17:02:02.7819552Z ##[endgroup] 2024-11-01T17:02:03.1843545Z NOTE: s3-prefix specified, ignoring name parameter 2024-11-01T17:02:03.1844257Z With the provided path, there will be 1 file uploaded 2024-11-01T17:02:03.1844964Z Uploading to s3 prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T17:02:03.1887837Z Starting upload of test-jsons-test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_32397035132.zip 2024-11-01T17:02:03.2754352Z Finished upload of test-jsons-test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_32397035132.zip 2024-11-01T17:02:03.2947174Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-11-01T17:02:03.2947775Z with: 2024-11-01T17:02:03.2948210Z s3-bucket: gha-artifacts 2024-11-01T17:02:03.2948650Z s3-prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T17:02:03.2949137Z retention-days: 14 2024-11-01T17:02:03.2949471Z if-no-files-found: error 2024-11-01T17:02:03.2949836Z path: test-reports-*.zip 2024-11-01T17:02:03.2950183Z name: artifact 2024-11-01T17:02:03.2950485Z region: us-east-1 2024-11-01T17:02:03.2950780Z env: 2024-11-01T17:02:03.2951044Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:02:03.2951504Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:02:03.2952291Z DOCKER_CONTAINER_ID: 1b47f5de842509be1b3efe91f0254870f158ff4faecdf9723b7904a3b4486cbe 2024-11-01T17:02:03.2952975Z ##[endgroup] 2024-11-01T17:02:03.6957960Z NOTE: s3-prefix specified, ignoring name parameter 2024-11-01T17:02:03.6958840Z With the provided path, there will be 1 file uploaded 2024-11-01T17:02:03.6959554Z Uploading to s3 prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T17:02:03.7002760Z Starting upload of test-reports-test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_32397035132.zip 2024-11-01T17:02:03.8199733Z Finished upload of test-reports-test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_32397035132.zip 2024-11-01T17:02:03.8388767Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-11-01T17:02:03.8389284Z with: 2024-11-01T17:02:03.8389577Z s3-bucket: gha-artifacts 2024-11-01T17:02:03.8390069Z s3-prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T17:02:03.8390612Z retention-days: 14 2024-11-01T17:02:03.8390980Z if-no-files-found: ignore 2024-11-01T17:02:03.8391376Z path: logs-*.zip 2024-11-01T17:02:03.8391697Z name: artifact 2024-11-01T17:02:03.8392026Z region: us-east-1 2024-11-01T17:02:03.8392350Z env: 2024-11-01T17:02:03.8392746Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:02:03.8393239Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:02:03.8394142Z DOCKER_CONTAINER_ID: 1b47f5de842509be1b3efe91f0254870f158ff4faecdf9723b7904a3b4486cbe 2024-11-01T17:02:03.8394831Z ##[endgroup] 2024-11-01T17:02:04.2388690Z NOTE: s3-prefix specified, ignoring name parameter 2024-11-01T17:02:04.2389416Z With the provided path, there will be 1 file uploaded 2024-11-01T17:02:04.2390143Z Uploading to s3 prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T17:02:04.2432228Z Starting upload of logs-test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_32397035132.zip 2024-11-01T17:02:04.3601390Z Finished upload of logs-test-distributed-3-3-linux.g4dn.12xlarge.nvidia.gpu_32397035132.zip 2024-11-01T17:02:04.3789780Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-11-01T17:02:04.3790291Z with: 2024-11-01T17:02:04.3790584Z s3-bucket: gha-artifacts 2024-11-01T17:02:04.3791084Z s3-prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T17:02:04.3791626Z retention-days: 14 2024-11-01T17:02:04.3791996Z if-no-files-found: ignore 2024-11-01T17:02:04.3792396Z path: debug-*.zip 2024-11-01T17:02:04.3792832Z name: artifact 2024-11-01T17:02:04.3793154Z region: us-east-1 2024-11-01T17:02:04.3793592Z env: 2024-11-01T17:02:04.3793868Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:02:04.3794352Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:02:04.3795190Z DOCKER_CONTAINER_ID: 1b47f5de842509be1b3efe91f0254870f158ff4faecdf9723b7904a3b4486cbe 2024-11-01T17:02:04.3796086Z ##[endgroup] 2024-11-01T17:02:04.7715264Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-11-01T17:02:04.7909189Z ##[group]Run # shellcheck disable=SC2156 2024-11-01T17:02:04.7909734Z # shellcheck disable=SC2156 2024-11-01T17:02:04.7910693Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-11-01T17:02:04.7917964Z shell: /usr/bin/bash -e {0} 2024-11-01T17:02:04.7918324Z env: 2024-11-01T17:02:04.7918586Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:02:04.7919060Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:02:04.7919861Z DOCKER_CONTAINER_ID: 1b47f5de842509be1b3efe91f0254870f158ff4faecdf9723b7904a3b4486cbe 2024-11-01T17:02:04.7920735Z ##[endgroup] 2024-11-01T17:02:05.0408974Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-11-01T17:02:05.0409543Z with: 2024-11-01T17:02:05.0409797Z env: 2024-11-01T17:02:05.0410066Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:02:05.0410531Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:02:05.0411321Z DOCKER_CONTAINER_ID: 1b47f5de842509be1b3efe91f0254870f158ff4faecdf9723b7904a3b4486cbe 2024-11-01T17:02:05.0412014Z ##[endgroup] 2024-11-01T17:02:05.0434898Z ##[group]Run set -eou pipefail 2024-11-01T17:02:05.0435302Z set -eou pipefail 2024-11-01T17:02:05.0435650Z  2024-11-01T17:02:05.0436171Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-11-01T17:02:05.0436841Z for _ in $(seq 1440); do 2024-11-01T17:02:05.0437320Z  # Break if no ssh session exists anymore 2024-11-01T17:02:05.0437818Z  if [ "$(who)" = "" ]; then 2024-11-01T17:02:05.0438235Z  break 2024-11-01T17:02:05.0438590Z  fi 2024-11-01T17:02:05.0438887Z  echo "." 2024-11-01T17:02:05.0439197Z  sleep 5 2024-11-01T17:02:05.0439510Z done 2024-11-01T17:02:05.0445173Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:02:05.0445690Z env: 2024-11-01T17:02:05.0445962Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:02:05.0446605Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:02:05.0447432Z DOCKER_CONTAINER_ID: 1b47f5de842509be1b3efe91f0254870f158ff4faecdf9723b7904a3b4486cbe 2024-11-01T17:02:05.0448240Z ##[endgroup] 2024-11-01T17:02:05.0472880Z Holding runner for 2 hours until all ssh sessions have logged out 2024-11-01T17:02:05.0539515Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-11-01T17:02:05.0540373Z # ignore expansion of "docker ps -q" since it could be empty 2024-11-01T17:02:05.0541127Z # shellcheck disable=SC2046 2024-11-01T17:02:05.0541809Z docker stop $(docker ps -q) || true 2024-11-01T17:02:05.0542378Z # Prune all of the docker images 2024-11-01T17:02:05.0542897Z docker system prune -af 2024-11-01T17:02:05.0549054Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:02:05.0549631Z env: 2024-11-01T17:02:05.0549935Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:02:05.0550445Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:02:05.0551331Z DOCKER_CONTAINER_ID: 1b47f5de842509be1b3efe91f0254870f158ff4faecdf9723b7904a3b4486cbe 2024-11-01T17:02:05.0552113Z ##[endgroup] 2024-11-01T17:02:05.5752903Z 1b47f5de8425 2024-11-01T17:02:05.9978347Z Deleted Containers: 2024-11-01T17:02:05.9978973Z 1b47f5de842509be1b3efe91f0254870f158ff4faecdf9723b7904a3b4486cbe 2024-11-01T17:02:05.9979516Z 2024-11-01T17:02:10.9379260Z Deleted Images: 2024-11-01T17:02:10.9381301Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T17:02:10.9383909Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9@sha256:7e55ca35c7b3a515f97404dcaef8e9d5373f6e358ac63c424d2d80dda1e618f7 2024-11-01T17:02:10.9385684Z deleted: sha256:f4475432de42c25077a2ffe0f2f56efb42124dedca9504f01daf084f9103394b 2024-11-01T17:02:10.9386740Z deleted: sha256:0a3b02914acc05dcfc3fdaea37e1a4336aaa696c207e6e993e87815010457060 2024-11-01T17:02:10.9387774Z deleted: sha256:287a6b6bfe47fa7d50e7409829056a1cf37ad073cb6890163e779734ff296fb2 2024-11-01T17:02:10.9388795Z deleted: sha256:53fa61dc822c6716c85f6363bb6a83cd7541c8879d5f9b93dea1064af2d6463c 2024-11-01T17:02:10.9389796Z deleted: sha256:1a8f08fa472d2fb692f63de804eaaa893862346178152979127e0609eb12d696 2024-11-01T17:02:10.9390814Z deleted: sha256:a1b5599bc63b031b0d8a037d7dd06b8878dc70ed0692585d42de476263f63af3 2024-11-01T17:02:10.9391827Z deleted: sha256:f8784943057bdd6a3eec2f0816741a058296e19c92a8ec86595e225509b9d244 2024-11-01T17:02:10.9393455Z deleted: sha256:f62e2601c7c54fd6b496f2b9162da63bc6078f0b98bcc590c8438df3191843b0 2024-11-01T17:02:10.9394443Z deleted: sha256:429b19ab4950a882a2573ce19266e5e84f52dd115bf27e8a23188bae420b5ec2 2024-11-01T17:02:10.9395407Z deleted: sha256:2a955783b858a5e071968bf711c0c32f0932a3ca65a4421074551c69d0bbf98f 2024-11-01T17:02:10.9396395Z deleted: sha256:de6fbaec1030affac12cf8d0d4a3728e048f86288dd421ff999b7f3ee058f0b6 2024-11-01T17:02:10.9397390Z deleted: sha256:381f67e4868ab208466a6d548b250ab4f4a05944a6b27ff3bbee4afef20473ca 2024-11-01T17:02:10.9398382Z deleted: sha256:8c153bbd827867cdf94b1c4237be87df7196b4b4b2043e4eb08fffa5af35245d 2024-11-01T17:02:10.9399359Z deleted: sha256:0e72c2f47638416e5761d4a2b6723d0c2176a3232a9d65650d6f2774c1b1a7bb 2024-11-01T17:02:10.9400335Z deleted: sha256:1af9aaf89749d8b264269a0da14092f9bb6ec46db7db6ab6c9edb2b7eacc1e36 2024-11-01T17:02:10.9401353Z deleted: sha256:b7fe349ffe9a77b7cd8df359c0d4f447eab8c3c5fe1d1f8474d5421fae0b52e7 2024-11-01T17:02:10.9402360Z deleted: sha256:89d4a9d4808254c489a6774ebe1a7fbe7a921a48e2867b0370db5ae6643ef591 2024-11-01T17:02:10.9403339Z deleted: sha256:a3e0a41d4550f3243e4b20bdb204324bcbcea605d9ecd04e8f3f1af843f57e86 2024-11-01T17:02:10.9404332Z deleted: sha256:019e53db4314f4ba6148551c897d2d50cdceda1a96a4f03554208af0e99de4b8 2024-11-01T17:02:10.9405308Z deleted: sha256:383583432ea93d7b02c0844ca27319cab73bfc40c49d3f0e1e04295ccbcd9662 2024-11-01T17:02:10.9406311Z deleted: sha256:af4dadd55bfef20fdd6e7e3ce936c5930b3a4030c6dd380e41c57ac57fa6d351 2024-11-01T17:02:10.9407339Z deleted: sha256:df578b0221ae1fdd0a5b9b02c806ab3ee607208c6c0e8b75876caff585bc6177 2024-11-01T17:02:10.9408337Z deleted: sha256:29baf54d9e67bc5615a74704ba363598e29ae1314b34fa4ca5c843f2b5fb1265 2024-11-01T17:02:10.9409312Z deleted: sha256:7b401ef5b5879bdc37e726876c721fbbbe5f4f7721c236560a37cccb8f8ddb32 2024-11-01T17:02:10.9410337Z deleted: sha256:3c65fa1a83a01d453087f56b23a9cdda9218e1cdfe05500cbfcb1e753a3ab978 2024-11-01T17:02:10.9411368Z deleted: sha256:008280bac2ca227cafab11733314d6f662de2f70bb060483e181e5634e8a347f 2024-11-01T17:02:10.9412384Z deleted: sha256:5d4a60ee9e3f8449e187468813913fda8f5408c95b59600b5508ac77126c045e 2024-11-01T17:02:10.9413371Z deleted: sha256:f0b9826ef432a31e2e43af6d32313a2a4ae3557a3636f010eb150f5fb5dfbbff 2024-11-01T17:02:10.9414362Z deleted: sha256:7e4497566413d4a3599e7034506c24a6ab96193b4647a7c214f50295649f5291 2024-11-01T17:02:10.9415316Z deleted: sha256:930abad7c169e40236fa09a9608a0ba7e8e9a966686a6133b78bd660bdfb9a59 2024-11-01T17:02:10.9416264Z deleted: sha256:77f2520779046086794a6a4f24626f4f688c329e9cee7de2d22254477299c457 2024-11-01T17:02:10.9417219Z deleted: sha256:67e6ef1843c893be2880063dee3735bf00e40894ef866be07f61b8d59cf3755a 2024-11-01T17:02:10.9418190Z deleted: sha256:2ddf875758f00a6403c9e203e755cc185eca0f9e423849e5700fdb076dfc562d 2024-11-01T17:02:10.9419178Z deleted: sha256:68d0c60c82c5f5afa713dd4784be9baa1f588998b8e108ce0ece68f6e976d122 2024-11-01T17:02:10.9420175Z deleted: sha256:c3d898e584b844147299b6b3eaf2ea2cb63b4e93f3640a7b52baa64e770bb5f3 2024-11-01T17:02:10.9421276Z deleted: sha256:b46bb13fe8c4577e03ee179b7042226178e6cd61eaeb80445a16d25b1bb1f35f 2024-11-01T17:02:10.9422440Z deleted: sha256:42cc15f51fb1309300408cfaf0056e28b11de1a0f6dc033c99a943473619e1fb 2024-11-01T17:02:10.9423454Z deleted: sha256:4418b6a425adc3a7406a74e6b47bd7f3788ad275193db6716e23729b892ce0a8 2024-11-01T17:02:10.9424470Z deleted: sha256:7c58b3261a79180e66c7932cc232845b4dbdaffc908c0f0a2a2cde84950ad553 2024-11-01T17:02:10.9425514Z deleted: sha256:ea3f9a836caadfa9bd915005c979b0febc5334ae0a607daae43e85f77cc72474 2024-11-01T17:02:10.9426564Z deleted: sha256:decc9ca252ac120ee1858488ad90a3f707afd34238019bd1c51e97b4d09c45f6 2024-11-01T17:02:10.9427594Z deleted: sha256:e033e0e1b98c30aaad1bf6c6c2434254968b11262e9c6e46b56eb4ebdb74ab51 2024-11-01T17:02:10.9428599Z deleted: sha256:f8a7f2c4c34a3c04a982d753851c220e47a6f1049b57f2a42b4144d0b6a17b9e 2024-11-01T17:02:10.9429612Z deleted: sha256:88766d7f595185d3bc2ff7a2476931bda2cdac996c0d13db9a95b301b2db75a9 2024-11-01T17:02:10.9430842Z deleted: sha256:5bf533e94db3a248573dcbfdf6a58089e558f5f0f912277788bce3b54545d01c 2024-11-01T17:02:10.9431876Z deleted: sha256:8d77e0f9c2de0b3a7b5bdc2992ad1034b843b2eb67d9f262f43160f55884bab8 2024-11-01T17:02:10.9432900Z deleted: sha256:54bdbdf9f9e33647140f0c0f8c1dc00c526479f42964765338b8f79a1049ee9c 2024-11-01T17:02:10.9434089Z deleted: sha256:2359a12b67991edef98665db87a5192ba02e183ee2232e283e6b58c94f8d3826 2024-11-01T17:02:10.9435083Z deleted: sha256:7727c4d3952195e6aaf87b225a3f67b89642a424451e601618fa1e7b52325cd7 2024-11-01T17:02:10.9435952Z deleted: sha256:6f9f382dd9bcb89f97553031ee64253426902b4910f2556a2bea7881309c376c 2024-11-01T17:02:10.9436850Z deleted: sha256:c35ad5f713dc248c630d50c7520cd35ba1ab936116cf355e419c3fbe74909a04 2024-11-01T17:02:10.9437748Z deleted: sha256:4a5efa02522a748ca3722e8b7c0142d4d2291584b8d195e63da582862a967528 2024-11-01T17:02:10.9438647Z deleted: sha256:a30107d370bc5fa598ef737c0c53419acc51af7ab8ef64d75696fae37670abcb 2024-11-01T17:02:10.9439570Z deleted: sha256:672ec49fd713d9e0cdfb23310a314ebf467b0a33d23f64599816ca4684cb57f5 2024-11-01T17:02:10.9440480Z deleted: sha256:b4de36d9c64ae16e75798019674dcb2fade5e6653e5095c2dfd2846384786772 2024-11-01T17:02:10.9441377Z deleted: sha256:20d847bb7af881db42903f7d4166e9e372334ebdfb3d1f34b8ef669623d0da58 2024-11-01T17:02:10.9442265Z deleted: sha256:c34468395058a30a439e859a7bd5133e780faa762a11e025e0fdeb88627550e4 2024-11-01T17:02:10.9443154Z deleted: sha256:27f5680f66a0d01ec1d74ae03418fcb2e03db96605d55e0d8933a40acf0def6c 2024-11-01T17:02:10.9444055Z deleted: sha256:9a974b67ee0b37916f8a38a233be94f2fa8205ec477fdee5d06905af1cb34a9c 2024-11-01T17:02:10.9444956Z deleted: sha256:a7e98e6ff938d43862e03ca9edb22288389b8a503b60d8b72b26562de329e745 2024-11-01T17:02:10.9445857Z deleted: sha256:f8dfc84a271d94297e3a777f3455db814bebc4c56d31dc2b589320a70c5b3e01 2024-11-01T17:02:10.9446757Z deleted: sha256:cb9ee3913530d2683b184f99aaeaafade4ee059d8d83ca203fe890c9b71a39ce 2024-11-01T17:02:10.9447667Z deleted: sha256:cf8424e79315a3b2ffb71729abf912d62e257dfe818f3c47b09812733723a948 2024-11-01T17:02:10.9448582Z deleted: sha256:22de19c76bb02aa83af513603ea7b11c598a75ea3d120371286253e6ce67f0f0 2024-11-01T17:02:10.9449485Z deleted: sha256:e3bb91f1c5e44b1d35320ec7e5220af244c47d23a888f50508efbb919f0b7b29 2024-11-01T17:02:10.9450388Z deleted: sha256:08c5645ed9673839aa7a6eae9119ab5e407ecce85b3b5466607c5a651c3f2920 2024-11-01T17:02:10.9451305Z deleted: sha256:32c37789f4e3eaa19d0b7d08bfd1b4a1fed982cbeedccde408a3e3e2afaa1f9e 2024-11-01T17:02:10.9452218Z deleted: sha256:8b0aee0f411b6479db05dfa28910475b3cbbb2e08c3b048b78a43939c48c7419 2024-11-01T17:02:10.9453133Z deleted: sha256:2f7d309060abdcfff184b9086398b192de12a03af2a8e018c807cc6fa89ec403 2024-11-01T17:02:10.9454043Z deleted: sha256:fa738ffc39690ea6d3e51036aaa9acff34100a51d635e157817603eb79dee259 2024-11-01T17:02:10.9454952Z deleted: sha256:07a2164b82cf81c6f9f6fc6859a6d498203b5e0fb9ecd9878bf00de3c4c9949d 2024-11-01T17:02:10.9455849Z deleted: sha256:89edf8c906678c217b9a88160e1561991f53909c3fd27a4a5e1f0ca0f7ff7888 2024-11-01T17:02:10.9456771Z deleted: sha256:d5ba67f8f9b0c2b6da06d2d5d9d5fedc9f17cbe1a5e13a1a4033a4619f4528ba 2024-11-01T17:02:10.9457677Z deleted: sha256:5d61311e1a293c0c9642e34871719525dfeec7028e4b45e08344a469e3fe24bf 2024-11-01T17:02:10.9458556Z deleted: sha256:b89bb70f745a94c0edd5e974aee294e160735f76fd666a1a06a0b9b1be3caf63 2024-11-01T17:02:10.9459472Z deleted: sha256:9e5477f1f21a38c3dfe21a56a7d77d1b1a42453aed3e11729afcbb9d5731770b 2024-11-01T17:02:10.9460399Z deleted: sha256:b76f4af91cffe2bb7b3f09960f5bdbb26f741419defda65c451c4407fc5b0a92 2024-11-01T17:02:10.9461545Z deleted: sha256:3f8748a00ebd394c0e9d9d481972d764dff58896841ef0150b3f7efaac0f4a4a 2024-11-01T17:02:10.9462554Z deleted: sha256:13989c771e3ba4d40f618344e55e5cb7062a35041b5efd69e2f740ba819f46ee 2024-11-01T17:02:10.9463579Z deleted: sha256:9ebb27837f6c67ad8c17e9af4e63c008faae03fa226c3641dd8bbbfeefb106c4 2024-11-01T17:02:10.9464580Z deleted: sha256:8c16907d735950f8093582bc3cf392b70e002456bf9c4224de5a5b86b78b3ea2 2024-11-01T17:02:10.9466601Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-11-01T17:02:10.9467232Z 2024-11-01T17:02:10.9467389Z Total reclaimed space: 34.51GB 2024-11-01T17:02:10.9552831Z Post job cleanup. 2024-11-01T17:02:10.9611424Z Post job cleanup. 2024-11-01T17:02:11.0581166Z [command]/usr/bin/git version 2024-11-01T17:02:11.0630013Z git version 2.40.1 2024-11-01T17:02:11.0671931Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/75a00056-2db5-41b6-bacf-a59ab74ac507' before making global git config changes 2024-11-01T17:02:11.0673577Z Adding repository directory to the temporary git global config as a safe directory 2024-11-01T17:02:11.0677485Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-11-01T17:02:11.0717687Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-11-01T17:02:11.0755235Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-11-01T17:02:11.1088666Z Entering 'android/libs/fbjni' 2024-11-01T17:02:11.1146744Z Entering 'third_party/FP16' 2024-11-01T17:02:11.1205197Z Entering 'third_party/FXdiv' 2024-11-01T17:02:11.1262023Z Entering 'third_party/NNPACK' 2024-11-01T17:02:11.1322025Z Entering 'third_party/NVTX' 2024-11-01T17:02:11.1381863Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T17:02:11.1440395Z Entering 'third_party/XNNPACK' 2024-11-01T17:02:11.1516318Z Entering 'third_party/benchmark' 2024-11-01T17:02:11.1572708Z Entering 'third_party/composable_kernel' 2024-11-01T17:02:11.1640396Z Entering 'third_party/cpp-httplib' 2024-11-01T17:02:11.1698205Z Entering 'third_party/cpuinfo' 2024-11-01T17:02:11.1757323Z Entering 'third_party/cudnn_frontend' 2024-11-01T17:02:11.1815060Z Entering 'third_party/cutlass' 2024-11-01T17:02:11.1880889Z Entering 'third_party/eigen' 2024-11-01T17:02:11.1942320Z Entering 'third_party/fbgemm' 2024-11-01T17:02:11.2000630Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T17:02:11.2061478Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T17:02:11.2121183Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T17:02:11.2184805Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T17:02:11.2242307Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T17:02:11.2299750Z Entering 'third_party/flatbuffers' 2024-11-01T17:02:11.2361866Z Entering 'third_party/fmt' 2024-11-01T17:02:11.2420580Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T17:02:11.2478280Z Entering 'third_party/gloo' 2024-11-01T17:02:11.2540213Z Entering 'third_party/googletest' 2024-11-01T17:02:11.2598219Z Entering 'third_party/ideep' 2024-11-01T17:02:11.2653684Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T17:02:11.2724115Z Entering 'third_party/ittapi' 2024-11-01T17:02:11.2779651Z Entering 'third_party/kineto' 2024-11-01T17:02:11.2838840Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T17:02:11.2895755Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T17:02:11.2953138Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T17:02:11.3012103Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T17:02:11.3067542Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T17:02:11.3124185Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T17:02:11.3183696Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T17:02:11.3241699Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T17:02:11.3299685Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T17:02:11.3362909Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T17:02:11.3421954Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T17:02:11.3481571Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T17:02:11.3539841Z Entering 'third_party/mimalloc' 2024-11-01T17:02:11.3597740Z Entering 'third_party/nccl/nccl' 2024-11-01T17:02:11.3655616Z Entering 'third_party/nlohmann' 2024-11-01T17:02:11.3718797Z Entering 'third_party/onnx' 2024-11-01T17:02:11.3794573Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T17:02:11.3855683Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T17:02:11.3918311Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T17:02:11.3974256Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T17:02:11.4033824Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T17:02:11.4093138Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T17:02:11.4149330Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T17:02:11.4205461Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T17:02:11.4262363Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T17:02:11.4320202Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T17:02:11.4381596Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T17:02:11.4442444Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T17:02:11.4522108Z Entering 'third_party/pocketfft' 2024-11-01T17:02:11.4578604Z Entering 'third_party/protobuf' 2024-11-01T17:02:11.4640295Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T17:02:11.4696395Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T17:02:11.4760921Z Entering 'third_party/psimd' 2024-11-01T17:02:11.4819053Z Entering 'third_party/pthreadpool' 2024-11-01T17:02:11.4878744Z Entering 'third_party/pybind11' 2024-11-01T17:02:11.4936946Z Entering 'third_party/python-peachpy' 2024-11-01T17:02:11.4996695Z Entering 'third_party/sleef' 2024-11-01T17:02:11.5052639Z Entering 'third_party/tensorpipe' 2024-11-01T17:02:11.5111126Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T17:02:11.5171926Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T17:02:11.5225813Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T17:02:11.5280870Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T17:02:11.5340366Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T17:02:11.5416737Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-11-01T17:02:11.5448237Z http.https://github.com/.extraheader 2024-11-01T17:02:11.5455951Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-11-01T17:02:11.5495041Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-11-01T17:02:11.5819563Z Entering 'android/libs/fbjni' 2024-11-01T17:02:11.5856567Z http.https://github.com/.extraheader 2024-11-01T17:02:11.5901682Z Entering 'third_party/FP16' 2024-11-01T17:02:11.5941811Z http.https://github.com/.extraheader 2024-11-01T17:02:11.5979729Z Entering 'third_party/FXdiv' 2024-11-01T17:02:11.6019250Z http.https://github.com/.extraheader 2024-11-01T17:02:11.6059225Z Entering 'third_party/NNPACK' 2024-11-01T17:02:11.6098783Z http.https://github.com/.extraheader 2024-11-01T17:02:11.6135979Z Entering 'third_party/NVTX' 2024-11-01T17:02:11.6175038Z http.https://github.com/.extraheader 2024-11-01T17:02:11.6212159Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T17:02:11.6250370Z http.https://github.com/.extraheader 2024-11-01T17:02:11.6285486Z Entering 'third_party/XNNPACK' 2024-11-01T17:02:11.6324754Z http.https://github.com/.extraheader 2024-11-01T17:02:11.6378814Z Entering 'third_party/benchmark' 2024-11-01T17:02:11.6416893Z http.https://github.com/.extraheader 2024-11-01T17:02:11.6453572Z Entering 'third_party/composable_kernel' 2024-11-01T17:02:11.6494121Z http.https://github.com/.extraheader 2024-11-01T17:02:11.6537083Z Entering 'third_party/cpp-httplib' 2024-11-01T17:02:11.6575355Z http.https://github.com/.extraheader 2024-11-01T17:02:11.6611936Z Entering 'third_party/cpuinfo' 2024-11-01T17:02:11.6649824Z http.https://github.com/.extraheader 2024-11-01T17:02:11.6685037Z Entering 'third_party/cudnn_frontend' 2024-11-01T17:02:11.6725249Z http.https://github.com/.extraheader 2024-11-01T17:02:11.6762440Z Entering 'third_party/cutlass' 2024-11-01T17:02:11.6801758Z http.https://github.com/.extraheader 2024-11-01T17:02:11.6849066Z Entering 'third_party/eigen' 2024-11-01T17:02:11.6887375Z http.https://github.com/.extraheader 2024-11-01T17:02:11.6927005Z Entering 'third_party/fbgemm' 2024-11-01T17:02:11.6966019Z http.https://github.com/.extraheader 2024-11-01T17:02:11.7002609Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T17:02:11.7039603Z http.https://github.com/.extraheader 2024-11-01T17:02:11.7079305Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T17:02:11.7118392Z http.https://github.com/.extraheader 2024-11-01T17:02:11.7153323Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T17:02:11.7191395Z http.https://github.com/.extraheader 2024-11-01T17:02:11.7235419Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T17:02:11.7272412Z http.https://github.com/.extraheader 2024-11-01T17:02:11.7311591Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T17:02:11.7349943Z http.https://github.com/.extraheader 2024-11-01T17:02:11.7391117Z Entering 'third_party/flatbuffers' 2024-11-01T17:02:11.7430805Z http.https://github.com/.extraheader 2024-11-01T17:02:11.7472374Z Entering 'third_party/fmt' 2024-11-01T17:02:11.7510657Z http.https://github.com/.extraheader 2024-11-01T17:02:11.7552168Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T17:02:11.7591082Z http.https://github.com/.extraheader 2024-11-01T17:02:11.7626200Z Entering 'third_party/gloo' 2024-11-01T17:02:11.7663220Z http.https://github.com/.extraheader 2024-11-01T17:02:11.7699467Z Entering 'third_party/googletest' 2024-11-01T17:02:11.7738465Z http.https://github.com/.extraheader 2024-11-01T17:02:11.7776211Z Entering 'third_party/ideep' 2024-11-01T17:02:11.7814196Z http.https://github.com/.extraheader 2024-11-01T17:02:11.7848862Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T17:02:11.7885085Z http.https://github.com/.extraheader 2024-11-01T17:02:11.7931042Z Entering 'third_party/ittapi' 2024-11-01T17:02:11.7969578Z http.https://github.com/.extraheader 2024-11-01T17:02:11.8005970Z Entering 'third_party/kineto' 2024-11-01T17:02:11.8043493Z http.https://github.com/.extraheader 2024-11-01T17:02:11.8082238Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T17:02:11.8120308Z http.https://github.com/.extraheader 2024-11-01T17:02:11.8161172Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T17:02:11.8198796Z http.https://github.com/.extraheader 2024-11-01T17:02:11.8242398Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T17:02:11.8279992Z http.https://github.com/.extraheader 2024-11-01T17:02:11.8321968Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T17:02:11.8359202Z http.https://github.com/.extraheader 2024-11-01T17:02:11.8401421Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T17:02:11.8442981Z http.https://github.com/.extraheader 2024-11-01T17:02:11.8480831Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T17:02:11.8519012Z http.https://github.com/.extraheader 2024-11-01T17:02:11.8564694Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T17:02:11.8602937Z http.https://github.com/.extraheader 2024-11-01T17:02:11.8639712Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T17:02:11.8677301Z http.https://github.com/.extraheader 2024-11-01T17:02:11.8713058Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T17:02:11.8751158Z http.https://github.com/.extraheader 2024-11-01T17:02:11.8795475Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T17:02:11.8831468Z http.https://github.com/.extraheader 2024-11-01T17:02:11.8868877Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T17:02:11.8906991Z http.https://github.com/.extraheader 2024-11-01T17:02:11.8950575Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T17:02:11.8985304Z http.https://github.com/.extraheader 2024-11-01T17:02:11.9029981Z Entering 'third_party/mimalloc' 2024-11-01T17:02:11.9066815Z http.https://github.com/.extraheader 2024-11-01T17:02:11.9103029Z Entering 'third_party/nccl/nccl' 2024-11-01T17:02:11.9140342Z http.https://github.com/.extraheader 2024-11-01T17:02:11.9177589Z Entering 'third_party/nlohmann' 2024-11-01T17:02:11.9215592Z http.https://github.com/.extraheader 2024-11-01T17:02:11.9253130Z Entering 'third_party/onnx' 2024-11-01T17:02:11.9293156Z http.https://github.com/.extraheader 2024-11-01T17:02:11.9343171Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T17:02:11.9379461Z http.https://github.com/.extraheader 2024-11-01T17:02:11.9422538Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T17:02:11.9460465Z http.https://github.com/.extraheader 2024-11-01T17:02:11.9500663Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T17:02:11.9537691Z http.https://github.com/.extraheader 2024-11-01T17:02:11.9573608Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T17:02:11.9612871Z http.https://github.com/.extraheader 2024-11-01T17:02:11.9648246Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T17:02:11.9685618Z http.https://github.com/.extraheader 2024-11-01T17:02:11.9721937Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T17:02:11.9758170Z http.https://github.com/.extraheader 2024-11-01T17:02:11.9796990Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T17:02:11.9834713Z http.https://github.com/.extraheader 2024-11-01T17:02:11.9868371Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T17:02:11.9906371Z http.https://github.com/.extraheader 2024-11-01T17:02:11.9943157Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T17:02:11.9979026Z http.https://github.com/.extraheader 2024-11-01T17:02:12.0015958Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T17:02:12.0053534Z http.https://github.com/.extraheader 2024-11-01T17:02:12.0093034Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T17:02:12.0129878Z http.https://github.com/.extraheader 2024-11-01T17:02:12.0167642Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T17:02:12.0205359Z http.https://github.com/.extraheader 2024-11-01T17:02:12.0262983Z Entering 'third_party/pocketfft' 2024-11-01T17:02:12.0302605Z http.https://github.com/.extraheader 2024-11-01T17:02:12.0341422Z Entering 'third_party/protobuf' 2024-11-01T17:02:12.0378257Z http.https://github.com/.extraheader 2024-11-01T17:02:12.0418656Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T17:02:12.0456786Z http.https://github.com/.extraheader 2024-11-01T17:02:12.0494574Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T17:02:12.0532565Z http.https://github.com/.extraheader 2024-11-01T17:02:12.0571077Z Entering 'third_party/psimd' 2024-11-01T17:02:12.0610772Z http.https://github.com/.extraheader 2024-11-01T17:02:12.0646709Z Entering 'third_party/pthreadpool' 2024-11-01T17:02:12.0685409Z http.https://github.com/.extraheader 2024-11-01T17:02:12.0721948Z Entering 'third_party/pybind11' 2024-11-01T17:02:12.0761011Z http.https://github.com/.extraheader 2024-11-01T17:02:12.0798422Z Entering 'third_party/python-peachpy' 2024-11-01T17:02:12.0837152Z http.https://github.com/.extraheader 2024-11-01T17:02:12.0872071Z Entering 'third_party/sleef' 2024-11-01T17:02:12.0915267Z http.https://github.com/.extraheader 2024-11-01T17:02:12.0953041Z Entering 'third_party/tensorpipe' 2024-11-01T17:02:12.0992086Z http.https://github.com/.extraheader 2024-11-01T17:02:12.1028867Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T17:02:12.1065964Z http.https://github.com/.extraheader 2024-11-01T17:02:12.1102768Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T17:02:12.1139612Z http.https://github.com/.extraheader 2024-11-01T17:02:12.1177901Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T17:02:12.1216074Z http.https://github.com/.extraheader 2024-11-01T17:02:12.1252723Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T17:02:12.1293134Z http.https://github.com/.extraheader 2024-11-01T17:02:12.1328209Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T17:02:12.1366802Z http.https://github.com/.extraheader 2024-11-01T17:02:12.1498589Z A job completed hook has been configured by the self-hosted runner administrator 2024-11-01T17:02:12.1521200Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-11-01T17:02:12.1527043Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:02:12.1527612Z ##[endgroup] 2024-11-01T17:02:19.1603646Z Cleaning up orphan processes