2024-11-01T16:42:17.5930240Z Current runner version: '2.320.0' 2024-11-01T16:42:17.5937055Z Runner name: 'i-06c394753ea076091' 2024-11-01T16:42:17.5937894Z Runner group name: 'Default' 2024-11-01T16:42:17.5938731Z Machine name: 'ip-10-0-7-207' 2024-11-01T16:42:17.5956443Z Testing runner upgrade compatibility 2024-11-01T16:42:17.7134065Z ##[group]GITHUB_TOKEN Permissions 2024-11-01T16:42:17.7136475Z Actions: read 2024-11-01T16:42:17.7137092Z Attestations: read 2024-11-01T16:42:17.7137689Z Checks: read 2024-11-01T16:42:17.7138139Z Contents: read 2024-11-01T16:42:17.7138651Z Deployments: read 2024-11-01T16:42:17.7139198Z Discussions: read 2024-11-01T16:42:17.7139702Z Issues: read 2024-11-01T16:42:17.7140189Z Metadata: read 2024-11-01T16:42:17.7140682Z Packages: read 2024-11-01T16:42:17.7141187Z Pages: read 2024-11-01T16:42:17.7141660Z PullRequests: read 2024-11-01T16:42:17.7142291Z RepositoryProjects: read 2024-11-01T16:42:17.7142908Z SecurityEvents: read 2024-11-01T16:42:17.7143406Z Statuses: read 2024-11-01T16:42:17.7144136Z ##[endgroup] 2024-11-01T16:42:17.7147872Z Secret source: Actions 2024-11-01T16:42:17.7148903Z Prepare workflow directory 2024-11-01T16:42:18.0878279Z Prepare all required actions 2024-11-01T16:42:18.1061457Z Getting action download info 2024-11-01T16:42:18.3140240Z Download action repository 'pytorch/test-infra@main' (SHA:49fb39b5efb49007791d74b09885044cd64544bf) 2024-11-01T16:42:18.6765603Z Download action repository 'pytorch/pytorch@main' (SHA:d79c5143d8e625947b93f8821bbe181419f47dee) 2024-11-01T16:42:22.0143542Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-11-01T16:42:22.1660328Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-11-01T16:42:22.4091311Z Getting action download info 2024-11-01T16:42:22.5119877Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-11-01T16:42:22.6723205Z Getting action download info 2024-11-01T16:42:22.7665798Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2024-11-01T16:42:22.9107863Z Getting action download info 2024-11-01T16:42:23.0480902Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-11-01T16:42:23.1895301Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/138766/merge (368da7510f5d755034681777a4e5ae6d33c07b38) 2024-11-01T16:42:23.1898182Z ##[group] Inputs 2024-11-01T16:42:23.1898703Z build-environment: linux-focal-cuda11.8-py3.10-gcc9 2024-11-01T16:42:23.1900794Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}]} 2024-11-01T16:42:23.1903814Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:42:23.1905072Z sync-tag: 2024-11-01T16:42:23.1905959Z timeout-minutes: 360 2024-11-01T16:42:23.1906327Z use-gha: 2024-11-01T16:42:23.1906638Z dashboard-tag: 2024-11-01T16:42:23.1906989Z s3-bucket: gha-artifacts 2024-11-01T16:42:23.1907385Z aws-role-to-assume: 2024-11-01T16:42:23.1907743Z ##[endgroup] 2024-11-01T16:42:23.1908899Z Complete job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-11-01T16:42:23.2794416Z A job started hook has been configured by the self-hosted runner administrator 2024-11-01T16:42:23.2952007Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-11-01T16:42:23.2962088Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:42:23.2962641Z ##[endgroup] 2024-11-01T16:42:24.7853270Z Runner Type: linux.g4dn.12xlarge.nvidia.gpu 2024-11-01T16:42:24.7854257Z Instance Type: g4dn.12xlarge 2024-11-01T16:42:24.7854634Z AMI Name: unknown 2024-11-01T16:42:24.7855183Z AMI ID: ami-0fff1b9a61dec8a5f 2024-11-01T16:42:30.6824838Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-11-01T16:42:30.6825534Z with: 2024-11-01T16:42:30.6826328Z github-secret: *** 2024-11-01T16:42:30.6827442Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-11-01T16:42:30.6828870Z activate-with-label: false 2024-11-01T16:42:30.6829251Z label: with-ssh 2024-11-01T16:42:30.6829570Z remove-existing-keys: true 2024-11-01T16:42:30.6829957Z fail-silently: true 2024-11-01T16:42:30.6830277Z env: 2024-11-01T16:42:30.6830554Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:42:30.6830915Z ##[endgroup] 2024-11-01T16:42:30.7740754Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-11-01T16:42:31.0313029Z Grabbing public ssh keys from https://github.com/c00w.keys 2024-11-01T16:42:31.0794941Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-11-01T16:42:31.0808879Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-11-01T16:42:31.0836750Z Login using: ssh ec2-user@ec2-52-87-208-166.compute-1.amazonaws.com 2024-11-01T16:42:31.0837651Z All testing is done inside the container, to start an interactive session run: 2024-11-01T16:42:31.0838601Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-11-01T16:42:31.0965822Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-11-01T16:42:31.0966391Z with: 2024-11-01T16:42:31.0966649Z submodules: recursive 2024-11-01T16:42:31.0966983Z fetch-depth: 0 2024-11-01T16:42:31.0967264Z env: 2024-11-01T16:42:31.0967528Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:42:31.0967871Z ##[endgroup] 2024-11-01T16:42:31.1168878Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:42:31.1170262Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:42:31.1178035Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:42:31.1178621Z env: 2024-11-01T16:42:31.1178924Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:42:31.1179313Z ##[endgroup] 2024-11-01T16:42:31.1271320Z ##[group]Run retry () { 2024-11-01T16:42:31.1271728Z retry () { 2024-11-01T16:42:31.1272230Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-11-01T16:42:31.1272789Z } 2024-11-01T16:42:31.1273092Z echo "${GITHUB_WORKSPACE}" 2024-11-01T16:42:31.1273519Z if [ -z "${NO_SUDO}" ]; then 2024-11-01T16:42:31.1274001Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-11-01T16:42:31.1274463Z else 2024-11-01T16:42:31.1274815Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-11-01T16:42:31.1275232Z fi 2024-11-01T16:42:31.1275529Z mkdir "${GITHUB_WORKSPACE}" 2024-11-01T16:42:31.1281347Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:42:31.1281856Z env: 2024-11-01T16:42:31.1282118Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:42:31.1282443Z NO_SUDO: 2024-11-01T16:42:31.1282771Z ##[endgroup] 2024-11-01T16:42:31.1307779Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-11-01T16:42:33.6504304Z ##[group]Run malfet/checkout@silent-checkout 2024-11-01T16:42:33.6504797Z with: 2024-11-01T16:42:33.6505159Z ref: d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:42:33.6505656Z fetch-depth: 0 2024-11-01T16:42:33.6505994Z submodules: recursive 2024-11-01T16:42:33.6506367Z quiet-checkout: true 2024-11-01T16:42:33.6506747Z repository: pytorch/pytorch 2024-11-01T16:42:33.6507297Z token: *** 2024-11-01T16:42:33.6507591Z ssh-strict: true 2024-11-01T16:42:33.6508271Z persist-credentials: true 2024-11-01T16:42:33.6508651Z clean: true 2024-11-01T16:42:33.6508985Z sparse-checkout-cone-mode: true 2024-11-01T16:42:33.6509400Z lfs: false 2024-11-01T16:42:33.6509713Z set-safe-directory: true 2024-11-01T16:42:33.6510059Z env: 2024-11-01T16:42:33.6510344Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:42:33.6510709Z ##[endgroup] 2024-11-01T16:42:33.7569264Z Syncing repository: pytorch/pytorch 2024-11-01T16:42:33.7570990Z ##[group]Getting Git version info 2024-11-01T16:42:33.7571824Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-11-01T16:42:33.7572758Z [command]/usr/bin/git version 2024-11-01T16:42:33.7573146Z git version 2.40.1 2024-11-01T16:42:33.7574525Z ##[endgroup] 2024-11-01T16:42:33.7587561Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/c7fcbeb7-bac4-436d-905d-cd83c87d67b7' before making global git config changes 2024-11-01T16:42:33.7589146Z Adding repository directory to the temporary git global config as a safe directory 2024-11-01T16:42:33.7590457Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-11-01T16:42:33.7592209Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-11-01T16:42:33.7593226Z ##[group]Initializing the repository 2024-11-01T16:42:33.7594012Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-11-01T16:42:33.7627005Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-11-01T16:42:33.7628409Z hint: is subject to change. To configure the initial branch name to use in all 2024-11-01T16:42:33.7629349Z hint: of your new repositories, which will suppress this warning, call: 2024-11-01T16:42:33.7629988Z hint: 2024-11-01T16:42:33.7630487Z hint: git config --global init.defaultBranch 2024-11-01T16:42:33.7630987Z hint: 2024-11-01T16:42:33.7631567Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-11-01T16:42:33.7632544Z hint: 'development'. The just-created branch can be renamed via this command: 2024-11-01T16:42:33.7633198Z hint: 2024-11-01T16:42:33.7633540Z hint: git branch -m 2024-11-01T16:42:33.7634407Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-11-01T16:42:33.7636690Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-11-01T16:42:33.7671382Z ##[endgroup] 2024-11-01T16:42:33.7672150Z ##[group]Disabling automatic garbage collection 2024-11-01T16:42:33.7673195Z [command]/usr/bin/git config --local gc.auto 0 2024-11-01T16:42:33.7705286Z ##[endgroup] 2024-11-01T16:42:33.7705868Z ##[group]Setting up auth 2024-11-01T16:42:33.7711551Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-11-01T16:42:33.7743554Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-11-01T16:42:33.8039098Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-11-01T16:42:33.8072043Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-11-01T16:42:33.8369045Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-11-01T16:42:33.8420425Z ##[endgroup] 2024-11-01T16:42:33.8421039Z ##[group]Fetching the repository 2024-11-01T16:42:33.8426680Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-11-01T16:42:36.6032234Z remote: Enumerating objects: 1053148 2024-11-01T16:42:36.6033306Z remote: Enumerating objects: 1053710, done. 2024-11-01T16:42:36.6034560Z remote: Counting objects: 0% (1/562) 2024-11-01T16:42:36.6035803Z remote: Counting objects: 1% (6/562) 2024-11-01T16:42:36.6036531Z remote: Counting objects: 2% (12/562) 2024-11-01T16:42:36.6037134Z remote: Counting objects: 3% (17/562) 2024-11-01T16:42:36.6038136Z remote: Counting objects: 4% (23/562) 2024-11-01T16:42:36.6038706Z remote: Counting objects: 5% (29/562) 2024-11-01T16:42:36.6039439Z remote: Counting objects: 6% (34/562) 2024-11-01T16:42:36.6040315Z remote: Counting objects: 7% (40/562) 2024-11-01T16:42:36.6040865Z remote: Counting objects: 8% (45/562) 2024-11-01T16:42:36.6041406Z remote: Counting objects: 9% (51/562) 2024-11-01T16:42:36.6041950Z remote: Counting objects: 10% (57/562) 2024-11-01T16:42:36.6042493Z remote: Counting objects: 11% (62/562) 2024-11-01T16:42:36.6043035Z remote: Counting objects: 12% (68/562) 2024-11-01T16:42:36.6043576Z remote: Counting objects: 13% (74/562) 2024-11-01T16:42:36.6044109Z remote: Counting objects: 14% (79/562) 2024-11-01T16:42:36.6044648Z remote: Counting objects: 15% (85/562) 2024-11-01T16:42:36.6045192Z remote: Counting objects: 16% (90/562) 2024-11-01T16:42:36.6045732Z remote: Counting objects: 17% (96/562) 2024-11-01T16:42:36.6046282Z remote: Counting objects: 18% (102/562) 2024-11-01T16:42:36.6046846Z remote: Counting objects: 19% (107/562) 2024-11-01T16:42:36.6047390Z remote: Counting objects: 20% (113/562) 2024-11-01T16:42:36.6048101Z remote: Counting objects: 21% (119/562) 2024-11-01T16:42:36.6048664Z remote: Counting objects: 22% (124/562) 2024-11-01T16:42:36.6049215Z remote: Counting objects: 23% (130/562) 2024-11-01T16:42:36.6049766Z remote: Counting objects: 24% (135/562) 2024-11-01T16:42:36.6050296Z remote: Counting objects: 25% (141/562) 2024-11-01T16:42:36.6050850Z remote: Counting objects: 26% (147/562) 2024-11-01T16:42:36.6051408Z remote: Counting objects: 27% (152/562) 2024-11-01T16:42:36.6051962Z remote: Counting objects: 28% (158/562) 2024-11-01T16:42:36.6052511Z remote: Counting objects: 29% (163/562) 2024-11-01T16:42:36.6053060Z remote: Counting objects: 30% (169/562) 2024-11-01T16:42:36.6053593Z remote: Counting objects: 31% (175/562) 2024-11-01T16:42:36.6054137Z remote: Counting objects: 32% (180/562) 2024-11-01T16:42:36.6054687Z remote: Counting objects: 33% (186/562) 2024-11-01T16:42:36.6055236Z remote: Counting objects: 34% (192/562) 2024-11-01T16:42:36.6055874Z remote: Counting objects: 35% (197/562) 2024-11-01T16:42:36.6056585Z remote: Counting objects: 36% (203/562) 2024-11-01T16:42:36.6057152Z remote: Counting objects: 37% (208/562) 2024-11-01T16:42:36.6057716Z remote: Counting objects: 38% (214/562) 2024-11-01T16:42:36.6058287Z remote: Counting objects: 39% (220/562) 2024-11-01T16:42:36.6058859Z remote: Counting objects: 40% (225/562) 2024-11-01T16:42:36.6059420Z remote: Counting objects: 41% (231/562) 2024-11-01T16:42:36.6059966Z remote: Counting objects: 42% (237/562) 2024-11-01T16:42:36.6060528Z remote: Counting objects: 43% (242/562) 2024-11-01T16:42:36.6061087Z remote: Counting objects: 44% (248/562) 2024-11-01T16:42:36.6061652Z remote: Counting objects: 45% (253/562) 2024-11-01T16:42:36.6062213Z remote: Counting objects: 46% (259/562) 2024-11-01T16:42:36.6062777Z remote: Counting objects: 47% (265/562) 2024-11-01T16:42:36.6063321Z remote: Counting objects: 48% (270/562) 2024-11-01T16:42:36.6063886Z remote: Counting objects: 49% (276/562) 2024-11-01T16:42:36.6064450Z remote: Counting objects: 50% (281/562) 2024-11-01T16:42:36.6065015Z remote: Counting objects: 51% (287/562) 2024-11-01T16:42:36.6065581Z remote: Counting objects: 52% (293/562) 2024-11-01T16:42:36.6066128Z remote: Counting objects: 53% (298/562) 2024-11-01T16:42:36.6066778Z remote: Counting objects: 54% (304/562) 2024-11-01T16:42:36.6067345Z remote: Counting objects: 55% (310/562) 2024-11-01T16:42:36.6067908Z remote: Counting objects: 56% (315/562) 2024-11-01T16:42:36.6068574Z remote: Counting objects: 57% (321/562) 2024-11-01T16:42:36.6069120Z remote: Counting objects: 58% (326/562) 2024-11-01T16:42:36.6069653Z remote: Counting objects: 59% (332/562) 2024-11-01T16:42:36.6070207Z remote: Counting objects: 60% (338/562) 2024-11-01T16:42:36.6070762Z remote: Counting objects: 61% (343/562) 2024-11-01T16:42:36.6071311Z remote: Counting objects: 62% (349/562) 2024-11-01T16:42:36.6071875Z remote: Counting objects: 63% (355/562) 2024-11-01T16:42:36.6072427Z remote: Counting objects: 64% (360/562) 2024-11-01T16:42:36.6072976Z remote: Counting objects: 65% (366/562) 2024-11-01T16:42:36.6073511Z remote: Counting objects: 66% (371/562) 2024-11-01T16:42:36.6074060Z remote: Counting objects: 67% (377/562) 2024-11-01T16:42:36.6074607Z remote: Counting objects: 68% (383/562) 2024-11-01T16:42:36.6075157Z remote: Counting objects: 69% (388/562) 2024-11-01T16:42:36.6075704Z remote: Counting objects: 70% (394/562) 2024-11-01T16:42:36.6076250Z remote: Counting objects: 71% (400/562) 2024-11-01T16:42:36.6076784Z remote: Counting objects: 72% (405/562) 2024-11-01T16:42:36.6077405Z remote: Counting objects: 73% (411/562) 2024-11-01T16:42:36.6077959Z remote: Counting objects: 74% (416/562) 2024-11-01T16:42:36.6078526Z remote: Counting objects: 75% (422/562) 2024-11-01T16:42:36.6079075Z remote: Counting objects: 76% (428/562) 2024-11-01T16:42:36.6079620Z remote: Counting objects: 77% (433/562) 2024-11-01T16:42:36.6080150Z remote: Counting objects: 78% (439/562) 2024-11-01T16:42:36.6080699Z remote: Counting objects: 79% (444/562) 2024-11-01T16:42:36.6081255Z remote: Counting objects: 80% (450/562) 2024-11-01T16:42:36.6081800Z remote: Counting objects: 81% (456/562) 2024-11-01T16:42:36.6082346Z remote: Counting objects: 82% (461/562) 2024-11-01T16:42:36.6082892Z remote: Counting objects: 83% (467/562) 2024-11-01T16:42:36.6083424Z remote: Counting objects: 84% (473/562) 2024-11-01T16:42:36.6083972Z remote: Counting objects: 85% (478/562) 2024-11-01T16:42:36.6084514Z remote: Counting objects: 86% (484/562) 2024-11-01T16:42:36.6085063Z remote: Counting objects: 87% (489/562) 2024-11-01T16:42:36.6085607Z remote: Counting objects: 88% (495/562) 2024-11-01T16:42:36.6086139Z remote: Counting objects: 89% (501/562) 2024-11-01T16:42:36.6086684Z remote: Counting objects: 90% (506/562) 2024-11-01T16:42:36.6087228Z remote: Counting objects: 91% (512/562) 2024-11-01T16:42:36.6087776Z remote: Counting objects: 92% (518/562) 2024-11-01T16:42:36.6088324Z remote: Counting objects: 93% (523/562) 2024-11-01T16:42:36.6088869Z remote: Counting objects: 94% (529/562) 2024-11-01T16:42:36.6089400Z remote: Counting objects: 95% (534/562) 2024-11-01T16:42:36.6089948Z remote: Counting objects: 96% (540/562) 2024-11-01T16:42:36.6090495Z remote: Counting objects: 97% (546/562) 2024-11-01T16:42:36.6091038Z remote: Counting objects: 98% (551/562) 2024-11-01T16:42:36.6091682Z remote: Counting objects: 99% (557/562) 2024-11-01T16:42:36.6092206Z remote: Counting objects: 100% (562/562) 2024-11-01T16:42:36.6092777Z remote: Counting objects: 100% (562/562), done. 2024-11-01T16:42:36.6093370Z remote: Compressing objects: 0% (1/298) 2024-11-01T16:42:36.6093929Z remote: Compressing objects: 1% (3/298) 2024-11-01T16:42:36.6158362Z remote: Compressing objects: 2% (6/298) 2024-11-01T16:42:36.6291144Z remote: Compressing objects: 3% (9/298) 2024-11-01T16:42:36.6490151Z remote: Compressing objects: 4% (12/298) 2024-11-01T16:42:36.6560967Z remote: Compressing objects: 5% (15/298) 2024-11-01T16:42:36.6746574Z remote: Compressing objects: 6% (18/298) 2024-11-01T16:42:36.6929228Z remote: Compressing objects: 7% (21/298) 2024-11-01T16:42:36.7050568Z remote: Compressing objects: 8% (24/298) 2024-11-01T16:42:36.7126442Z remote: Compressing objects: 9% (27/298) 2024-11-01T16:42:36.7151938Z remote: Compressing objects: 10% (30/298) 2024-11-01T16:42:36.7154490Z remote: Compressing objects: 11% (33/298) 2024-11-01T16:42:36.7156761Z remote: Compressing objects: 12% (36/298) 2024-11-01T16:42:36.7158285Z remote: Compressing objects: 13% (39/298) 2024-11-01T16:42:36.7159789Z remote: Compressing objects: 14% (42/298) 2024-11-01T16:42:36.7160614Z remote: Compressing objects: 15% (45/298) 2024-11-01T16:42:36.7161202Z remote: Compressing objects: 16% (48/298) 2024-11-01T16:42:36.7161779Z remote: Compressing objects: 17% (51/298) 2024-11-01T16:42:36.7162367Z remote: Compressing objects: 18% (54/298) 2024-11-01T16:42:36.7162947Z remote: Compressing objects: 19% (57/298) 2024-11-01T16:42:36.7163536Z remote: Compressing objects: 20% (60/298) 2024-11-01T16:42:36.7164118Z remote: Compressing objects: 21% (63/298) 2024-11-01T16:42:36.7164698Z remote: Compressing objects: 22% (66/298) 2024-11-01T16:42:36.7165468Z remote: Compressing objects: 23% (69/298) 2024-11-01T16:42:36.7166059Z remote: Compressing objects: 24% (72/298) 2024-11-01T16:42:36.7167550Z remote: Compressing objects: 25% (75/298) 2024-11-01T16:42:36.7170147Z remote: Compressing objects: 26% (78/298) 2024-11-01T16:42:36.7170735Z remote: Compressing objects: 27% (81/298) 2024-11-01T16:42:36.7171315Z remote: Compressing objects: 28% (84/298) 2024-11-01T16:42:36.7172544Z remote: Compressing objects: 29% (87/298) 2024-11-01T16:42:36.7173121Z remote: Compressing objects: 30% (90/298) 2024-11-01T16:42:36.7175169Z remote: Compressing objects: 31% (93/298) 2024-11-01T16:42:36.7175854Z remote: Compressing objects: 32% (96/298) 2024-11-01T16:42:36.7177710Z remote: Compressing objects: 33% (99/298) 2024-11-01T16:42:36.7180489Z remote: Compressing objects: 34% (102/298) 2024-11-01T16:42:36.7185874Z remote: Compressing objects: 35% (105/298) 2024-11-01T16:42:36.7188217Z remote: Compressing objects: 36% (108/298) 2024-11-01T16:42:36.7189730Z remote: Compressing objects: 37% (111/298) 2024-11-01T16:42:36.7197444Z remote: Compressing objects: 38% (114/298) 2024-11-01T16:42:36.7201982Z remote: Compressing objects: 39% (117/298) 2024-11-01T16:42:36.7204544Z remote: Compressing objects: 40% (120/298) 2024-11-01T16:42:36.7205132Z remote: Compressing objects: 41% (123/298) 2024-11-01T16:42:36.7212822Z remote: Compressing objects: 42% (126/298) 2024-11-01T16:42:36.7215186Z remote: Compressing objects: 43% (129/298) 2024-11-01T16:42:36.7217020Z remote: Compressing objects: 44% (132/298) 2024-11-01T16:42:36.7222516Z remote: Compressing objects: 45% (135/298) 2024-11-01T16:42:36.7224885Z remote: Compressing objects: 46% (138/298) 2024-11-01T16:42:36.7227524Z remote: Compressing objects: 47% (141/298) 2024-11-01T16:42:36.7229163Z remote: Compressing objects: 48% (144/298) 2024-11-01T16:42:36.7232258Z remote: Compressing objects: 49% (147/298) 2024-11-01T16:42:36.7235108Z remote: Compressing objects: 50% (149/298) 2024-11-01T16:42:36.7237883Z remote: Compressing objects: 51% (152/298) 2024-11-01T16:42:36.7238468Z remote: Compressing objects: 52% (155/298) 2024-11-01T16:42:36.7240732Z remote: Compressing objects: 53% (158/298) 2024-11-01T16:42:36.7244201Z remote: Compressing objects: 54% (161/298) 2024-11-01T16:42:36.7246705Z remote: Compressing objects: 55% (164/298) 2024-11-01T16:42:36.7249949Z remote: Compressing objects: 56% (167/298) 2024-11-01T16:42:36.7252589Z remote: Compressing objects: 57% (170/298) 2024-11-01T16:42:36.7255049Z remote: Compressing objects: 58% (173/298) 2024-11-01T16:42:36.7257872Z remote: Compressing objects: 59% (176/298) 2024-11-01T16:42:36.7258682Z remote: Compressing objects: 60% (179/298) 2024-11-01T16:42:36.7259300Z remote: Compressing objects: 61% (182/298) 2024-11-01T16:42:36.7259914Z remote: Compressing objects: 62% (185/298) 2024-11-01T16:42:36.7260511Z remote: Compressing objects: 63% (188/298) 2024-11-01T16:42:36.7261113Z remote: Compressing objects: 64% (191/298) 2024-11-01T16:42:36.7261701Z remote: Compressing objects: 65% (194/298) 2024-11-01T16:42:36.7262298Z remote: Compressing objects: 66% (197/298) 2024-11-01T16:42:36.7262899Z remote: Compressing objects: 67% (200/298) 2024-11-01T16:42:36.7263503Z remote: Compressing objects: 68% (203/298) 2024-11-01T16:42:36.7264102Z remote: Compressing objects: 69% (206/298) 2024-11-01T16:42:36.7264685Z remote: Compressing objects: 70% (209/298) 2024-11-01T16:42:36.7265285Z remote: Compressing objects: 71% (212/298) 2024-11-01T16:42:36.7265887Z remote: Compressing objects: 72% (215/298) 2024-11-01T16:42:36.7266489Z remote: Compressing objects: 73% (218/298) 2024-11-01T16:42:36.7267205Z remote: Compressing objects: 74% (221/298) 2024-11-01T16:42:36.7267811Z remote: Compressing objects: 75% (224/298) 2024-11-01T16:42:36.7268502Z remote: Compressing objects: 76% (227/298) 2024-11-01T16:42:36.7269087Z remote: Compressing objects: 77% (230/298) 2024-11-01T16:42:36.7269673Z remote: Compressing objects: 78% (233/298) 2024-11-01T16:42:36.7270259Z remote: Compressing objects: 79% (236/298) 2024-11-01T16:42:36.7270853Z remote: Compressing objects: 80% (239/298) 2024-11-01T16:42:36.7271435Z remote: Compressing objects: 81% (242/298) 2024-11-01T16:42:36.7272021Z remote: Compressing objects: 82% (245/298) 2024-11-01T16:42:36.7272854Z remote: Compressing objects: 83% (248/298) 2024-11-01T16:42:36.7273437Z remote: Compressing objects: 84% (251/298) 2024-11-01T16:42:36.7275426Z remote: Compressing objects: 85% (254/298) 2024-11-01T16:42:36.7278164Z remote: Compressing objects: 86% (257/298) 2024-11-01T16:42:36.7278750Z remote: Compressing objects: 87% (260/298) 2024-11-01T16:42:36.7279334Z remote: Compressing objects: 88% (263/298) 2024-11-01T16:42:36.7279936Z remote: Compressing objects: 89% (266/298) 2024-11-01T16:42:36.7280518Z remote: Compressing objects: 90% (269/298) 2024-11-01T16:42:36.7281107Z remote: Compressing objects: 91% (272/298) 2024-11-01T16:42:36.7281676Z remote: Compressing objects: 92% (275/298) 2024-11-01T16:42:36.7283314Z remote: Compressing objects: 93% (278/298) 2024-11-01T16:42:36.7283903Z remote: Compressing objects: 94% (281/298) 2024-11-01T16:42:36.7284480Z remote: Compressing objects: 95% (284/298) 2024-11-01T16:42:36.7285067Z remote: Compressing objects: 96% (287/298) 2024-11-01T16:42:36.7285830Z remote: Compressing objects: 97% (290/298) 2024-11-01T16:42:36.7286410Z remote: Compressing objects: 98% (293/298) 2024-11-01T16:42:36.7286985Z remote: Compressing objects: 99% (296/298) 2024-11-01T16:42:36.7288538Z remote: Compressing objects: 100% (298/298) 2024-11-01T16:42:36.7289178Z remote: Compressing objects: 100% (298/298), done. 2024-11-01T16:42:58.9978441Z remote: Total 1053710 (delta 346), reused 431 (delta 263), pack-reused 1053148 (from 1) 2024-11-01T16:43:19.6544140Z [command]/usr/bin/git rev-parse --verify --quiet d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea^{object} 2024-11-01T16:43:19.6574295Z d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:43:19.6578408Z ##[endgroup] 2024-11-01T16:43:19.6579004Z ##[group]Determining the checkout info 2024-11-01T16:43:19.6581234Z ##[endgroup] 2024-11-01T16:43:19.6581833Z ##[group]Checking out the ref 2024-11-01T16:43:19.6583590Z [command]/usr/bin/git checkout --quiet --force d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:43:21.1493043Z ##[endgroup] 2024-11-01T16:43:21.1498278Z ##[group]Setting up auth for fetching submodules 2024-11-01T16:43:21.1499590Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-11-01T16:43:21.1551741Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-11-01T16:43:21.1583847Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-11-01T16:43:21.1617932Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-11-01T16:43:21.1646098Z ##[endgroup] 2024-11-01T16:43:21.1646659Z ##[group]Fetching submodules 2024-11-01T16:43:21.1650319Z [command]/usr/bin/git submodule sync --recursive 2024-11-01T16:43:21.1971440Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-11-01T16:43:21.2288531Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-11-01T16:43:21.2290472Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-11-01T16:43:21.2292147Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-11-01T16:43:21.2293762Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-11-01T16:43:21.2295669Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2024-11-01T16:43:21.2299544Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-11-01T16:43:21.2303590Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-11-01T16:43:21.2306880Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-11-01T16:43:21.2310260Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel' 2024-11-01T16:43:21.2313487Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-11-01T16:43:21.2316666Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-11-01T16:43:21.2320244Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-11-01T16:43:21.2323575Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-11-01T16:43:21.2327175Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-11-01T16:43:21.2330856Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-11-01T16:43:21.2334745Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-11-01T16:43:21.2340533Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-11-01T16:43:21.2345044Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:43:21.2349110Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-11-01T16:43:21.2353591Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-11-01T16:43:21.2357740Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-11-01T16:43:21.2362226Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-11-01T16:43:21.2366715Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-11-01T16:43:21.2371391Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-11-01T16:43:21.2376084Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-11-01T16:43:21.2381395Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-11-01T16:43:21.2386384Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-11-01T16:43:21.2391991Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-11-01T16:43:21.2396859Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-11-01T16:43:21.2402663Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-11-01T16:43:21.2408117Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-11-01T16:43:21.2413987Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-11-01T16:43:21.2419790Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-11-01T16:43:21.2427227Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-11-01T16:43:21.2433099Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-11-01T16:43:21.2439181Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-11-01T16:43:21.2472784Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-11-01T16:43:21.5503547Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-11-01T16:43:21.7427469Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-11-01T16:43:21.9313429Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-11-01T16:43:22.1889396Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2024-11-01T16:43:22.4977169Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-11-01T16:43:24.3179827Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-11-01T16:43:33.6729791Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-11-01T16:43:34.1544235Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'... 2024-11-01T16:43:35.7656931Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-11-01T16:43:36.2357834Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-11-01T16:43:36.8467272Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-11-01T16:43:38.0223048Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-11-01T16:43:40.1009197Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-11-01T16:43:45.4420032Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-11-01T16:43:46.9330994Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-11-01T16:43:48.5143324Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-11-01T16:43:49.5666307Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-11-01T16:43:50.0311549Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-11-01T16:43:50.4363743Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-11-01T16:43:51.3969132Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-11-01T16:43:51.7870962Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-11-01T16:43:52.1028406Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-11-01T16:43:53.7817156Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-11-01T16:43:54.5090175Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-11-01T16:43:54.9018672Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-11-01T16:44:00.4617782Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-11-01T16:44:02.2038045Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-11-01T16:44:08.8041887Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-11-01T16:44:09.0549728Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-11-01T16:44:17.8020091Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-11-01T16:44:17.9920259Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-11-01T16:44:18.2069968Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-11-01T16:44:19.1974170Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-11-01T16:44:19.5350258Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-11-01T16:44:20.2505847Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-11-01T16:44:20.6826915Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-11-01T16:44:20.6952361Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-11-01T16:44:20.7048347Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-11-01T16:44:20.7300369Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-11-01T16:44:20.7611247Z Submodule path 'third_party/NVTX': checked out 'e170594ac7cf1dac584da473d4ca9301087090c1' 2024-11-01T16:44:20.8014343Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-11-01T16:44:21.6267644Z Submodule path 'third_party/XNNPACK': checked out '87ee0b46b834f67bad9025d4a82ed5654f3403d3' 2024-11-01T16:44:21.6516397Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-11-01T16:44:21.8660705Z Submodule path 'third_party/composable_kernel': checked out 'cedccd59c94cb0c74e7ec0d0f6c791aed081febc' 2024-11-01T16:44:21.9153222Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-11-01T16:44:22.0154458Z Submodule path 'third_party/cpuinfo': checked out '1e83a2fdd3102f65c6f1fb602c1b320486218a99' 2024-11-01T16:44:22.0493285Z Submodule path 'third_party/cudnn_frontend': checked out '936021bfed8c91dc416af1588b2c4eca631a9e45' 2024-11-01T16:44:22.5745900Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-11-01T16:44:22.8372267Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-11-01T16:44:22.9305765Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-11-01T16:44:22.9324755Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:44:22.9326509Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:44:22.9328251Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:44:22.9330298Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:44:22.9333677Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:44:22.9364472Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-11-01T16:44:23.9028223Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-11-01T16:44:24.5138413Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-11-01T16:44:26.5841182Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-11-01T16:44:27.5088532Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-11-01T16:44:27.9085280Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-11-01T16:44:28.0081707Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-11-01T16:44:28.4254677Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-11-01T16:44:28.4903352Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-11-01T16:44:28.5033648Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-11-01T16:44:28.6242966Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-11-01T16:44:28.6645647Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-11-01T16:44:28.7051450Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-11-01T16:44:28.7319152Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-11-01T16:44:28.7789727Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-11-01T16:44:28.7925002Z Submodule path 'third_party/ideep': checked out '41d636c2bbcea6bff0faf97cdb65a48cdde987af' 2024-11-01T16:44:28.7941324Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-11-01T16:44:28.7967917Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-11-01T16:44:40.6772640Z Submodule path 'third_party/ideep/mkl-dnn': checked out '66f0cb9eb66affd2da3bf5f8d897376f04aae6af' 2024-11-01T16:44:40.6967344Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-11-01T16:44:40.7879662Z Submodule path 'third_party/kineto': checked out 'ed052ea024b9468908d558b15cd3f7584fb0f492' 2024-11-01T16:44:40.7897638Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:44:40.7899759Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:44:40.7902078Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:44:40.7930970Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-11-01T16:44:41.3361947Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-11-01T16:44:42.3993158Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-11-01T16:44:43.3936819Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-11-01T16:44:43.3955761Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:44:43.3958405Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:44:43.3960596Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:44:43.3962809Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:44:43.3965132Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:44:43.3967708Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:44:43.3971146Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:44:43.3974458Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:44:43.4007019Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-11-01T16:44:44.2700125Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-11-01T16:44:44.6692951Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-11-01T16:44:45.6938394Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-11-01T16:44:46.0327570Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-11-01T16:44:46.6148603Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-11-01T16:44:47.5273383Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-11-01T16:44:53.0432569Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-11-01T16:44:53.4674694Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-11-01T16:44:53.4864516Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-11-01T16:44:53.5247219Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-11-01T16:44:53.5381737Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-11-01T16:44:53.5399312Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:44:53.5425669Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-11-01T16:44:53.8794870Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-11-01T16:44:53.8979749Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-11-01T16:44:53.9415153Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-11-01T16:44:54.0473582Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-11-01T16:44:54.0650854Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-11-01T16:44:54.1062841Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-11-01T16:44:54.1680399Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-11-01T16:44:54.2076991Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-11-01T16:44:54.2390731Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-11-01T16:44:54.3491995Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-11-01T16:44:54.7263288Z Submodule path 'third_party/onnx': checked out 'b8baa8446686496da4cc8fda09f2b6fe65c2a02c' 2024-11-01T16:44:54.7299411Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-11-01T16:44:54.7329251Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-11-01T16:44:56.0473782Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-11-01T16:44:56.1201919Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-11-01T16:44:56.1221542Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:44:56.1224074Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:44:56.1226167Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:44:56.1229377Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:44:56.1232435Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:44:56.1235161Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:44:56.1238165Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:44:56.1240973Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:44:56.1272088Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-11-01T16:44:56.6538413Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-11-01T16:44:57.5262261Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-11-01T16:44:57.9631196Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-11-01T16:45:03.6483337Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-11-01T16:45:03.9345993Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-11-01T16:45:04.1553908Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-11-01T16:45:04.5321695Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-11-01T16:45:10.6690416Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-11-01T16:45:10.7116815Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-11-01T16:45:10.7278564Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-11-01T16:45:10.8297874Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-11-01T16:45:10.8440038Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-11-01T16:45:10.8587342Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-11-01T16:45:10.8756916Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-11-01T16:45:10.8774012Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:45:10.8776549Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:45:10.8803501Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-11-01T16:45:12.5881928Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-11-01T16:45:13.7419778Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-11-01T16:45:13.7904474Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-11-01T16:45:14.2562750Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-11-01T16:45:14.2689278Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-11-01T16:45:14.5617687Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-11-01T16:45:14.5641812Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:45:14.5643912Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-11-01T16:45:14.5673512Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-11-01T16:45:14.9950464Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-11-01T16:45:15.9250002Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-11-01T16:45:15.9979840Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-11-01T16:45:16.0076438Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-11-01T16:45:16.0204490Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-11-01T16:45:16.0567015Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2024-11-01T16:45:16.0876848Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-11-01T16:45:16.1312950Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-11-01T16:45:16.1583751Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-11-01T16:45:16.1603060Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:45:16.1604977Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:45:16.1606717Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:45:16.1608512Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:45:16.1637745Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-11-01T16:45:17.1605839Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-11-01T16:45:17.4245325Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-11-01T16:45:18.6482532Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-11-01T16:45:19.7161946Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-11-01T16:45:19.7319239Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-11-01T16:45:19.8031996Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-11-01T16:45:19.8324100Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-11-01T16:45:19.8339342Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:45:19.8368175Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-11-01T16:45:20.1530671Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-11-01T16:45:20.1571548Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-11-01T16:45:20.1890917Z Entering 'android/libs/fbjni' 2024-11-01T16:45:20.1935971Z Entering 'third_party/FP16' 2024-11-01T16:45:20.1978867Z Entering 'third_party/FXdiv' 2024-11-01T16:45:20.2021636Z Entering 'third_party/NNPACK' 2024-11-01T16:45:20.2064199Z Entering 'third_party/NVTX' 2024-11-01T16:45:20.2108021Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T16:45:20.2156317Z Entering 'third_party/XNNPACK' 2024-11-01T16:45:20.2217714Z Entering 'third_party/benchmark' 2024-11-01T16:45:20.2258979Z Entering 'third_party/composable_kernel' 2024-11-01T16:45:20.2308150Z Entering 'third_party/cpp-httplib' 2024-11-01T16:45:20.2352725Z Entering 'third_party/cpuinfo' 2024-11-01T16:45:20.2394971Z Entering 'third_party/cudnn_frontend' 2024-11-01T16:45:20.2439298Z Entering 'third_party/cutlass' 2024-11-01T16:45:20.2490485Z Entering 'third_party/eigen' 2024-11-01T16:45:20.2537789Z Entering 'third_party/fbgemm' 2024-11-01T16:45:20.2580483Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:45:20.2622596Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:45:20.2664156Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:45:20.2720868Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:45:20.2762429Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:45:20.2804186Z Entering 'third_party/flatbuffers' 2024-11-01T16:45:20.2853053Z Entering 'third_party/fmt' 2024-11-01T16:45:20.2896887Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:45:20.2939095Z Entering 'third_party/gloo' 2024-11-01T16:45:20.2981511Z Entering 'third_party/googletest' 2024-11-01T16:45:20.3023956Z Entering 'third_party/ideep' 2024-11-01T16:45:20.3071711Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T16:45:20.3123486Z Entering 'third_party/ittapi' 2024-11-01T16:45:20.3168333Z Entering 'third_party/kineto' 2024-11-01T16:45:20.3211118Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:45:20.3260426Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:45:20.3303796Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:45:20.3345704Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:45:20.3394038Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:45:20.3437046Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:45:20.3481987Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:45:20.3524947Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:45:20.3572071Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:45:20.3618001Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:45:20.3659824Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:45:20.3702663Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:45:20.3745694Z Entering 'third_party/mimalloc' 2024-11-01T16:45:20.3791290Z Entering 'third_party/nccl/nccl' 2024-11-01T16:45:20.3836347Z Entering 'third_party/nlohmann' 2024-11-01T16:45:20.3880321Z Entering 'third_party/onnx' 2024-11-01T16:45:20.3938267Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T16:45:20.3981686Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T16:45:20.4027820Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:45:20.4071627Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:45:20.4114809Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:45:20.4156077Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:45:20.4198861Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:45:20.4241282Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:45:20.4283079Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:45:20.4324829Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:45:20.4370755Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:45:20.4416317Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:45:20.4479522Z Entering 'third_party/pocketfft' 2024-11-01T16:45:20.4521459Z Entering 'third_party/protobuf' 2024-11-01T16:45:20.4566524Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:45:20.4611979Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T16:45:20.4662231Z Entering 'third_party/psimd' 2024-11-01T16:45:20.4705489Z Entering 'third_party/pthreadpool' 2024-11-01T16:45:20.4752736Z Entering 'third_party/pybind11' 2024-11-01T16:45:20.4794964Z Entering 'third_party/python-peachpy' 2024-11-01T16:45:20.4839184Z Entering 'third_party/sleef' 2024-11-01T16:45:20.4881733Z Entering 'third_party/tensorpipe' 2024-11-01T16:45:20.4925236Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:45:20.4973583Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:45:20.5020119Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:45:20.5061816Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:45:20.5102189Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:45:20.5161988Z ##[endgroup] 2024-11-01T16:45:20.5164046Z ##[group]Persisting credentials for submodules 2024-11-01T16:45:20.5165940Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-11-01T16:45:20.5481032Z Entering 'android/libs/fbjni' 2024-11-01T16:45:20.5537888Z Entering 'third_party/FP16' 2024-11-01T16:45:20.5595309Z Entering 'third_party/FXdiv' 2024-11-01T16:45:20.5653057Z Entering 'third_party/NNPACK' 2024-11-01T16:45:20.5711803Z Entering 'third_party/NVTX' 2024-11-01T16:45:20.5768811Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T16:45:20.5824205Z Entering 'third_party/XNNPACK' 2024-11-01T16:45:20.5898061Z Entering 'third_party/benchmark' 2024-11-01T16:45:20.5954541Z Entering 'third_party/composable_kernel' 2024-11-01T16:45:20.6017792Z Entering 'third_party/cpp-httplib' 2024-11-01T16:45:20.6073925Z Entering 'third_party/cpuinfo' 2024-11-01T16:45:20.6134240Z Entering 'third_party/cudnn_frontend' 2024-11-01T16:45:20.6190815Z Entering 'third_party/cutlass' 2024-11-01T16:45:20.6257564Z Entering 'third_party/eigen' 2024-11-01T16:45:20.6318326Z Entering 'third_party/fbgemm' 2024-11-01T16:45:20.6374897Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:45:20.6432319Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:45:20.6488423Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:45:20.6559767Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:45:20.6618206Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:45:20.6675913Z Entering 'third_party/flatbuffers' 2024-11-01T16:45:20.6737427Z Entering 'third_party/fmt' 2024-11-01T16:45:20.6796324Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:45:20.6853884Z Entering 'third_party/gloo' 2024-11-01T16:45:20.6910345Z Entering 'third_party/googletest' 2024-11-01T16:45:20.6967532Z Entering 'third_party/ideep' 2024-11-01T16:45:20.7019556Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T16:45:20.7084075Z Entering 'third_party/ittapi' 2024-11-01T16:45:20.7138864Z Entering 'third_party/kineto' 2024-11-01T16:45:20.7195134Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:45:20.7254640Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:45:20.7316712Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:45:20.7375247Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:45:20.7431334Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:45:20.7490898Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:45:20.7547577Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:45:20.7603102Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:45:20.7661790Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:45:20.7719809Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:45:20.7777690Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:45:20.7835078Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:45:20.7892087Z Entering 'third_party/mimalloc' 2024-11-01T16:45:20.7950972Z Entering 'third_party/nccl/nccl' 2024-11-01T16:45:20.8006627Z Entering 'third_party/nlohmann' 2024-11-01T16:45:20.8063055Z Entering 'third_party/onnx' 2024-11-01T16:45:20.8133727Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T16:45:20.8193496Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T16:45:20.8252806Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:45:20.8306525Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:45:20.8361379Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:45:20.8422447Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:45:20.8479090Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:45:20.8540081Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:45:20.8596056Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:45:20.8651815Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:45:20.8713242Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:45:20.8773917Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:45:20.8855041Z Entering 'third_party/pocketfft' 2024-11-01T16:45:20.8909345Z Entering 'third_party/protobuf' 2024-11-01T16:45:20.8971767Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:45:20.9025308Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T16:45:20.9082934Z Entering 'third_party/psimd' 2024-11-01T16:45:20.9138212Z Entering 'third_party/pthreadpool' 2024-11-01T16:45:20.9195515Z Entering 'third_party/pybind11' 2024-11-01T16:45:20.9254160Z Entering 'third_party/python-peachpy' 2024-11-01T16:45:20.9316373Z Entering 'third_party/sleef' 2024-11-01T16:45:20.9375267Z Entering 'third_party/tensorpipe' 2024-11-01T16:45:20.9433031Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:45:20.9489142Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:45:20.9553725Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:45:20.9612495Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:45:20.9671915Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:45:20.9782966Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-11-01T16:45:21.0058633Z Entering 'android/libs/fbjni' 2024-11-01T16:45:21.0108805Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-11-01T16:45:21.0127022Z Entering 'third_party/FP16' 2024-11-01T16:45:21.0178256Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-11-01T16:45:21.0195676Z Entering 'third_party/FXdiv' 2024-11-01T16:45:21.0248730Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-11-01T16:45:21.0263458Z Entering 'third_party/NNPACK' 2024-11-01T16:45:21.0315860Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-11-01T16:45:21.0333287Z Entering 'third_party/NVTX' 2024-11-01T16:45:21.0388169Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2024-11-01T16:45:21.0404356Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T16:45:21.0457387Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-11-01T16:45:21.0474210Z Entering 'third_party/XNNPACK' 2024-11-01T16:45:21.0525036Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-11-01T16:45:21.0561395Z Entering 'third_party/benchmark' 2024-11-01T16:45:21.0612115Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-11-01T16:45:21.0631555Z Entering 'third_party/composable_kernel' 2024-11-01T16:45:21.0683182Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2024-11-01T16:45:21.0704837Z Entering 'third_party/cpp-httplib' 2024-11-01T16:45:21.0759514Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-11-01T16:45:21.0776822Z Entering 'third_party/cpuinfo' 2024-11-01T16:45:21.0828529Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-11-01T16:45:21.0847757Z Entering 'third_party/cudnn_frontend' 2024-11-01T16:45:21.0898683Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-11-01T16:45:21.0917216Z Entering 'third_party/cutlass' 2024-11-01T16:45:21.0967071Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-11-01T16:45:21.0995478Z Entering 'third_party/eigen' 2024-11-01T16:45:21.1051981Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-11-01T16:45:21.1073331Z Entering 'third_party/fbgemm' 2024-11-01T16:45:21.1125070Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-11-01T16:45:21.1139844Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:45:21.1193065Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-11-01T16:45:21.1207561Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:45:21.1269237Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-11-01T16:45:21.1288855Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:45:21.1341721Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-11-01T16:45:21.1367044Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:45:21.1418760Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:45:21.1436339Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:45:21.1487222Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-11-01T16:45:21.1504009Z Entering 'third_party/flatbuffers' 2024-11-01T16:45:21.1556641Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-11-01T16:45:21.1577901Z Entering 'third_party/fmt' 2024-11-01T16:45:21.1630992Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-11-01T16:45:21.1649303Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:45:21.1699138Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-11-01T16:45:21.1718331Z Entering 'third_party/gloo' 2024-11-01T16:45:21.1772940Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-11-01T16:45:21.1791684Z Entering 'third_party/googletest' 2024-11-01T16:45:21.1849613Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:45:21.1865241Z Entering 'third_party/ideep' 2024-11-01T16:45:21.1920878Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-11-01T16:45:21.1937327Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T16:45:21.1985980Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-11-01T16:45:21.2011350Z Entering 'third_party/ittapi' 2024-11-01T16:45:21.2061846Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-11-01T16:45:21.2079263Z Entering 'third_party/kineto' 2024-11-01T16:45:21.2133180Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-11-01T16:45:21.2151567Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:45:21.2203886Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-11-01T16:45:21.2219078Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:45:21.2274096Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-11-01T16:45:21.2293590Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:45:21.2344345Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-11-01T16:45:21.2362122Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:45:21.2413760Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-11-01T16:45:21.2433389Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:45:21.2486525Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-11-01T16:45:21.2501137Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:45:21.2554451Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-11-01T16:45:21.2573769Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:45:21.2625238Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-11-01T16:45:21.2643079Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:45:21.2695337Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:45:21.2711283Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:45:21.2766185Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-11-01T16:45:21.2782972Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:45:21.2837983Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-11-01T16:45:21.2856270Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:45:21.2907400Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-11-01T16:45:21.2925035Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:45:21.2976962Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-11-01T16:45:21.2996432Z Entering 'third_party/mimalloc' 2024-11-01T16:45:21.3048818Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-11-01T16:45:21.3063625Z Entering 'third_party/nccl/nccl' 2024-11-01T16:45:21.3117686Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-11-01T16:45:21.3134937Z Entering 'third_party/nlohmann' 2024-11-01T16:45:21.3184712Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-11-01T16:45:21.3203081Z Entering 'third_party/onnx' 2024-11-01T16:45:21.3256506Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-11-01T16:45:21.3289821Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T16:45:21.3344557Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-11-01T16:45:21.3365043Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T16:45:21.3418206Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-11-01T16:45:21.3436402Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:45:21.3485877Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-11-01T16:45:21.3501168Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:45:21.3554132Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:45:21.3571342Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:45:21.3621817Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-11-01T16:45:21.3639048Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:45:21.3690756Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-11-01T16:45:21.3707815Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:45:21.3758901Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-11-01T16:45:21.3777621Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:45:21.3829882Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-11-01T16:45:21.3846174Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:45:21.3898753Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-11-01T16:45:21.3915832Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:45:21.3966645Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-11-01T16:45:21.3984113Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:45:21.4037875Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-11-01T16:45:21.4055193Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:45:21.4107213Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-11-01T16:45:21.4142211Z Entering 'third_party/pocketfft' 2024-11-01T16:45:21.4194208Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-11-01T16:45:21.4209645Z Entering 'third_party/protobuf' 2024-11-01T16:45:21.4263149Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-11-01T16:45:21.4283745Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:45:21.4335709Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-11-01T16:45:21.4352819Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T16:45:21.4404831Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:45:21.4422552Z Entering 'third_party/psimd' 2024-11-01T16:45:21.4474870Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-11-01T16:45:21.4492215Z Entering 'third_party/pthreadpool' 2024-11-01T16:45:21.4543279Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-11-01T16:45:21.4561486Z Entering 'third_party/pybind11' 2024-11-01T16:45:21.4611242Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-11-01T16:45:21.4629017Z Entering 'third_party/python-peachpy' 2024-11-01T16:45:21.4683372Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-11-01T16:45:21.4702866Z Entering 'third_party/sleef' 2024-11-01T16:45:21.4754998Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-11-01T16:45:21.4770974Z Entering 'third_party/tensorpipe' 2024-11-01T16:45:21.4823889Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-11-01T16:45:21.4841603Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:45:21.4893112Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:45:21.4908372Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:45:21.4960601Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-11-01T16:45:21.4977827Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:45:21.5032703Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-11-01T16:45:21.5049357Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:45:21.5100270Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-11-01T16:45:21.5118121Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:45:21.5170464Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-11-01T16:45:21.6095366Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-11-01T16:45:21.6417988Z Entering 'android/libs/fbjni' 2024-11-01T16:45:21.6459957Z Entering 'third_party/FP16' 2024-11-01T16:45:21.6502642Z Entering 'third_party/FXdiv' 2024-11-01T16:45:21.6545135Z Entering 'third_party/NNPACK' 2024-11-01T16:45:21.6587616Z Entering 'third_party/NVTX' 2024-11-01T16:45:21.6635128Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T16:45:21.6678157Z Entering 'third_party/XNNPACK' 2024-11-01T16:45:21.6737315Z Entering 'third_party/benchmark' 2024-11-01T16:45:21.6780339Z Entering 'third_party/composable_kernel' 2024-11-01T16:45:21.6832017Z Entering 'third_party/cpp-httplib' 2024-11-01T16:45:21.6875997Z Entering 'third_party/cpuinfo' 2024-11-01T16:45:21.6920203Z Entering 'third_party/cudnn_frontend' 2024-11-01T16:45:21.6962667Z Entering 'third_party/cutlass' 2024-11-01T16:45:21.7010711Z Entering 'third_party/eigen' 2024-11-01T16:45:21.7057669Z Entering 'third_party/fbgemm' 2024-11-01T16:45:21.7100269Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:45:21.7141225Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:45:21.7183644Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:45:21.7233365Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:45:21.7275278Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:45:21.7320686Z Entering 'third_party/flatbuffers' 2024-11-01T16:45:21.7366129Z Entering 'third_party/fmt' 2024-11-01T16:45:21.7411075Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:45:21.7454365Z Entering 'third_party/gloo' 2024-11-01T16:45:21.7497904Z Entering 'third_party/googletest' 2024-11-01T16:45:21.7538863Z Entering 'third_party/ideep' 2024-11-01T16:45:21.7581691Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T16:45:21.7634673Z Entering 'third_party/ittapi' 2024-11-01T16:45:21.7677284Z Entering 'third_party/kineto' 2024-11-01T16:45:21.7720080Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:45:21.7762248Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:45:21.7805558Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:45:21.7851131Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:45:21.7894917Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:45:21.7937898Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:45:21.7985406Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:45:21.8028513Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:45:21.8073391Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:45:21.8118297Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:45:21.8164477Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:45:21.8206137Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:45:21.8253185Z Entering 'third_party/mimalloc' 2024-11-01T16:45:21.8297930Z Entering 'third_party/nccl/nccl' 2024-11-01T16:45:21.8340802Z Entering 'third_party/nlohmann' 2024-11-01T16:45:21.8384150Z Entering 'third_party/onnx' 2024-11-01T16:45:21.8445371Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T16:45:21.8491389Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T16:45:21.8542280Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:45:21.8583162Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:45:21.8627317Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:45:21.8670662Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:45:21.8719037Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:45:21.8760566Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:45:21.8802804Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:45:21.8844434Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:45:21.8893829Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:45:21.8940492Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:45:21.9012727Z Entering 'third_party/pocketfft' 2024-11-01T16:45:21.9056125Z Entering 'third_party/protobuf' 2024-11-01T16:45:21.9100348Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:45:21.9141757Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T16:45:21.9184693Z Entering 'third_party/psimd' 2024-11-01T16:45:21.9230792Z Entering 'third_party/pthreadpool' 2024-11-01T16:45:21.9274717Z Entering 'third_party/pybind11' 2024-11-01T16:45:21.9317788Z Entering 'third_party/python-peachpy' 2024-11-01T16:45:21.9359438Z Entering 'third_party/sleef' 2024-11-01T16:45:21.9402605Z Entering 'third_party/tensorpipe' 2024-11-01T16:45:21.9447717Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:45:21.9491078Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:45:21.9534528Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:45:21.9577025Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:45:21.9621281Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:45:21.9678772Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-11-01T16:45:21.9997484Z Entering 'android/libs/fbjni' 2024-11-01T16:45:22.0040898Z Entering 'third_party/FP16' 2024-11-01T16:45:22.0083103Z Entering 'third_party/FXdiv' 2024-11-01T16:45:22.0126981Z Entering 'third_party/NNPACK' 2024-11-01T16:45:22.0170808Z Entering 'third_party/NVTX' 2024-11-01T16:45:22.0217593Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T16:45:22.0258968Z Entering 'third_party/XNNPACK' 2024-11-01T16:45:22.0324884Z Entering 'third_party/benchmark' 2024-11-01T16:45:22.0371257Z Entering 'third_party/composable_kernel' 2024-11-01T16:45:22.0419988Z Entering 'third_party/cpp-httplib' 2024-11-01T16:45:22.0462881Z Entering 'third_party/cpuinfo' 2024-11-01T16:45:22.0507879Z Entering 'third_party/cudnn_frontend' 2024-11-01T16:45:22.0553713Z Entering 'third_party/cutlass' 2024-11-01T16:45:22.0603592Z Entering 'third_party/eigen' 2024-11-01T16:45:22.0651434Z Entering 'third_party/fbgemm' 2024-11-01T16:45:22.0694663Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:45:22.0737525Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:45:22.0778530Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:45:22.0827354Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:45:22.0871808Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:45:22.0916914Z Entering 'third_party/flatbuffers' 2024-11-01T16:45:22.0962435Z Entering 'third_party/fmt' 2024-11-01T16:45:22.1003981Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:45:22.1049682Z Entering 'third_party/gloo' 2024-11-01T16:45:22.1093121Z Entering 'third_party/googletest' 2024-11-01T16:45:22.1138001Z Entering 'third_party/ideep' 2024-11-01T16:45:22.1177840Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T16:45:22.1226883Z Entering 'third_party/ittapi' 2024-11-01T16:45:22.1272536Z Entering 'third_party/kineto' 2024-11-01T16:45:22.1315407Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:45:22.1359646Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:45:22.1403230Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:45:22.1452022Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:45:22.1495944Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:45:22.1537939Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:45:22.1581161Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:45:22.1622832Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:45:22.1673813Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:45:22.1718752Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:45:22.1763595Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:45:22.1803222Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:45:22.1848646Z Entering 'third_party/mimalloc' 2024-11-01T16:45:22.1892362Z Entering 'third_party/nccl/nccl' 2024-11-01T16:45:22.1937079Z Entering 'third_party/nlohmann' 2024-11-01T16:45:22.1979387Z Entering 'third_party/onnx' 2024-11-01T16:45:22.2039190Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T16:45:22.2083675Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T16:45:22.2128326Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:45:22.2170618Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:45:22.2212603Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:45:22.2255509Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:45:22.2297775Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:45:22.2339400Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:45:22.2391962Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:45:22.2434837Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:45:22.2477841Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:45:22.2523716Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:45:22.2585262Z Entering 'third_party/pocketfft' 2024-11-01T16:45:22.2633446Z Entering 'third_party/protobuf' 2024-11-01T16:45:22.2678264Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:45:22.2720420Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T16:45:22.2764070Z Entering 'third_party/psimd' 2024-11-01T16:45:22.2806830Z Entering 'third_party/pthreadpool' 2024-11-01T16:45:22.2852949Z Entering 'third_party/pybind11' 2024-11-01T16:45:22.2894552Z Entering 'third_party/python-peachpy' 2024-11-01T16:45:22.2937860Z Entering 'third_party/sleef' 2024-11-01T16:45:22.2979016Z Entering 'third_party/tensorpipe' 2024-11-01T16:45:22.3022532Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:45:22.3063130Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:45:22.3109324Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:45:22.3155548Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:45:22.3195169Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:45:22.3254318Z ##[endgroup] 2024-11-01T16:45:22.3292804Z [command]/usr/bin/git log -1 --format='%H' 2024-11-01T16:45:22.3322792Z 'd1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea' 2024-11-01T16:45:22.3528088Z Prepare all required actions 2024-11-01T16:45:22.3528560Z Getting action download info 2024-11-01T16:45:22.5145725Z ##[group]Run ./.github/actions/setup-linux 2024-11-01T16:45:22.5146197Z env: 2024-11-01T16:45:22.5146509Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:22.5146900Z ##[endgroup] 2024-11-01T16:45:22.5216335Z ##[group]Run set -euo pipefail 2024-11-01T16:45:22.5216828Z set -euo pipefail 2024-11-01T16:45:22.5217279Z function get_ec2_metadata() { 2024-11-01T16:45:22.5217910Z  # Pulled from instance metadata endpoint for EC2 2024-11-01T16:45:22.5218979Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-11-01T16:45:22.5219908Z  category=$1 2024-11-01T16:45:22.5220513Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-11-01T16:45:22.5221257Z  runner_name_str=i-06c394753ea076091 2024-11-01T16:45:22.5221824Z  if [[ -f /.inarc ]]; then 2024-11-01T16:45:22.5222395Z  echo "ARC Runner, no info on ec2 metadata" 2024-11-01T16:45:22.5223042Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-11-01T16:45:22.5223856Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-11-01T16:45:22.5224581Z  else 2024-11-01T16:45:22.5226045Z  curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-11-01T16:45:22.5227590Z  fi 2024-11-01T16:45:22.5228065Z } 2024-11-01T16:45:22.5228464Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-11-01T16:45:22.5229114Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-11-01T16:45:22.5229863Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-11-01T16:45:22.5230506Z echo "system info $(uname -a)" 2024-11-01T16:45:22.5237785Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:22.5238337Z env: 2024-11-01T16:45:22.5238631Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:22.5238992Z ##[endgroup] 2024-11-01T16:45:22.5383122Z ami-id: ami-0fff1b9a61dec8a5f 2024-11-01T16:45:22.5494670Z instance-id: i-06c394753ea076091 2024-11-01T16:45:22.5603948Z instance-type: g4dn.12xlarge 2024-11-01T16:45:22.5614926Z system info Linux ip-10-0-7-207.ec2.internal 6.1.109-118.189.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 10 08:59:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-11-01T16:45:22.5639987Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:45:22.5641365Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:45:22.5647780Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:22.5648307Z env: 2024-11-01T16:45:22.5648575Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:22.5648915Z ##[endgroup] 2024-11-01T16:45:22.5709274Z ##[group]Run if systemctl is-active --quiet docker; then 2024-11-01T16:45:22.5709932Z if systemctl is-active --quiet docker; then 2024-11-01T16:45:22.5710528Z  echo "Docker daemon is running..."; 2024-11-01T16:45:22.5711025Z else 2024-11-01T16:45:22.5711564Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-11-01T16:45:22.5712213Z fi 2024-11-01T16:45:22.5717760Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:22.5718386Z env: 2024-11-01T16:45:22.5718652Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:22.5718996Z ##[endgroup] 2024-11-01T16:45:22.5791774Z Docker daemon is running... 2024-11-01T16:45:22.5859751Z ##[group]Run nick-fields/retry@v3.0.0 2024-11-01T16:45:22.5860191Z with: 2024-11-01T16:45:22.5860459Z shell: bash 2024-11-01T16:45:22.5860948Z timeout_minutes: 5 2024-11-01T16:45:22.5861301Z max_attempts: 3 2024-11-01T16:45:22.5861640Z retry_wait_seconds: 30 2024-11-01T16:45:22.5865080Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-11-01T16:45:22.5868559Z polling_interval_seconds: 1 2024-11-01T16:45:22.5868946Z warning_on_retry: true 2024-11-01T16:45:22.5869314Z continue_on_error: false 2024-11-01T16:45:22.5869659Z env: 2024-11-01T16:45:22.5869922Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:22.5870288Z AWS_RETRY_MODE: standard 2024-11-01T16:45:22.5870647Z AWS_MAX_ATTEMPTS: 5 2024-11-01T16:45:22.5870993Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:45:22.5871379Z ##[endgroup] 2024-11-01T16:45:23.8791749Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-11-01T16:45:23.8792745Z Configure a credential helper to remove this warning. See 2024-11-01T16:45:23.8793767Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-11-01T16:45:23.8794374Z 2024-11-01T16:45:23.8794507Z Login Succeeded 2024-11-01T16:45:24.7400844Z Command completed after 1 attempt(s). 2024-11-01T16:45:24.7469931Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-11-01T16:45:24.7470721Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-11-01T16:45:24.7471461Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-11-01T16:45:24.7479791Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:24.7480304Z env: 2024-11-01T16:45:24.7480578Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:24.7480920Z ##[endgroup] 2024-11-01T16:45:24.7568545Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-11-01T16:45:24.7569346Z # ignore expansion of "docker ps -q" since it could be empty 2024-11-01T16:45:24.7569963Z # shellcheck disable=SC2046 2024-11-01T16:45:24.7570435Z docker stop $(docker ps -q) || true 2024-11-01T16:45:24.7570929Z # Prune all of the docker images 2024-11-01T16:45:24.7571397Z docker system prune -af 2024-11-01T16:45:24.7577811Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:24.7578384Z env: 2024-11-01T16:45:24.7578696Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:24.7579092Z ##[endgroup] 2024-11-01T16:45:24.7855577Z "docker stop" requires at least 1 argument. 2024-11-01T16:45:24.7856959Z See 'docker stop --help'. 2024-11-01T16:45:24.7857251Z 2024-11-01T16:45:24.7857529Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-11-01T16:45:24.7858405Z 2024-11-01T16:45:24.7858705Z Stop one or more running containers 2024-11-01T16:45:24.8003163Z Total reclaimed space: 0B 2024-11-01T16:45:24.8037038Z ##[group]Run set +e 2024-11-01T16:45:24.8037407Z set +e 2024-11-01T16:45:24.8037700Z set -x 2024-11-01T16:45:24.8037993Z  2024-11-01T16:45:24.8038322Z PT_DOMAIN=download.pytorch.org 2024-11-01T16:45:24.8039180Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-11-01T16:45:24.8040388Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-11-01T16:45:24.8041347Z # one is returned at random 2024-11-01T16:45:24.8041928Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-11-01T16:45:24.8042459Z  2024-11-01T16:45:24.8042900Z if [ -z "${RESOLVED_IP}" ]; then 2024-11-01T16:45:24.8043570Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-11-01T16:45:24.8044402Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-11-01T16:45:24.8045007Z  2024-11-01T16:45:24.8045319Z  if [ -z "${RESOLVED_IP}" ]; then 2024-11-01T16:45:24.8045906Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-11-01T16:45:24.8046446Z  exit 1 2024-11-01T16:45:24.8046765Z  fi 2024-11-01T16:45:24.8047055Z fi 2024-11-01T16:45:24.8047330Z  2024-11-01T16:45:24.8047682Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-11-01T16:45:24.8048234Z  # Clean up any old records first 2024-11-01T16:45:24.8048770Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-11-01T16:45:24.8049239Z fi 2024-11-01T16:45:24.8049512Z  2024-11-01T16:45:24.8049952Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-11-01T16:45:24.8050542Z cat /etc/hosts 2024-11-01T16:45:24.8056789Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:24.8057343Z env: 2024-11-01T16:45:24.8057638Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:24.8058016Z ##[endgroup] 2024-11-01T16:45:24.8084162Z + PT_DOMAIN=download.pytorch.org 2024-11-01T16:45:24.8089633Z ++ dig -4 +short download.pytorch.org 2024-11-01T16:45:24.8090129Z ++ tail -n1 2024-11-01T16:45:24.8380688Z + RESOLVED_IP=18.160.10.36 2024-11-01T16:45:24.8381258Z + '[' -z 18.160.10.36 ']' 2024-11-01T16:45:24.8381770Z + grep -r download.pytorch.org /etc/hosts 2024-11-01T16:45:24.8394760Z 18.160.200.112 download.pytorch.org 2024-11-01T16:45:24.8396391Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-11-01T16:45:24.9676999Z + echo '18.160.10.36 download.pytorch.org' 2024-11-01T16:45:24.9677610Z + sudo tee -a /etc/hosts 2024-11-01T16:45:25.0092476Z 18.160.10.36 download.pytorch.org 2024-11-01T16:45:25.0105623Z + cat /etc/hosts 2024-11-01T16:45:25.0114716Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-11-01T16:45:25.0126759Z ::1 localhost6 localhost6.localdomain6 2024-11-01T16:45:25.0127391Z 18.160.10.36 download.pytorch.org 2024-11-01T16:45:25.0337688Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-11-01T16:45:25.0338403Z with: 2024-11-01T16:45:25.0339572Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:25.0340912Z docker-build-dir: .ci/docker 2024-11-01T16:45:25.0341347Z working-directory: . 2024-11-01T16:45:25.0341893Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:25.0342518Z force-push: false 2024-11-01T16:45:25.0342853Z env: 2024-11-01T16:45:25.0343158Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:25.0343546Z ##[endgroup] 2024-11-01T16:45:25.0367075Z ##[group]Run set -ex 2024-11-01T16:45:25.0367448Z set -ex 2024-11-01T16:45:25.0367751Z  2024-11-01T16:45:25.0368330Z # If the docker build directory or the build script doesn't exist, the action will 2024-11-01T16:45:25.0369399Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-11-01T16:45:25.0370253Z # job could then download the pre-built image as usual 2024-11-01T16:45:25.0371030Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-11-01T16:45:25.0371749Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:25.0372413Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:25.0373287Z  2024-11-01T16:45:25.0373848Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-11-01T16:45:25.0374542Z  exit 0 2024-11-01T16:45:25.0374851Z else 2024-11-01T16:45:25.0375243Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:25.0375826Z fi 2024-11-01T16:45:25.0376131Z  2024-11-01T16:45:25.0376836Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-11-01T16:45:25.0377865Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-11-01T16:45:25.0378764Z  # use it as it is, but first let's extract the tag 2024-11-01T16:45:25.0379596Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-11-01T16:45:25.0380454Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:25.0381286Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:25.0381941Z else 2024-11-01T16:45:25.0382436Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-11-01T16:45:25.0383215Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:25.0384265Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:25.0385166Z fi 2024-11-01T16:45:25.0391497Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:25.0392023Z env: 2024-11-01T16:45:25.0392300Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:25.0392660Z REPO_NAME: pytorch 2024-11-01T16:45:25.0393714Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:25.0394879Z DOCKER_BUILD_DIR: .ci/docker 2024-11-01T16:45:25.0395393Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:25.0395939Z ##[endgroup] 2024-11-01T16:45:25.0422675Z + [[ ! -d .ci/docker ]] 2024-11-01T16:45:25.0423177Z + [[ ! -f .ci/docker/build.sh ]] 2024-11-01T16:45:25.0423607Z + echo skip=false 2024-11-01T16:45:25.0425602Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-11-01T16:45:25.0430132Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:25.0431379Z ++ awk -F '[:,]' '{print $2}' 2024-11-01T16:45:25.0452498Z + DOCKER_TAG=bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:25.0453450Z + echo docker-tag=bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:25.0455037Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:25.0485910Z ##[group]Run set +e 2024-11-01T16:45:25.0486276Z set +e 2024-11-01T16:45:25.0486576Z set -x 2024-11-01T16:45:25.0486858Z  2024-11-01T16:45:25.0487120Z login() { 2024-11-01T16:45:25.0487792Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-11-01T16:45:25.0488543Z } 2024-11-01T16:45:25.0488803Z  2024-11-01T16:45:25.0489071Z retry () { 2024-11-01T16:45:25.0489452Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-11-01T16:45:25.0489902Z } 2024-11-01T16:45:25.0490150Z  2024-11-01T16:45:25.0490453Z retry login "${DOCKER_REGISTRY}" 2024-11-01T16:45:25.0490871Z  2024-11-01T16:45:25.0491342Z # Check if image already exists, if it does then skip building it 2024-11-01T16:45:25.0492200Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-11-01T16:45:25.0492700Z  exit 0 2024-11-01T16:45:25.0492993Z fi 2024-11-01T16:45:25.0493257Z  2024-11-01T16:45:25.0493764Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-11-01T16:45:25.0494622Z # be empty. The default action would be to continue rebuild the image 2024-11-01T16:45:25.0495380Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-11-01T16:45:25.0496331Z  # if we're on the base branch then use the parent commit 2024-11-01T16:45:25.0497071Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-11-01T16:45:25.0497558Z else 2024-11-01T16:45:25.0498086Z  # otherwise we're on a PR, so use the most recent base commit 2024-11-01T16:45:25.0498892Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-11-01T16:45:25.0499468Z fi 2024-11-01T16:45:25.0499751Z  2024-11-01T16:45:25.0500095Z if [[ -z "${MERGE_BASE}" ]]; then 2024-11-01T16:45:25.0500663Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:25.0501443Z  2024-11-01T16:45:25.0502199Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-11-01T16:45:25.0503098Z  exit 0 2024-11-01T16:45:25.0503410Z fi 2024-11-01T16:45:25.0503707Z  2024-11-01T16:45:25.0504193Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-11-01T16:45:25.0505344Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-11-01T16:45:25.0506286Z  exit 1 2024-11-01T16:45:25.0506611Z fi 2024-11-01T16:45:25.0506890Z  2024-11-01T16:45:25.0507442Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-11-01T16:45:25.0508574Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-11-01T16:45:25.0509610Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-11-01T16:45:25.0510762Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-11-01T16:45:25.0512190Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-11-01T16:45:25.0513063Z fi 2024-11-01T16:45:25.0513356Z  2024-11-01T16:45:25.0513873Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:25.0519960Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:25.0520608Z env: 2024-11-01T16:45:25.0520891Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:25.0521275Z DOCKER_BUILD_DIR: .ci/docker 2024-11-01T16:45:25.0521772Z BASE_REVISION: 2055d3c8dc8b93c52eda6b9c07d716448d9e28c9 2024-11-01T16:45:25.0523061Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:25.0524338Z DOCKER_TAG: bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:25.0524981Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:25.0525540Z ##[endgroup] 2024-11-01T16:45:25.0549240Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:25.0549977Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:25.0552528Z + aws ecr get-login-password --region us-east-1 2024-11-01T16:45:25.0569359Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:25.6009071Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-11-01T16:45:25.6010283Z Configure a credential helper to remove this warning. See 2024-11-01T16:45:25.6011279Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-11-01T16:45:25.6012152Z 2024-11-01T16:45:25.6012296Z Login Succeeded 2024-11-01T16:45:25.6024665Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:25.8020038Z { 2024-11-01T16:45:25.8020374Z "schemaVersion": 2, 2024-11-01T16:45:25.8021017Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-11-01T16:45:25.8021690Z "config": { 2024-11-01T16:45:25.8022233Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-11-01T16:45:25.8022885Z "size": 49259, 2024-11-01T16:45:25.8023522Z "digest": "sha256:f4475432de42c25077a2ffe0f2f56efb42124dedca9504f01daf084f9103394b" 2024-11-01T16:45:25.8024254Z }, 2024-11-01T16:45:25.8024534Z "layers": [ 2024-11-01T16:45:25.8024814Z { 2024-11-01T16:45:25.8025312Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8026079Z "size": 28580681, 2024-11-01T16:45:25.8026805Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-11-01T16:45:25.8027512Z }, 2024-11-01T16:45:25.8027785Z { 2024-11-01T16:45:25.8028255Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8028877Z "size": 7943353, 2024-11-01T16:45:25.8029508Z "digest": "sha256:09d415c238d76b32a7ea4a6e6add9542db9a5641f7f183af70aae185d0709e58" 2024-11-01T16:45:25.8030224Z }, 2024-11-01T16:45:25.8030480Z { 2024-11-01T16:45:25.8030959Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8031591Z "size": 56226389, 2024-11-01T16:45:25.8032221Z "digest": "sha256:9fe6e2e61518cba6844870c03b285737daec35e62baf25ae7744629ed3a7b470" 2024-11-01T16:45:25.8032919Z }, 2024-11-01T16:45:25.8033178Z { 2024-11-01T16:45:25.8033641Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8034266Z "size": 187, 2024-11-01T16:45:25.8034866Z "digest": "sha256:41f16248e682693ff20b3032c1d5e5541cc87c5af898ae2ff9b24d2940e59100" 2024-11-01T16:45:25.8035560Z }, 2024-11-01T16:45:25.8035815Z { 2024-11-01T16:45:25.8036298Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8036906Z "size": 6885, 2024-11-01T16:45:25.8037523Z "digest": "sha256:95d7b781703928cf3c4eece39d800cccb76728c375fedf51ecd83833fb25e458" 2024-11-01T16:45:25.8038225Z }, 2024-11-01T16:45:25.8038478Z { 2024-11-01T16:45:25.8038954Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8039843Z "size": 1383662475, 2024-11-01T16:45:25.8040455Z "digest": "sha256:8f6c9048534734f4c873935293b7296225846ceb31c1a158400a67ea170dde7f" 2024-11-01T16:45:25.8041131Z }, 2024-11-01T16:45:25.8041390Z { 2024-11-01T16:45:25.8041869Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8042697Z + exit 0 2024-11-01T16:45:25.8042964Z "size": 62282, 2024-11-01T16:45:25.8043566Z "digest": "sha256:ab17245097e491b9368790714f9d90ed447bf0973bd677cfe6f2456d62b72a13" 2024-11-01T16:45:25.8044258Z }, 2024-11-01T16:45:25.8044513Z { 2024-11-01T16:45:25.8044989Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8045602Z "size": 1684, 2024-11-01T16:45:25.8046228Z "digest": "sha256:dfecd7e9912b76ed460b8edd5a85f1943666e38a973ab5458177cf2c7c3110e3" 2024-11-01T16:45:25.8046937Z }, 2024-11-01T16:45:25.8047193Z { 2024-11-01T16:45:25.8047671Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8048281Z "size": 1522, 2024-11-01T16:45:25.8048888Z "digest": "sha256:464a8f74544589bf7b57f9a4cadcb6681e5ed00758f6c35025e691df4e88e890" 2024-11-01T16:45:25.8049587Z }, 2024-11-01T16:45:25.8049841Z { 2024-11-01T16:45:25.8050316Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8050938Z "size": 2445884909, 2024-11-01T16:45:25.8051544Z "digest": "sha256:d67c111fa58886d8904335b5122039816c8e51af5844cb6fe4e14154d9a7c016" 2024-11-01T16:45:25.8052345Z }, 2024-11-01T16:45:25.8052708Z { 2024-11-01T16:45:25.8053175Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8053784Z "size": 85882, 2024-11-01T16:45:25.8054362Z "digest": "sha256:0723abde19c09431db80b5d0f0e003541f6232c2da5c4f35c635c861875937ae" 2024-11-01T16:45:25.8055038Z }, 2024-11-01T16:45:25.8055284Z { 2024-11-01T16:45:25.8055843Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8056632Z "size": 1822, 2024-11-01T16:45:25.8057277Z "digest": "sha256:353d86cedcdc414207029061854fbc806c438ad1f5c00ac8aebf6daa9c2e9e9f" 2024-11-01T16:45:25.8058009Z }, 2024-11-01T16:45:25.8058273Z { 2024-11-01T16:45:25.8058767Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8059414Z "size": 655141106, 2024-11-01T16:45:25.8060046Z "digest": "sha256:0299f4e4d5284c3c947696ef5f6ca6f7b5fab7881f4c4a6096399ca3e64f2c44" 2024-11-01T16:45:25.8060769Z }, 2024-11-01T16:45:25.8061031Z { 2024-11-01T16:45:25.8061524Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8062161Z "size": 703, 2024-11-01T16:45:25.8062781Z "digest": "sha256:86c353c3bc581a31d9abdc8486224b29531a0b199d09e0ea8c5dadce351325b9" 2024-11-01T16:45:25.8063487Z }, 2024-11-01T16:45:25.8063752Z { 2024-11-01T16:45:25.8064236Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8064879Z "size": 1259, 2024-11-01T16:45:25.8065515Z "digest": "sha256:4f10af237965d5a943f19bb5409eef94fae79f531a7e7137ea8a8815aff476b1" 2024-11-01T16:45:25.8066221Z }, 2024-11-01T16:45:25.8066484Z { 2024-11-01T16:45:25.8066975Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8067619Z "size": 484, 2024-11-01T16:45:25.8068356Z "digest": "sha256:b7b00fcb07ca8c1ce9232c36e9c4bcd528d3167479c20a9e0b05a4c318417a21" 2024-11-01T16:45:25.8069124Z }, 2024-11-01T16:45:25.8069364Z { 2024-11-01T16:45:25.8069800Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8070376Z "size": 91712745, 2024-11-01T16:45:25.8070976Z "digest": "sha256:db97392a2fce6d95ddfdcafbdee9fc4102fa976535ce9687d8f32dde3e188f61" 2024-11-01T16:45:25.8071646Z }, 2024-11-01T16:45:25.8071868Z { 2024-11-01T16:45:25.8072304Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8072878Z "size": 3678, 2024-11-01T16:45:25.8073775Z "digest": "sha256:27e2d54f402a8b72082e53cf941bbc159517167ef364e2f9bee99de378effd71" 2024-11-01T16:45:25.8074457Z }, 2024-11-01T16:45:25.8074693Z { 2024-11-01T16:45:25.8075160Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8075768Z "size": 1801, 2024-11-01T16:45:25.8076346Z "digest": "sha256:8269c56845c337b349db7a6d46a81a462af0d1183f2761ab3489909f0663638c" 2024-11-01T16:45:25.8077013Z }, 2024-11-01T16:45:25.8077248Z { 2024-11-01T16:45:25.8077714Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8078324Z "size": 702, 2024-11-01T16:45:25.8078910Z "digest": "sha256:6c7e3cfd6320c1a2e3323c5b6ed269549b5102449ba19199811783455948b5cd" 2024-11-01T16:45:25.8079588Z }, 2024-11-01T16:45:25.8079823Z { 2024-11-01T16:45:25.8080283Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8080892Z "size": 2921113290, 2024-11-01T16:45:25.8081499Z "digest": "sha256:82c15385d36d076d32b291ce1d513b5a7a94f02b56c06254edfca098f5e7ecba" 2024-11-01T16:45:25.8082179Z }, 2024-11-01T16:45:25.8082414Z { 2024-11-01T16:45:25.8082877Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8083474Z "size": 380, 2024-11-01T16:45:25.8084048Z "digest": "sha256:5ffae31f204af230644f54b5d1a9178b0313f3591541a866397342ba4177e782" 2024-11-01T16:45:25.8084722Z }, 2024-11-01T16:45:25.8084969Z { 2024-11-01T16:45:25.8085504Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8086154Z "size": 12890, 2024-11-01T16:45:25.8086702Z "digest": "sha256:879b9cee6016a801826c1899830336c41370e72ae7b77305190ac6d2016bc8df" 2024-11-01T16:45:25.8087326Z }, 2024-11-01T16:45:25.8087559Z { 2024-11-01T16:45:25.8087983Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8088552Z "size": 862, 2024-11-01T16:45:25.8089104Z "digest": "sha256:ead67a6283927375ef4a93495b43516a6a077bce28c304063c69a1c09c40cf72" 2024-11-01T16:45:25.8089745Z }, 2024-11-01T16:45:25.8089982Z { 2024-11-01T16:45:25.8090419Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8090990Z "size": 106, 2024-11-01T16:45:25.8091562Z "digest": "sha256:2f25291ccc900ebaceb5c80edef126cf7641178b30cbc9ae518fad289b0beef0" 2024-11-01T16:45:25.8092216Z }, 2024-11-01T16:45:25.8092452Z { 2024-11-01T16:45:25.8092879Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8093458Z "size": 505, 2024-11-01T16:45:25.8094029Z "digest": "sha256:5cdf685c64c3bd498aa6eb1dfce3a7e059b6f631b0c53f0bed7a78dc821391ef" 2024-11-01T16:45:25.8094691Z }, 2024-11-01T16:45:25.8094926Z { 2024-11-01T16:45:25.8095361Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8096039Z "size": 121477626, 2024-11-01T16:45:25.8096845Z "digest": "sha256:4eab1e4b631667fa7e17e7f2631d88a23b6869b53b25f0bce68b294ff2a25753" 2024-11-01T16:45:25.8097574Z }, 2024-11-01T16:45:25.8097836Z { 2024-11-01T16:45:25.8098334Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8098966Z "size": 109, 2024-11-01T16:45:25.8099594Z "digest": "sha256:7caa1481847d89677e0ca3d3e4cfb637968c2e5fa17d24a78d061617629477a0" 2024-11-01T16:45:25.8100311Z }, 2024-11-01T16:45:25.8100573Z { 2024-11-01T16:45:25.8101278Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8101914Z "size": 489, 2024-11-01T16:45:25.8102569Z "digest": "sha256:dafa8b6784dd9f01c5b1aefb90f8c8e21681ad0e63c7f6a4bb43db1e7e1eb0e2" 2024-11-01T16:45:25.8103319Z }, 2024-11-01T16:45:25.8103582Z { 2024-11-01T16:45:25.8104072Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8104700Z "size": 296, 2024-11-01T16:45:25.8105354Z "digest": "sha256:dfccec371c6fccbb4240dd4ebb403f011e78b159e5dce84145a1ea58da02937e" 2024-11-01T16:45:25.8106107Z }, 2024-11-01T16:45:25.8106368Z { 2024-11-01T16:45:25.8106989Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8107634Z "size": 103, 2024-11-01T16:45:25.8108251Z "digest": "sha256:e41e511e406bc1694c2e5da2a5ee58d9450141ae66b41c3e9ca24989dacc957c" 2024-11-01T16:45:25.8108968Z }, 2024-11-01T16:45:25.8109236Z { 2024-11-01T16:45:25.8109723Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8110367Z "size": 1473, 2024-11-01T16:45:25.8110963Z "digest": "sha256:7cd18613846984c3f25803460d6ec45d1d41db66bd4f9527b365889495421604" 2024-11-01T16:45:25.8111677Z }, 2024-11-01T16:45:25.8111939Z { 2024-11-01T16:45:25.8112430Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8113178Z "size": 426740495, 2024-11-01T16:45:25.8113749Z "digest": "sha256:38aadcbec80791616741dde49d32a2b56f6d3079d6cf8e823b4220a02eabcf1c" 2024-11-01T16:45:25.8114398Z }, 2024-11-01T16:45:25.8114631Z { 2024-11-01T16:45:25.8115068Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8115646Z "size": 164, 2024-11-01T16:45:25.8116195Z "digest": "sha256:f0b80c9d9a164550e271aa228fb110aefe145520e814c2fa44aa345ecb40f815" 2024-11-01T16:45:25.8116844Z }, 2024-11-01T16:45:25.8117079Z { 2024-11-01T16:45:25.8117524Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8118097Z "size": 565, 2024-11-01T16:45:25.8118653Z "digest": "sha256:bad6bd964ba483454e0e3f1a2b99160c38b3900658032c6597e8afb9a4c9c9c8" 2024-11-01T16:45:25.8120830Z }, 2024-11-01T16:45:25.8121073Z { 2024-11-01T16:45:25.8121513Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8122092Z "size": 35806344, 2024-11-01T16:45:25.8122666Z "digest": "sha256:a348147f958bd269f1391a5802dc1c9d7c6414005de995ca5e2d1d8b4867d087" 2024-11-01T16:45:25.8123283Z }, 2024-11-01T16:45:25.8123524Z { 2024-11-01T16:45:25.8123960Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8124531Z "size": 104, 2024-11-01T16:45:25.8125109Z "digest": "sha256:a3dfd1341f5cb909b01d7c3816d56e35af65a2728bffc76a810ca40cfc3a6dc3" 2024-11-01T16:45:25.8125745Z }, 2024-11-01T16:45:25.8125981Z { 2024-11-01T16:45:25.8126421Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8127000Z "size": 425, 2024-11-01T16:45:25.8127561Z "digest": "sha256:c6d6b49273d1097ce850bdcaa1a0f5a19ad7a7897388642b190dcbdfb906f7f1" 2024-11-01T16:45:25.8128211Z }, 2024-11-01T16:45:25.8128450Z { 2024-11-01T16:45:25.8128885Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8129457Z "size": 20262172, 2024-11-01T16:45:25.8130023Z "digest": "sha256:9dec96f6a1c93d47374302f17840755ee42274e1a7c0aa05659042c2e3b3c5fb" 2024-11-01T16:45:25.8130665Z }, 2024-11-01T16:45:25.8130887Z { 2024-11-01T16:45:25.8131326Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8131897Z "size": 440, 2024-11-01T16:45:25.8132450Z "digest": "sha256:98a2657c4e6010d698d905ba80b545f25d8103e49e3da9e17c15359af100b9ca" 2024-11-01T16:45:25.8133087Z }, 2024-11-01T16:45:25.8133309Z { 2024-11-01T16:45:25.8133752Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8134320Z "size": 702, 2024-11-01T16:45:25.8134867Z "digest": "sha256:6c7e3cfd6320c1a2e3323c5b6ed269549b5102449ba19199811783455948b5cd" 2024-11-01T16:45:25.8135506Z }, 2024-11-01T16:45:25.8135821Z { 2024-11-01T16:45:25.8136459Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8137144Z "size": 141, 2024-11-01T16:45:25.8137939Z "digest": "sha256:955cea5b7f364a6e63ba32e894866f7a53b915e04275cc94a39cddd565d9649f" 2024-11-01T16:45:25.8138661Z }, 2024-11-01T16:45:25.8138909Z { 2024-11-01T16:45:25.8139398Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8140042Z "size": 135, 2024-11-01T16:45:25.8140758Z "digest": "sha256:f38b3a72f6838f3a8895fa755edba3b26894109b9b4cf953a046145ffead84c7" 2024-11-01T16:45:25.8141476Z }, 2024-11-01T16:45:25.8141727Z { 2024-11-01T16:45:25.8142212Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8142858Z "size": 32, 2024-11-01T16:45:25.8143491Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:45:25.8144224Z }, 2024-11-01T16:45:25.8144481Z { 2024-11-01T16:45:25.8144955Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8145607Z "size": 195, 2024-11-01T16:45:25.8146260Z "digest": "sha256:c5cb1a39bf9af5ffdd89e40e62f8019aae4b2dccfd34bc5972e4ab5ae8300688" 2024-11-01T16:45:25.8147004Z }, 2024-11-01T16:45:25.8147262Z { 2024-11-01T16:45:25.8147738Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8148578Z "size": 565, 2024-11-01T16:45:25.8149149Z "digest": "sha256:6765d3ee658a82d6af47206ebbed1dd650339da0dacbd454d6c3dbb8f635a9ab" 2024-11-01T16:45:25.8149800Z }, 2024-11-01T16:45:25.8150040Z { 2024-11-01T16:45:25.8150464Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8151039Z "size": 43163402, 2024-11-01T16:45:25.8151603Z "digest": "sha256:cf81cf49414b46152b641c670668b110a04ac48412f9c4dc6613311cf3a0942d" 2024-11-01T16:45:25.8152236Z }, 2024-11-01T16:45:25.8152473Z { 2024-11-01T16:45:25.8152895Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8153544Z "size": 106, 2024-11-01T16:45:25.8154089Z "digest": "sha256:303676327e274ec659dcce26b1e2613e6952e538ab3568f619a5e1a8b603ec89" 2024-11-01T16:45:25.8154723Z }, 2024-11-01T16:45:25.8154956Z { 2024-11-01T16:45:25.8155395Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8155985Z "size": 1401, 2024-11-01T16:45:25.8156558Z "digest": "sha256:09aac48ef4835fec34e2ae2ba94b6081c53d8a2d4fd257e6a0b73349800b9864" 2024-11-01T16:45:25.8157211Z }, 2024-11-01T16:45:25.8157447Z { 2024-11-01T16:45:25.8157896Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8158452Z "size": 702, 2024-11-01T16:45:25.8158998Z "digest": "sha256:6c7e3cfd6320c1a2e3323c5b6ed269549b5102449ba19199811783455948b5cd" 2024-11-01T16:45:25.8159636Z }, 2024-11-01T16:45:25.8159872Z { 2024-11-01T16:45:25.8160311Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8160869Z "size": 136, 2024-11-01T16:45:25.8161450Z "digest": "sha256:ce24c94fd6add217e412ea0dd91c6e0d8cee5c7f27d8700aa6fe56f32a46ef85" 2024-11-01T16:45:25.8162115Z }, 2024-11-01T16:45:25.8162351Z { 2024-11-01T16:45:25.8162790Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8163353Z "size": 120, 2024-11-01T16:45:25.8163905Z "digest": "sha256:8fd476fea7ee8bf52b382a9037418721f15c17b35692cc1f728e52380dfa773c" 2024-11-01T16:45:25.8164547Z }, 2024-11-01T16:45:25.8164781Z { 2024-11-01T16:45:25.8165225Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8165797Z "size": 4310238020, 2024-11-01T16:45:25.8166370Z "digest": "sha256:4cf3da11584940e52b04ab8799eb5c048bd93de4b8ae9a7a8cde0f70aad2326e" 2024-11-01T16:45:25.8167009Z }, 2024-11-01T16:45:25.8167246Z { 2024-11-01T16:45:25.8167680Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8168248Z "size": 176, 2024-11-01T16:45:25.8168795Z "digest": "sha256:1de6b248716cf53af496379d6fd310a4a060fd7d801c9398c3cb180e9b1358a6" 2024-11-01T16:45:25.8169438Z }, 2024-11-01T16:45:25.8169673Z { 2024-11-01T16:45:25.8170110Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8170681Z "size": 907, 2024-11-01T16:45:25.8171241Z "digest": "sha256:f4eec39b74d5ab5024e1ee6c0a12e5c875dff461e7f65e95cab5adf5b7b0e558" 2024-11-01T16:45:25.8171906Z }, 2024-11-01T16:45:25.8172142Z { 2024-11-01T16:45:25.8172579Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8173229Z "size": 702, 2024-11-01T16:45:25.8173934Z "digest": "sha256:6c7e3cfd6320c1a2e3323c5b6ed269549b5102449ba19199811783455948b5cd" 2024-11-01T16:45:25.8174612Z }, 2024-11-01T16:45:25.8174867Z { 2024-11-01T16:45:25.8175330Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8176018Z "size": 135, 2024-11-01T16:45:25.8176817Z "digest": "sha256:bc86c61c49d69deaa90f9b4c601c930d1ae073fcac2b1e505aa15210056bfc49" 2024-11-01T16:45:25.8177548Z }, 2024-11-01T16:45:25.8177812Z { 2024-11-01T16:45:25.8178303Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8178946Z "size": 32, 2024-11-01T16:45:25.8179573Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:45:25.8180291Z }, 2024-11-01T16:45:25.8180554Z { 2024-11-01T16:45:25.8181039Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8181680Z "size": 158, 2024-11-01T16:45:25.8182314Z "digest": "sha256:4790041b71a9d94be048d9d17de7eaf5ccc4fc6824ba1988a6c0e8d5d8bb39d8" 2024-11-01T16:45:25.8183025Z }, 2024-11-01T16:45:25.8183288Z { 2024-11-01T16:45:25.8183776Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8184417Z "size": 2047, 2024-11-01T16:45:25.8185028Z "digest": "sha256:9113890e17732f4e737f88fd824bf1d95f4990250788e87d7eafca3ca8d3c145" 2024-11-01T16:45:25.8185811Z }, 2024-11-01T16:45:25.8186059Z { 2024-11-01T16:45:25.8186549Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8187188Z "size": 7529790, 2024-11-01T16:45:25.8187837Z "digest": "sha256:ad6d4c88cacf37e10c86d50f66a2ba3cc404356243a1399b9986c73beeee06c0" 2024-11-01T16:45:25.8188764Z }, 2024-11-01T16:45:25.8188997Z { 2024-11-01T16:45:25.8189463Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8190069Z "size": 165, 2024-11-01T16:45:25.8190664Z "digest": "sha256:760e1aa1b5e32af659fb52bd61597d28609b31878840d1d752e99f3cfd8ce3ec" 2024-11-01T16:45:25.8191341Z }, 2024-11-01T16:45:25.8191575Z { 2024-11-01T16:45:25.8192034Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8192640Z "size": 7943, 2024-11-01T16:45:25.8193249Z "digest": "sha256:3c4483bbdce846f42d57f201dfe75cda1f5046ec2ac3bcade9042e5adb8c5079" 2024-11-01T16:45:25.8193950Z }, 2024-11-01T16:45:25.8194191Z { 2024-11-01T16:45:25.8194651Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8195256Z "size": 8071, 2024-11-01T16:45:25.8195848Z "digest": "sha256:8764bbbef59040f5d6a8293cfe6e35d1e0178ad129615f69281bcb3b4788b0f0" 2024-11-01T16:45:25.8196532Z }, 2024-11-01T16:45:25.8196869Z { 2024-11-01T16:45:25.8197310Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8197883Z "size": 301, 2024-11-01T16:45:25.8198432Z "digest": "sha256:71826d5dd434366bb74c562c6725263f57f398f696093b870a0e1aa0af3834fe" 2024-11-01T16:45:25.8199064Z }, 2024-11-01T16:45:25.8199303Z { 2024-11-01T16:45:25.8199729Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8200310Z "size": 7630059, 2024-11-01T16:45:25.8200888Z "digest": "sha256:7b4c7f0a396ed0af23fe3f9a24cdd2783c8558177cb3702476b8529d23144f4a" 2024-11-01T16:45:25.8202049Z }, 2024-11-01T16:45:25.8202315Z { 2024-11-01T16:45:25.8202802Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8203447Z "size": 108, 2024-11-01T16:45:25.8204072Z "digest": "sha256:1200fc258863ca7884a699c7e393d80eb77d6dc143b3d42cc2fbba1cb84826c6" 2024-11-01T16:45:25.8204795Z }, 2024-11-01T16:45:25.8205055Z { 2024-11-01T16:45:25.8205531Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8206167Z "size": 54145652, 2024-11-01T16:45:25.8206812Z "digest": "sha256:1b6c965722e8b660c32a130dec4fcc09d3f8e1924d3351990c2317af7dae6911" 2024-11-01T16:45:25.8207670Z }, 2024-11-01T16:45:25.8207933Z { 2024-11-01T16:45:25.8208411Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8209055Z "size": 472, 2024-11-01T16:45:25.8209692Z "digest": "sha256:6cf78d9b64805ade5c1afb24be9ca48e9ffa016e6091322be4ede273692a7d82" 2024-11-01T16:45:25.8210426Z }, 2024-11-01T16:45:25.8210691Z { 2024-11-01T16:45:25.8211183Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8211824Z "size": 1373229322, 2024-11-01T16:45:25.8212496Z "digest": "sha256:accad7e3a239e2b010d87ef44b7db0fc6939535831f7ac907de46fd089b84fcb" 2024-11-01T16:45:25.8213338Z }, 2024-11-01T16:45:25.8213591Z { 2024-11-01T16:45:25.8214271Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8214828Z "size": 103, 2024-11-01T16:45:25.8215394Z "digest": "sha256:b0b6f60fa25319ceba8a48c89bae7353160dafff3324f625d2c62b808da4d98e" 2024-11-01T16:45:25.8216297Z }, 2024-11-01T16:45:25.8216566Z { 2024-11-01T16:45:25.8217058Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8217682Z "size": 569, 2024-11-01T16:45:25.8218320Z "digest": "sha256:acbcfa904fd51dbd77c9918823c59fc72b24bc49e9d3e843530877c2120b8db0" 2024-11-01T16:45:25.8219057Z }, 2024-11-01T16:45:25.8219316Z { 2024-11-01T16:45:25.8219804Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8220565Z "size": 36532055, 2024-11-01T16:45:25.8221212Z "digest": "sha256:024bd3c6bf01b2391930e2c9aa2648a121639bc8eadf45eaf325f74eb1a29822" 2024-11-01T16:45:25.8221947Z }, 2024-11-01T16:45:25.8222210Z { 2024-11-01T16:45:25.8222702Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8223346Z "size": 111, 2024-11-01T16:45:25.8223964Z "digest": "sha256:9cc11594aa2a6ba9e369c7fdf6f7a27181e53af46336a4e825fb3fa888c2cdbc" 2024-11-01T16:45:25.8224701Z }, 2024-11-01T16:45:25.8224962Z { 2024-11-01T16:45:25.8225459Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8226102Z "size": 531, 2024-11-01T16:45:25.8226705Z "digest": "sha256:01fd6f2e9fa39599a7a515a787a896943a1e32b00f54701d01633e1ab2d42449" 2024-11-01T16:45:25.8227420Z }, 2024-11-01T16:45:25.8227685Z { 2024-11-01T16:45:25.8228175Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8228886Z "size": 25817, 2024-11-01T16:45:25.8229447Z "digest": "sha256:d06fa63dc0b7603ba664281c70a33befe32c8b1d0fa355cf9a520049165b88ea" 2024-11-01T16:45:25.8230095Z }, 2024-11-01T16:45:25.8230330Z { 2024-11-01T16:45:25.8230768Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8231337Z "size": 106, 2024-11-01T16:45:25.8231871Z "digest": "sha256:72612b798350e7abf0b29d04df95850b30c9b7cb9b0b15787b9090a3b18ee673" 2024-11-01T16:45:25.8232504Z }, 2024-11-01T16:45:25.8232739Z { 2024-11-01T16:45:25.8233185Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8233760Z "size": 32, 2024-11-01T16:45:25.8234318Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:45:25.8234954Z }, 2024-11-01T16:45:25.8235189Z { 2024-11-01T16:45:25.8235629Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8236199Z "size": 32, 2024-11-01T16:45:25.8236755Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:45:25.8237400Z }, 2024-11-01T16:45:25.8237637Z { 2024-11-01T16:45:25.8238075Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8238643Z "size": 32, 2024-11-01T16:45:25.8239202Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:45:25.8239840Z }, 2024-11-01T16:45:25.8240078Z { 2024-11-01T16:45:25.8240516Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:25.8241169Z "size": 32, 2024-11-01T16:45:25.8241727Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:45:25.8242359Z } 2024-11-01T16:45:25.8242598Z ] 2024-11-01T16:45:25.8242833Z } 2024-11-01T16:45:25.8370171Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-11-01T16:45:25.8370650Z tag=${ECR_DOCKER_IMAGE##*/} 2024-11-01T16:45:25.8371208Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-11-01T16:45:25.8377930Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:25.8378509Z env: 2024-11-01T16:45:25.8378807Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:25.8380039Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:25.8381285Z ##[endgroup] 2024-11-01T16:45:25.8409553Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9-bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:25.8466586Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-11-01T16:45:25.8467253Z with: 2024-11-01T16:45:25.8468469Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:25.8469834Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:25.8470481Z env: 2024-11-01T16:45:25.8470750Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:25.8471092Z ##[endgroup] 2024-11-01T16:45:25.8490171Z ##[group]Run set -x 2024-11-01T16:45:25.8490516Z set -x 2024-11-01T16:45:25.8490796Z set +e 2024-11-01T16:45:25.8491082Z  2024-11-01T16:45:25.8491350Z login() { 2024-11-01T16:45:25.8492031Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-11-01T16:45:25.8492785Z } 2024-11-01T16:45:25.8493055Z  2024-11-01T16:45:25.8493358Z retry () { 2024-11-01T16:45:25.8493729Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-11-01T16:45:25.8494181Z } 2024-11-01T16:45:25.8494442Z  2024-11-01T16:45:25.8494750Z retry login "${DOCKER_REGISTRY}" 2024-11-01T16:45:25.8495168Z  2024-11-01T16:45:25.8495413Z set -e 2024-11-01T16:45:25.8495980Z # ignore output since only exit code is used for conditional 2024-11-01T16:45:25.8496949Z # only pull docker image if it's not available locally 2024-11-01T16:45:25.8497842Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-11-01T16:45:25.8498652Z  retry docker pull "${DOCKER_IMAGE}" 2024-11-01T16:45:25.8499148Z fi 2024-11-01T16:45:25.8505627Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:25.8506184Z env: 2024-11-01T16:45:25.8506491Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:25.8507704Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:25.8509121Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:25.8509721Z ##[endgroup] 2024-11-01T16:45:25.8534377Z + set +e 2024-11-01T16:45:25.8535045Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:25.8535898Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:25.8538467Z + aws ecr get-login-password --region us-east-1 2024-11-01T16:45:25.8539451Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:26.3957710Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-11-01T16:45:26.3960140Z Configure a credential helper to remove this warning. See 2024-11-01T16:45:26.3961598Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-11-01T16:45:26.3962201Z 2024-11-01T16:45:26.3962354Z Login Succeeded 2024-11-01T16:45:26.3973503Z + set -e 2024-11-01T16:45:26.3974884Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:26.4111857Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:26.4114265Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:26.6627098Z bdd298b12da59246147f016e0693ffd722419941: Pulling from pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9 2024-11-01T16:45:26.6628080Z 7a2c55901189: Pulling fs layer 2024-11-01T16:45:26.6628676Z 09d415c238d7: Pulling fs layer 2024-11-01T16:45:26.6629088Z 9fe6e2e61518: Pulling fs layer 2024-11-01T16:45:26.6629504Z 41f16248e682: Pulling fs layer 2024-11-01T16:45:26.6630186Z 95d7b7817039: Pulling fs layer 2024-11-01T16:45:26.6630624Z 8f6c90485347: Pulling fs layer 2024-11-01T16:45:26.6631036Z ab17245097e4: Pulling fs layer 2024-11-01T16:45:26.6631441Z dfecd7e9912b: Pulling fs layer 2024-11-01T16:45:26.6631857Z 464a8f745445: Pulling fs layer 2024-11-01T16:45:26.6632266Z d67c111fa588: Pulling fs layer 2024-11-01T16:45:26.6632685Z 0723abde19c0: Pulling fs layer 2024-11-01T16:45:26.6633095Z 353d86cedcdc: Pulling fs layer 2024-11-01T16:45:26.6633648Z 0299f4e4d528: Pulling fs layer 2024-11-01T16:45:26.6634043Z 86c353c3bc58: Pulling fs layer 2024-11-01T16:45:26.6634455Z 4f10af237965: Pulling fs layer 2024-11-01T16:45:26.6634870Z b7b00fcb07ca: Pulling fs layer 2024-11-01T16:45:26.6635286Z db97392a2fce: Pulling fs layer 2024-11-01T16:45:26.6635696Z 27e2d54f402a: Pulling fs layer 2024-11-01T16:45:26.6636173Z 8269c56845c3: Pulling fs layer 2024-11-01T16:45:26.6636587Z 6c7e3cfd6320: Pulling fs layer 2024-11-01T16:45:26.6636988Z dfecd7e9912b: Waiting 2024-11-01T16:45:26.6637342Z 464a8f745445: Waiting 2024-11-01T16:45:26.6637699Z 82c15385d36d: Pulling fs layer 2024-11-01T16:45:26.6638101Z 5ffae31f204a: Pulling fs layer 2024-11-01T16:45:26.6638554Z d67c111fa588: Waiting 2024-11-01T16:45:26.6638916Z 879b9cee6016: Pulling fs layer 2024-11-01T16:45:26.6639327Z ead67a628392: Pulling fs layer 2024-11-01T16:45:26.6639739Z 2f25291ccc90: Pulling fs layer 2024-11-01T16:45:26.6640149Z 5cdf685c64c3: Pulling fs layer 2024-11-01T16:45:26.6640555Z 0723abde19c0: Waiting 2024-11-01T16:45:26.6640889Z 353d86cedcdc: Waiting 2024-11-01T16:45:26.6641229Z 0299f4e4d528: Waiting 2024-11-01T16:45:26.6641587Z 4eab1e4b6316: Pulling fs layer 2024-11-01T16:45:26.6642004Z 7caa1481847d: Pulling fs layer 2024-11-01T16:45:26.6642421Z dafa8b6784dd: Pulling fs layer 2024-11-01T16:45:26.6642805Z 86c353c3bc58: Waiting 2024-11-01T16:45:26.6643169Z dfccec371c6f: Pulling fs layer 2024-11-01T16:45:26.6643589Z e41e511e406b: Pulling fs layer 2024-11-01T16:45:26.6643984Z db97392a2fce: Waiting 2024-11-01T16:45:26.6644349Z 7cd186138469: Pulling fs layer 2024-11-01T16:45:26.6644735Z 5ffae31f204a: Waiting 2024-11-01T16:45:26.6645099Z 38aadcbec807: Pulling fs layer 2024-11-01T16:45:26.6645517Z f0b80c9d9a16: Pulling fs layer 2024-11-01T16:45:26.6645914Z 879b9cee6016: Waiting 2024-11-01T16:45:26.6646257Z 27e2d54f402a: Waiting 2024-11-01T16:45:26.6646588Z ead67a628392: Waiting 2024-11-01T16:45:26.6646924Z 8269c56845c3: Waiting 2024-11-01T16:45:26.6647283Z bad6bd964ba4: Pulling fs layer 2024-11-01T16:45:26.6647713Z a348147f958b: Pulling fs layer 2024-11-01T16:45:26.6648129Z a3dfd1341f5c: Pulling fs layer 2024-11-01T16:45:26.6648529Z c6d6b49273d1: Pulling fs layer 2024-11-01T16:45:26.6648928Z b7b00fcb07ca: Waiting 2024-11-01T16:45:26.6649292Z 9dec96f6a1c9: Pulling fs layer 2024-11-01T16:45:26.6649705Z 98a2657c4e60: Pulling fs layer 2024-11-01T16:45:26.6650101Z 4f10af237965: Waiting 2024-11-01T16:45:26.6650425Z 41f16248e682: Waiting 2024-11-01T16:45:26.6650897Z 955cea5b7f36: Pulling fs layer 2024-11-01T16:45:26.6651286Z ab17245097e4: Waiting 2024-11-01T16:45:26.6651642Z f38b3a72f683: Pulling fs layer 2024-11-01T16:45:26.6652047Z 4f4fb700ef54: Pulling fs layer 2024-11-01T16:45:26.6652422Z 95d7b7817039: Waiting 2024-11-01T16:45:26.6652773Z c5cb1a39bf9a: Pulling fs layer 2024-11-01T16:45:26.6653174Z 6765d3ee658a: Pulling fs layer 2024-11-01T16:45:26.6653577Z cf81cf49414b: Pulling fs layer 2024-11-01T16:45:26.6653960Z 4eab1e4b6316: Waiting 2024-11-01T16:45:26.6654296Z 303676327e27: Pulling fs layer 2024-11-01T16:45:26.6654679Z 82c15385d36d: Waiting 2024-11-01T16:45:26.6655010Z f38b3a72f683: Waiting 2024-11-01T16:45:26.6655345Z 4f4fb700ef54: Waiting 2024-11-01T16:45:26.6655697Z 09aac48ef483: Pulling fs layer 2024-11-01T16:45:26.6656385Z 2f25291ccc90: Waiting 2024-11-01T16:45:26.6656752Z ce24c94fd6ad: Pulling fs layer 2024-11-01T16:45:26.6657158Z c5cb1a39bf9a: Waiting 2024-11-01T16:45:26.6657526Z 8fd476fea7ee: Pulling fs layer 2024-11-01T16:45:26.6657934Z dafa8b6784dd: Waiting 2024-11-01T16:45:26.6658284Z 4cf3da115849: Pulling fs layer 2024-11-01T16:45:26.6658813Z 1de6b248716c: Pulling fs layer 2024-11-01T16:45:26.6659218Z 5cdf685c64c3: Waiting 2024-11-01T16:45:26.6659585Z f4eec39b74d5: Pulling fs layer 2024-11-01T16:45:26.6659984Z e41e511e406b: Waiting 2024-11-01T16:45:26.6660315Z c6d6b49273d1: Waiting 2024-11-01T16:45:26.6660657Z cf81cf49414b: Waiting 2024-11-01T16:45:26.6661001Z 98a2657c4e60: Waiting 2024-11-01T16:45:26.6661362Z bc86c61c49d6: Pulling fs layer 2024-11-01T16:45:26.6661850Z 4790041b71a9: Pulling fs layer 2024-11-01T16:45:26.6662230Z 7caa1481847d: Waiting 2024-11-01T16:45:26.6662586Z 9113890e1773: Pulling fs layer 2024-11-01T16:45:26.6662997Z ad6d4c88cacf: Pulling fs layer 2024-11-01T16:45:26.6663420Z 760e1aa1b5e3: Pulling fs layer 2024-11-01T16:45:26.6663817Z a3dfd1341f5c: Waiting 2024-11-01T16:45:26.6664169Z 3c4483bbdce8: Pulling fs layer 2024-11-01T16:45:26.6664568Z 9dec96f6a1c9: Waiting 2024-11-01T16:45:26.6664928Z 8764bbbef590: Pulling fs layer 2024-11-01T16:45:26.6665337Z 71826d5dd434: Pulling fs layer 2024-11-01T16:45:26.6665751Z 7b4c7f0a396e: Pulling fs layer 2024-11-01T16:45:26.6666160Z 1200fc258863: Pulling fs layer 2024-11-01T16:45:26.6666532Z 303676327e27: Waiting 2024-11-01T16:45:26.6666885Z 1b6c965722e8: Pulling fs layer 2024-11-01T16:45:26.6667298Z 6cf78d9b6480: Pulling fs layer 2024-11-01T16:45:26.6667709Z accad7e3a239: Pulling fs layer 2024-11-01T16:45:26.6668109Z 8fd476fea7ee: Waiting 2024-11-01T16:45:26.6668574Z b0b6f60fa253: Pulling fs layer 2024-11-01T16:45:26.6669015Z acbcfa904fd5: Pulling fs layer 2024-11-01T16:45:26.6669398Z 9113890e1773: Waiting 2024-11-01T16:45:26.6669741Z 024bd3c6bf01: Pulling fs layer 2024-11-01T16:45:26.6670122Z a348147f958b: Waiting 2024-11-01T16:45:26.6670439Z 71826d5dd434: Waiting 2024-11-01T16:45:26.6670769Z 7b4c7f0a396e: Waiting 2024-11-01T16:45:26.6671117Z 9cc11594aa2a: Pulling fs layer 2024-11-01T16:45:26.6671520Z 01fd6f2e9fa3: Pulling fs layer 2024-11-01T16:45:26.6671906Z 09aac48ef483: Waiting 2024-11-01T16:45:26.6672246Z d06fa63dc0b7: Pulling fs layer 2024-11-01T16:45:26.6672648Z 72612b798350: Pulling fs layer 2024-11-01T16:45:26.6673027Z b0b6f60fa253: Waiting 2024-11-01T16:45:26.6673358Z 6cf78d9b6480: Waiting 2024-11-01T16:45:26.6673693Z f0b80c9d9a16: Waiting 2024-11-01T16:45:26.6674014Z 01fd6f2e9fa3: Waiting 2024-11-01T16:45:26.6674349Z 1de6b248716c: Waiting 2024-11-01T16:45:26.6674680Z accad7e3a239: Waiting 2024-11-01T16:45:26.6675015Z ce24c94fd6ad: Waiting 2024-11-01T16:45:26.6675345Z 8764bbbef590: Waiting 2024-11-01T16:45:26.6675666Z 760e1aa1b5e3: Waiting 2024-11-01T16:45:26.6675996Z 38aadcbec807: Waiting 2024-11-01T16:45:26.6676329Z 3c4483bbdce8: Waiting 2024-11-01T16:45:26.6676656Z 1200fc258863: Waiting 2024-11-01T16:45:26.6676992Z 1b6c965722e8: Waiting 2024-11-01T16:45:26.6677309Z d06fa63dc0b7: Waiting 2024-11-01T16:45:26.6677636Z 72612b798350: Waiting 2024-11-01T16:45:26.6677964Z f4eec39b74d5: Waiting 2024-11-01T16:45:26.6678296Z 9cc11594aa2a: Waiting 2024-11-01T16:45:26.7899826Z 09d415c238d7: Verifying Checksum 2024-11-01T16:45:26.7900336Z 09d415c238d7: Download complete 2024-11-01T16:45:26.8635969Z 41f16248e682: Verifying Checksum 2024-11-01T16:45:26.8636553Z 41f16248e682: Download complete 2024-11-01T16:45:26.9403222Z 95d7b7817039: Download complete 2024-11-01T16:45:27.0081677Z 7a2c55901189: Download complete 2024-11-01T16:45:27.0822711Z ab17245097e4: Verifying Checksum 2024-11-01T16:45:27.0823228Z ab17245097e4: Download complete 2024-11-01T16:45:27.1437122Z dfecd7e9912b: Verifying Checksum 2024-11-01T16:45:27.1437821Z dfecd7e9912b: Download complete 2024-11-01T16:45:27.2219669Z 464a8f745445: Download complete 2024-11-01T16:45:27.2853291Z 9fe6e2e61518: Verifying Checksum 2024-11-01T16:45:27.2853900Z 9fe6e2e61518: Download complete 2024-11-01T16:45:27.3765573Z 0723abde19c0: Verifying Checksum 2024-11-01T16:45:27.3766138Z 0723abde19c0: Download complete 2024-11-01T16:45:27.4480216Z 353d86cedcdc: Download complete 2024-11-01T16:45:27.7471527Z 7a2c55901189: Pull complete 2024-11-01T16:45:28.0526738Z 09d415c238d7: Pull complete 2024-11-01T16:45:28.8681587Z 9fe6e2e61518: Pull complete 2024-11-01T16:45:28.8889503Z 41f16248e682: Pull complete 2024-11-01T16:45:28.9087421Z 95d7b7817039: Pull complete 2024-11-01T16:45:34.0556826Z 0299f4e4d528: Verifying Checksum 2024-11-01T16:45:34.0557348Z 0299f4e4d528: Download complete 2024-11-01T16:45:34.1242446Z 86c353c3bc58: Verifying Checksum 2024-11-01T16:45:34.1242978Z 86c353c3bc58: Download complete 2024-11-01T16:45:34.1967700Z 4f10af237965: Verifying Checksum 2024-11-01T16:45:34.1969485Z 4f10af237965: Download complete 2024-11-01T16:45:34.2962348Z b7b00fcb07ca: Download complete 2024-11-01T16:45:35.2618609Z db97392a2fce: Verifying Checksum 2024-11-01T16:45:35.2619284Z db97392a2fce: Download complete 2024-11-01T16:45:35.3395135Z 27e2d54f402a: Verifying Checksum 2024-11-01T16:45:35.3395648Z 27e2d54f402a: Download complete 2024-11-01T16:45:35.4058036Z 8269c56845c3: Download complete 2024-11-01T16:45:35.4853685Z 6c7e3cfd6320: Download complete 2024-11-01T16:45:40.8317593Z 8f6c90485347: Verifying Checksum 2024-11-01T16:45:40.8318101Z 8f6c90485347: Download complete 2024-11-01T16:45:40.9058206Z 5ffae31f204a: Download complete 2024-11-01T16:45:40.9761218Z 879b9cee6016: Download complete 2024-11-01T16:45:41.0522157Z ead67a628392: Download complete 2024-11-01T16:45:41.1474791Z 2f25291ccc90: Verifying Checksum 2024-11-01T16:45:41.1475466Z 2f25291ccc90: Download complete 2024-11-01T16:45:41.2339867Z 5cdf685c64c3: Verifying Checksum 2024-11-01T16:45:41.2340549Z 5cdf685c64c3: Download complete 2024-11-01T16:45:42.5098124Z 4eab1e4b6316: Verifying Checksum 2024-11-01T16:45:42.5099037Z 4eab1e4b6316: Download complete 2024-11-01T16:45:42.5816699Z 7caa1481847d: Verifying Checksum 2024-11-01T16:45:42.5817318Z 7caa1481847d: Download complete 2024-11-01T16:45:42.6872681Z dafa8b6784dd: Verifying Checksum 2024-11-01T16:45:42.6873367Z dafa8b6784dd: Download complete 2024-11-01T16:45:42.7733488Z dfccec371c6f: Verifying Checksum 2024-11-01T16:45:42.7734309Z dfccec371c6f: Download complete 2024-11-01T16:45:42.8431483Z e41e511e406b: Verifying Checksum 2024-11-01T16:45:42.8432245Z e41e511e406b: Download complete 2024-11-01T16:45:42.9108825Z 7cd186138469: Verifying Checksum 2024-11-01T16:45:42.9109846Z 7cd186138469: Download complete 2024-11-01T16:45:47.2368613Z 38aadcbec807: Verifying Checksum 2024-11-01T16:45:47.2369500Z 38aadcbec807: Download complete 2024-11-01T16:45:47.3291318Z f0b80c9d9a16: Verifying Checksum 2024-11-01T16:45:47.3292195Z f0b80c9d9a16: Download complete 2024-11-01T16:45:47.3883636Z bad6bd964ba4: Verifying Checksum 2024-11-01T16:45:47.3884280Z bad6bd964ba4: Download complete 2024-11-01T16:45:47.8058552Z a348147f958b: Verifying Checksum 2024-11-01T16:45:47.8059130Z a348147f958b: Download complete 2024-11-01T16:45:47.9073992Z a3dfd1341f5c: Verifying Checksum 2024-11-01T16:45:47.9074949Z a3dfd1341f5c: Download complete 2024-11-01T16:45:47.9786838Z c6d6b49273d1: Verifying Checksum 2024-11-01T16:45:47.9787741Z c6d6b49273d1: Download complete 2024-11-01T16:45:48.2263922Z 9dec96f6a1c9: Verifying Checksum 2024-11-01T16:45:48.2264519Z 9dec96f6a1c9: Download complete 2024-11-01T16:45:48.2958102Z 98a2657c4e60: Verifying Checksum 2024-11-01T16:45:48.2958786Z 98a2657c4e60: Download complete 2024-11-01T16:45:48.3747439Z 955cea5b7f36: Verifying Checksum 2024-11-01T16:45:48.3748135Z 955cea5b7f36: Download complete 2024-11-01T16:45:48.4532773Z f38b3a72f683: Download complete 2024-11-01T16:45:48.4620338Z 4f4fb700ef54: Verifying Checksum 2024-11-01T16:45:48.4621128Z 4f4fb700ef54: Download complete 2024-11-01T16:45:48.5325462Z c5cb1a39bf9a: Verifying Checksum 2024-11-01T16:45:48.5325971Z c5cb1a39bf9a: Download complete 2024-11-01T16:45:48.6245542Z 6765d3ee658a: Download complete 2024-11-01T16:45:49.1457252Z cf81cf49414b: Verifying Checksum 2024-11-01T16:45:49.1457816Z cf81cf49414b: Download complete 2024-11-01T16:45:49.2152715Z 303676327e27: Verifying Checksum 2024-11-01T16:45:49.2153225Z 303676327e27: Download complete 2024-11-01T16:45:49.2941498Z 09aac48ef483: Verifying Checksum 2024-11-01T16:45:49.2942080Z 09aac48ef483: Download complete 2024-11-01T16:45:49.3600271Z ce24c94fd6ad: Verifying Checksum 2024-11-01T16:45:49.3601444Z ce24c94fd6ad: Download complete 2024-11-01T16:45:49.4315079Z 8fd476fea7ee: Verifying Checksum 2024-11-01T16:45:49.4315992Z 8fd476fea7ee: Download complete 2024-11-01T16:45:51.7811680Z d67c111fa588: Verifying Checksum 2024-11-01T16:45:51.7812473Z d67c111fa588: Download complete 2024-11-01T16:45:51.8580461Z 1de6b248716c: Download complete 2024-11-01T16:45:51.9652506Z f4eec39b74d5: Download complete 2024-11-01T16:45:52.1097604Z bc86c61c49d6: Verifying Checksum 2024-11-01T16:45:52.1098107Z bc86c61c49d6: Download complete 2024-11-01T16:45:52.1930466Z 4790041b71a9: Download complete 2024-11-01T16:45:52.2664942Z 9113890e1773: Verifying Checksum 2024-11-01T16:45:52.2665470Z 9113890e1773: Download complete 2024-11-01T16:45:52.4165059Z ad6d4c88cacf: Verifying Checksum 2024-11-01T16:45:52.4165609Z ad6d4c88cacf: Download complete 2024-11-01T16:45:52.4891705Z 760e1aa1b5e3: Verifying Checksum 2024-11-01T16:45:52.4892225Z 760e1aa1b5e3: Download complete 2024-11-01T16:45:52.5705784Z 3c4483bbdce8: Download complete 2024-11-01T16:45:52.6352221Z 8764bbbef590: Download complete 2024-11-01T16:45:52.7125901Z 71826d5dd434: Verifying Checksum 2024-11-01T16:45:52.7126453Z 71826d5dd434: Download complete 2024-11-01T16:45:52.8516780Z 7b4c7f0a396e: Verifying Checksum 2024-11-01T16:45:52.8517317Z 7b4c7f0a396e: Download complete 2024-11-01T16:45:52.9310757Z 1200fc258863: Download complete 2024-11-01T16:45:53.5296422Z 1b6c965722e8: Verifying Checksum 2024-11-01T16:45:53.5297146Z 1b6c965722e8: Download complete 2024-11-01T16:45:53.6026565Z 6cf78d9b6480: Verifying Checksum 2024-11-01T16:45:53.6027101Z 6cf78d9b6480: Download complete 2024-11-01T16:45:54.8566940Z 8f6c90485347: Pull complete 2024-11-01T16:45:55.3185058Z ab17245097e4: Pull complete 2024-11-01T16:45:55.8079098Z dfecd7e9912b: Pull complete 2024-11-01T16:45:56.2567074Z 464a8f745445: Pull complete 2024-11-01T16:46:04.7498285Z 82c15385d36d: Verifying Checksum 2024-11-01T16:46:04.8297532Z 82c15385d36d: Download complete 2024-11-01T16:46:04.8298088Z b0b6f60fa253: Verifying Checksum 2024-11-01T16:46:04.8298668Z b0b6f60fa253: Download complete 2024-11-01T16:46:05.0477850Z acbcfa904fd5: Download complete 2024-11-01T16:46:05.4648023Z 024bd3c6bf01: Verifying Checksum 2024-11-01T16:46:05.4648900Z 024bd3c6bf01: Download complete 2024-11-01T16:46:05.5419761Z 9cc11594aa2a: Verifying Checksum 2024-11-01T16:46:05.5420603Z 9cc11594aa2a: Download complete 2024-11-01T16:46:05.6149613Z 01fd6f2e9fa3: Verifying Checksum 2024-11-01T16:46:05.6150173Z 01fd6f2e9fa3: Download complete 2024-11-01T16:46:05.6833044Z d06fa63dc0b7: Download complete 2024-11-01T16:46:05.7547105Z 72612b798350: Verifying Checksum 2024-11-01T16:46:05.7547690Z 72612b798350: Download complete 2024-11-01T16:46:07.3899008Z accad7e3a239: Verifying Checksum 2024-11-01T16:46:07.3899552Z accad7e3a239: Download complete 2024-11-01T16:46:21.2418407Z d67c111fa588: Pull complete 2024-11-01T16:46:21.7187731Z 0723abde19c0: Pull complete 2024-11-01T16:46:22.1315442Z 353d86cedcdc: Pull complete 2024-11-01T16:46:31.7047937Z 0299f4e4d528: Pull complete 2024-11-01T16:46:32.1790215Z 86c353c3bc58: Pull complete 2024-11-01T16:46:32.5817821Z 4cf3da115849: Verifying Checksum 2024-11-01T16:46:32.5818314Z 4cf3da115849: Download complete 2024-11-01T16:46:32.6394368Z 4f10af237965: Pull complete 2024-11-01T16:46:33.0838525Z b7b00fcb07ca: Pull complete 2024-11-01T16:46:35.3968505Z db97392a2fce: Pull complete 2024-11-01T16:46:35.8380400Z 27e2d54f402a: Pull complete 2024-11-01T16:46:36.3131784Z 8269c56845c3: Pull complete 2024-11-01T16:46:36.7630983Z 6c7e3cfd6320: Pull complete 2024-11-01T16:47:17.5932643Z 82c15385d36d: Pull complete 2024-11-01T16:47:17.9236285Z 5ffae31f204a: Pull complete 2024-11-01T16:47:18.4440154Z 879b9cee6016: Pull complete 2024-11-01T16:47:18.8833438Z ead67a628392: Pull complete 2024-11-01T16:47:19.3215585Z 2f25291ccc90: Pull complete 2024-11-01T16:47:19.7913703Z 5cdf685c64c3: Pull complete 2024-11-01T16:47:22.7673110Z 4eab1e4b6316: Pull complete 2024-11-01T16:47:23.2007036Z 7caa1481847d: Pull complete 2024-11-01T16:47:23.6244497Z dafa8b6784dd: Pull complete 2024-11-01T16:47:24.0302999Z dfccec371c6f: Pull complete 2024-11-01T16:47:24.4337098Z e41e511e406b: Pull complete 2024-11-01T16:47:24.9070461Z 7cd186138469: Pull complete 2024-11-01T16:47:32.1427505Z 38aadcbec807: Pull complete 2024-11-01T16:47:32.5602570Z f0b80c9d9a16: Pull complete 2024-11-01T16:47:32.9921107Z bad6bd964ba4: Pull complete 2024-11-01T16:47:34.0078986Z a348147f958b: Pull complete 2024-11-01T16:47:34.3252389Z a3dfd1341f5c: Pull complete 2024-11-01T16:47:34.5345684Z c6d6b49273d1: Pull complete 2024-11-01T16:47:35.1952022Z 9dec96f6a1c9: Pull complete 2024-11-01T16:47:35.5841749Z 98a2657c4e60: Pull complete 2024-11-01T16:47:36.4409767Z 955cea5b7f36: Pull complete 2024-11-01T16:47:36.8094118Z f38b3a72f683: Pull complete 2024-11-01T16:47:37.2886990Z 4f4fb700ef54: Pull complete 2024-11-01T16:47:37.7582143Z c5cb1a39bf9a: Pull complete 2024-11-01T16:47:38.2396048Z 6765d3ee658a: Pull complete 2024-11-01T16:47:40.3386672Z cf81cf49414b: Pull complete 2024-11-01T16:47:40.8031022Z 303676327e27: Pull complete 2024-11-01T16:47:41.1050801Z 09aac48ef483: Pull complete 2024-11-01T16:47:41.6728092Z ce24c94fd6ad: Pull complete 2024-11-01T16:47:41.9814837Z 8fd476fea7ee: Pull complete 2024-11-01T16:48:46.7705744Z 4cf3da115849: Pull complete 2024-11-01T16:48:47.1616636Z 1de6b248716c: Pull complete 2024-11-01T16:48:47.5857041Z f4eec39b74d5: Pull complete 2024-11-01T16:48:48.5162731Z bc86c61c49d6: Pull complete 2024-11-01T16:48:49.4318741Z 4790041b71a9: Pull complete 2024-11-01T16:48:49.9076485Z 9113890e1773: Pull complete 2024-11-01T16:48:50.4724463Z ad6d4c88cacf: Pull complete 2024-11-01T16:48:50.9384082Z 760e1aa1b5e3: Pull complete 2024-11-01T16:48:51.3937645Z 3c4483bbdce8: Pull complete 2024-11-01T16:48:51.8503425Z 8764bbbef590: Pull complete 2024-11-01T16:48:52.2934383Z 71826d5dd434: Pull complete 2024-11-01T16:48:53.4981476Z 7b4c7f0a396e: Pull complete 2024-11-01T16:48:53.9642514Z 1200fc258863: Pull complete 2024-11-01T16:48:55.9262581Z 1b6c965722e8: Pull complete 2024-11-01T16:48:56.3003552Z 6cf78d9b6480: Pull complete 2024-11-01T16:49:09.2093378Z accad7e3a239: Pull complete 2024-11-01T16:49:09.6436930Z b0b6f60fa253: Pull complete 2024-11-01T16:49:10.1128574Z acbcfa904fd5: Pull complete 2024-11-01T16:49:11.0250971Z 024bd3c6bf01: Pull complete 2024-11-01T16:49:11.5077467Z 9cc11594aa2a: Pull complete 2024-11-01T16:49:11.8982347Z 01fd6f2e9fa3: Pull complete 2024-11-01T16:49:12.2922246Z d06fa63dc0b7: Pull complete 2024-11-01T16:49:12.7695624Z 72612b798350: Pull complete 2024-11-01T16:49:14.1409002Z Digest: sha256:7e55ca35c7b3a515f97404dcaef8e9d5373f6e358ac63c424d2d80dda1e618f7 2024-11-01T16:49:14.2274950Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:49:14.2809841Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:49:14.2869760Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:49:14.2871221Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:49:14.2878479Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:49:14.2879028Z env: 2024-11-01T16:49:14.2879329Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:49:14.2879696Z ##[endgroup] 2024-11-01T16:49:14.3121032Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-11-01T16:49:14.3121579Z with: 2024-11-01T16:49:14.3121851Z driver-version: 550.54.15 2024-11-01T16:49:14.3122187Z env: 2024-11-01T16:49:14.3122457Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:49:14.3122964Z ##[endgroup] 2024-11-01T16:49:14.3373955Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-11-01T16:49:14.3374577Z with: 2024-11-01T16:49:14.3374861Z timeout_minutes: 10 2024-11-01T16:49:14.3375192Z max_attempts: 3 2024-11-01T16:49:14.3417435Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-11-01T16:49:14.3458295Z retry_wait_seconds: 10 2024-11-01T16:49:14.3458701Z polling_interval_seconds: 1 2024-11-01T16:49:14.3459114Z warning_on_retry: true 2024-11-01T16:49:14.3459499Z continue_on_error: false 2024-11-01T16:49:14.3459857Z env: 2024-11-01T16:49:14.3460145Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:49:14.3460535Z DRIVER_VERSION: 550.54.15 2024-11-01T16:49:14.3460913Z ##[endgroup] 2024-11-01T16:49:14.4288208Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-11-01T16:49:14.4290398Z + pre_install_nvidia_driver_amzn2 2024-11-01T16:49:14.4291115Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-11-01T16:49:14.8284888Z No match for argument: nvidia-driver-latest-dkms 2024-11-01T16:49:14.8285479Z No packages marked for removal. 2024-11-01T16:49:14.8354498Z Dependencies resolved. 2024-11-01T16:49:14.8362594Z Nothing to do. 2024-11-01T16:49:14.8363184Z Complete! 2024-11-01T16:49:14.9626012Z + install_nvidia_driver_common 2024-11-01T16:49:14.9628471Z + echo 'Before installing NVIDIA driver' 2024-11-01T16:49:14.9629065Z + lspci 2024-11-01T16:49:14.9630125Z Before installing NVIDIA driver 2024-11-01T16:49:14.9766400Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-11-01T16:49:14.9767274Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-11-01T16:49:14.9768357Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-11-01T16:49:14.9769268Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-11-01T16:49:14.9770207Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-11-01T16:49:14.9771122Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-11-01T16:49:14.9771983Z 00:1b.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-11-01T16:49:14.9772798Z 00:1c.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-11-01T16:49:14.9773863Z 00:1d.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-11-01T16:49:14.9774678Z 00:1e.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-11-01T16:49:14.9775636Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-11-01T16:49:14.9776436Z + lsmod 2024-11-01T16:49:14.9804622Z Module Size Used by 2024-11-01T16:49:14.9805122Z ib_core 454656 0 2024-11-01T16:49:14.9805537Z veth 36864 0 2024-11-01T16:49:14.9805954Z nvidia_modeset 1351680 0 2024-11-01T16:49:14.9806423Z video 65536 1 nvidia_modeset 2024-11-01T16:49:14.9806918Z wmi 36864 1 video 2024-11-01T16:49:14.9807360Z nvidia_uvm 4706304 0 2024-11-01T16:49:14.9807872Z nvidia 54071296 19 nvidia_uvm,nvidia_modeset 2024-11-01T16:49:14.9808447Z drm 602112 1 nvidia 2024-11-01T16:49:14.9809071Z drm_panel_orientation_quirks 32768 1 drm 2024-11-01T16:49:14.9809886Z backlight 24576 3 video,drm,nvidia_modeset 2024-11-01T16:49:14.9810479Z i2c_core 106496 2 nvidia,drm 2024-11-01T16:49:14.9810958Z xt_conntrack 16384 1 2024-11-01T16:49:14.9811366Z nft_chain_nat 16384 3 2024-11-01T16:49:14.9811789Z xt_MASQUERADE 20480 1 2024-11-01T16:49:14.9812304Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-11-01T16:49:14.9812861Z nf_conntrack_netlink 57344 0 2024-11-01T16:49:14.9813532Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-11-01T16:49:14.9814371Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-11-01T16:49:14.9814880Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-11-01T16:49:14.9815348Z xfrm_user 57344 1 2024-11-01T16:49:14.9815773Z xfrm_algo 16384 1 xfrm_user 2024-11-01T16:49:14.9816330Z xt_addrtype 16384 2 2024-11-01T16:49:14.9816926Z nft_compat 20480 4 2024-11-01T16:49:14.9817429Z nf_tables 311296 57 nft_compat,nft_chain_nat 2024-11-01T16:49:14.9818138Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-11-01T16:49:14.9818762Z br_netfilter 36864 0 2024-11-01T16:49:14.9819286Z bridge 323584 1 br_netfilter 2024-11-01T16:49:14.9819860Z stp 16384 1 bridge 2024-11-01T16:49:14.9820330Z llc 16384 2 bridge,stp 2024-11-01T16:49:14.9820785Z overlay 167936 0 2024-11-01T16:49:14.9821192Z tls 114688 0 2024-11-01T16:49:14.9821591Z nls_ascii 16384 1 2024-11-01T16:49:14.9822000Z nls_cp437 20480 1 2024-11-01T16:49:14.9822402Z vfat 24576 1 2024-11-01T16:49:14.9822805Z fat 86016 1 vfat 2024-11-01T16:49:14.9823240Z sunrpc 692224 1 2024-11-01T16:49:14.9823643Z ena 167936 0 2024-11-01T16:49:14.9824049Z skx_edac_common 28672 0 2024-11-01T16:49:14.9824454Z i8042 45056 0 2024-11-01T16:49:14.9824858Z serio 28672 3 i8042 2024-11-01T16:49:14.9825294Z button 24576 0 2024-11-01T16:49:14.9825704Z ghash_clmulni_intel 16384 0 2024-11-01T16:49:14.9826123Z sch_fq_codel 20480 33 2024-11-01T16:49:14.9826534Z dm_mod 188416 0 2024-11-01T16:49:14.9826918Z fuse 163840 1 2024-11-01T16:49:14.9827325Z dax 45056 1 dm_mod 2024-11-01T16:49:14.9827765Z configfs 57344 1 2024-11-01T16:49:14.9828177Z loop 36864 0 2024-11-01T16:49:14.9828686Z dmi_sysfs 20480 0 2024-11-01T16:49:14.9829070Z crc32_pclmul 16384 0 2024-11-01T16:49:14.9829469Z crc32c_intel 24576 0 2024-11-01T16:49:14.9829863Z efivarfs 24576 1 2024-11-01T16:49:14.9830363Z + modinfo nvidia 2024-11-01T16:49:14.9831185Z filename: /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-11-01T16:49:14.9831992Z alias: char-major-195-* 2024-11-01T16:49:14.9832554Z version: 550.54.15 2024-11-01T16:49:14.9832913Z supported: external 2024-11-01T16:49:14.9833276Z license: NVIDIA 2024-11-01T16:49:14.9833681Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-11-01T16:49:14.9834212Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-11-01T16:49:14.9834715Z srcversion: 833721318DA517F0C2FEC97 2024-11-01T16:49:14.9835213Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-11-01T16:49:14.9835763Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-11-01T16:49:14.9836302Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-11-01T16:49:14.9836839Z depends: i2c-core,drm 2024-11-01T16:49:14.9837222Z retpoline: Y 2024-11-01T16:49:14.9837531Z name: nvidia 2024-11-01T16:49:14.9838232Z vermagic: 6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-11-01T16:49:14.9839036Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-11-01T16:49:14.9839846Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-11-01T16:49:14.9840511Z parm: NVreg_ResmanDebugLevel:int 2024-11-01T16:49:14.9840997Z parm: NVreg_RmLogonRC:int 2024-11-01T16:49:14.9841458Z parm: NVreg_ModifyDeviceFiles:int 2024-11-01T16:49:14.9841958Z parm: NVreg_DeviceFileUID:int 2024-11-01T16:49:14.9842436Z parm: NVreg_DeviceFileGID:int 2024-11-01T16:49:14.9842916Z parm: NVreg_DeviceFileMode:int 2024-11-01T16:49:14.9843502Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-11-01T16:49:14.9844122Z parm: NVreg_UsePageAttributeTable:int 2024-11-01T16:49:14.9844749Z parm: NVreg_EnablePCIeGen3:int 2024-11-01T16:49:14.9845201Z parm: NVreg_EnableMSI:int 2024-11-01T16:49:14.9845639Z parm: NVreg_TCEBypassMode:int 2024-11-01T16:49:14.9846123Z parm: NVreg_EnableStreamMemOPs:int 2024-11-01T16:49:14.9846697Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-11-01T16:49:14.9847334Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-11-01T16:49:14.9847945Z parm: NVreg_EnableS0ixPowerManagement:int 2024-11-01T16:49:14.9848612Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-11-01T16:49:14.9849272Z parm: NVreg_DynamicPowerManagement:int 2024-11-01T16:49:14.9849942Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-11-01T16:49:14.9850580Z parm: NVreg_EnableGpuFirmware:int 2024-11-01T16:49:14.9851106Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-11-01T16:49:14.9851683Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-11-01T16:49:14.9852277Z parm: NVreg_EnableUserNUMAManagement:int 2024-11-01T16:49:14.9852979Z parm: NVreg_MemoryPoolSize:int 2024-11-01T16:49:14.9853488Z parm: NVreg_KMallocHeapMaxSize:int 2024-11-01T16:49:14.9854006Z parm: NVreg_VMallocHeapMaxSize:int 2024-11-01T16:49:14.9854524Z parm: NVreg_IgnoreMMIOCheck:int 2024-11-01T16:49:14.9855021Z parm: NVreg_NvLinkDisable:int 2024-11-01T16:49:14.9855578Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-11-01T16:49:14.9856273Z parm: NVreg_RegisterPCIDriver:int 2024-11-01T16:49:14.9856967Z parm: NVreg_EnableResizableBar:int 2024-11-01T16:49:14.9857519Z parm: NVreg_EnableDbgBreakpoint:int 2024-11-01T16:49:14.9858092Z parm: NVreg_EnableNonblockingOpen:int 2024-11-01T16:49:14.9858649Z parm: NVreg_RegistryDwords:charp 2024-11-01T16:49:14.9859222Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-11-01T16:49:14.9859780Z parm: NVreg_RmMsg:charp 2024-11-01T16:49:14.9860277Z parm: NVreg_GpuBlacklist:charp 2024-11-01T16:49:14.9860810Z parm: NVreg_TemporaryFilePath:charp 2024-11-01T16:49:14.9861363Z parm: NVreg_ExcludedGpus:charp 2024-11-01T16:49:14.9861890Z parm: NVreg_DmaRemapPeerMmio:int 2024-11-01T16:49:14.9862435Z parm: NVreg_RmNvlinkBandwidth:charp 2024-11-01T16:49:14.9863061Z parm: NVreg_ImexChannelCount:int 2024-11-01T16:49:14.9863580Z parm: rm_firmware_active:charp 2024-11-01T16:49:14.9864066Z + HAS_NVIDIA_DRIVER=0 2024-11-01T16:49:14.9864506Z ++ command -v nvidia-smi 2024-11-01T16:49:14.9864975Z + '[' -x /usr/bin/nvidia-smi ']' 2024-11-01T16:49:14.9865395Z + set +e 2024-11-01T16:49:14.9865992Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-11-01T16:49:15.0141102Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-11-01T16:49:15.0141601Z + NVIDIA_SMI_STATUS=0 2024-11-01T16:49:15.0142092Z + '[' 0 -ne 0 ']' 2024-11-01T16:49:15.0142498Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-11-01T16:49:15.0142919Z + HAS_NVIDIA_DRIVER=1 2024-11-01T16:49:15.0143776Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-11-01T16:49:15.0144605Z + set -e 2024-11-01T16:49:15.0144918Z + '[' 1 -eq 0 ']' 2024-11-01T16:49:15.0145581Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-11-01T16:49:15.0146617Z + post_install_nvidia_driver_common 2024-11-01T16:49:15.0147068Z + sudo modprobe nvidia 2024-11-01T16:49:15.1875445Z + echo 'After installing NVIDIA driver' 2024-11-01T16:49:15.1875939Z + lspci 2024-11-01T16:49:15.1876571Z After installing NVIDIA driver 2024-11-01T16:49:15.2003662Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-11-01T16:49:15.2004562Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-11-01T16:49:15.2005680Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-11-01T16:49:15.2006622Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-11-01T16:49:15.2007582Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-11-01T16:49:15.2008528Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-11-01T16:49:15.2009403Z 00:1b.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-11-01T16:49:15.2010261Z 00:1c.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-11-01T16:49:15.2011090Z 00:1d.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-11-01T16:49:15.2011909Z 00:1e.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-11-01T16:49:15.2012889Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-11-01T16:49:15.2013577Z + lsmod 2024-11-01T16:49:15.2032866Z Module Size Used by 2024-11-01T16:49:15.2033370Z ib_core 454656 0 2024-11-01T16:49:15.2033795Z veth 36864 0 2024-11-01T16:49:15.2034203Z nvidia_modeset 1351680 0 2024-11-01T16:49:15.2034654Z video 65536 1 nvidia_modeset 2024-11-01T16:49:15.2035146Z wmi 36864 1 video 2024-11-01T16:49:15.2035561Z nvidia_uvm 4706304 0 2024-11-01T16:49:15.2036062Z nvidia 54071296 19 nvidia_uvm,nvidia_modeset 2024-11-01T16:49:15.2036620Z drm 602112 1 nvidia 2024-11-01T16:49:15.2037107Z drm_panel_orientation_quirks 32768 1 drm 2024-11-01T16:49:15.2037710Z backlight 24576 3 video,drm,nvidia_modeset 2024-11-01T16:49:15.2038277Z i2c_core 106496 2 nvidia,drm 2024-11-01T16:49:15.2038723Z xt_conntrack 16384 1 2024-11-01T16:49:15.2039130Z nft_chain_nat 16384 3 2024-11-01T16:49:15.2039542Z xt_MASQUERADE 20480 1 2024-11-01T16:49:15.2040039Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-11-01T16:49:15.2040583Z nf_conntrack_netlink 57344 0 2024-11-01T16:49:15.2041235Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-11-01T16:49:15.2041949Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-11-01T16:49:15.2042462Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-11-01T16:49:15.2042928Z xfrm_user 57344 1 2024-11-01T16:49:15.2043351Z xfrm_algo 16384 1 xfrm_user 2024-11-01T16:49:15.2044002Z xt_addrtype 16384 2 2024-11-01T16:49:15.2044399Z nft_compat 20480 4 2024-11-01T16:49:15.2044897Z nf_tables 311296 57 nft_compat,nft_chain_nat 2024-11-01T16:49:15.2045589Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-11-01T16:49:15.2046192Z br_netfilter 36864 0 2024-11-01T16:49:15.2046636Z bridge 323584 1 br_netfilter 2024-11-01T16:49:15.2047119Z stp 16384 1 bridge 2024-11-01T16:49:15.2047560Z llc 16384 2 bridge,stp 2024-11-01T16:49:15.2048034Z overlay 167936 0 2024-11-01T16:49:15.2048430Z tls 114688 0 2024-11-01T16:49:15.2049009Z nls_ascii 16384 1 2024-11-01T16:49:15.2049414Z nls_cp437 20480 1 2024-11-01T16:49:15.2049804Z vfat 24576 1 2024-11-01T16:49:15.2050214Z fat 86016 1 vfat 2024-11-01T16:49:15.2050646Z sunrpc 692224 1 2024-11-01T16:49:15.2051166Z ena 167936 0 2024-11-01T16:49:15.2051578Z skx_edac_common 28672 0 2024-11-01T16:49:15.2051972Z i8042 45056 0 2024-11-01T16:49:15.2052384Z serio 28672 3 i8042 2024-11-01T16:49:15.2052819Z button 24576 0 2024-11-01T16:49:15.2053234Z ghash_clmulni_intel 16384 0 2024-11-01T16:49:15.2053657Z sch_fq_codel 20480 33 2024-11-01T16:49:15.2054056Z dm_mod 188416 0 2024-11-01T16:49:15.2054456Z fuse 163840 1 2024-11-01T16:49:15.2054871Z dax 45056 1 dm_mod 2024-11-01T16:49:15.2055316Z configfs 57344 1 2024-11-01T16:49:15.2055722Z loop 36864 0 2024-11-01T16:49:15.2056248Z dmi_sysfs 20480 0 2024-11-01T16:49:15.2056659Z crc32_pclmul 16384 0 2024-11-01T16:49:15.2057070Z crc32c_intel 24576 0 2024-11-01T16:49:15.2057481Z efivarfs 24576 1 2024-11-01T16:49:15.2057887Z + modinfo nvidia 2024-11-01T16:49:15.2058736Z filename: /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-11-01T16:49:15.2059554Z alias: char-major-195-* 2024-11-01T16:49:15.2059978Z version: 550.54.15 2024-11-01T16:49:15.2060360Z supported: external 2024-11-01T16:49:15.2060728Z license: NVIDIA 2024-11-01T16:49:15.2061145Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-11-01T16:49:15.2061690Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-11-01T16:49:15.2062196Z srcversion: 833721318DA517F0C2FEC97 2024-11-01T16:49:15.2062720Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-11-01T16:49:15.2063283Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-11-01T16:49:15.2063837Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-11-01T16:49:15.2064392Z depends: i2c-core,drm 2024-11-01T16:49:15.2064773Z retpoline: Y 2024-11-01T16:49:15.2065105Z name: nvidia 2024-11-01T16:49:15.2065830Z vermagic: 6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-11-01T16:49:15.2066673Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-11-01T16:49:15.2067429Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-11-01T16:49:15.2068244Z parm: NVreg_ResmanDebugLevel:int 2024-11-01T16:49:15.2068714Z parm: NVreg_RmLogonRC:int 2024-11-01T16:49:15.2069186Z parm: NVreg_ModifyDeviceFiles:int 2024-11-01T16:49:15.2069685Z parm: NVreg_DeviceFileUID:int 2024-11-01T16:49:15.2070158Z parm: NVreg_DeviceFileGID:int 2024-11-01T16:49:15.2070639Z parm: NVreg_DeviceFileMode:int 2024-11-01T16:49:15.2071214Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-11-01T16:49:15.2071845Z parm: NVreg_UsePageAttributeTable:int 2024-11-01T16:49:15.2072373Z parm: NVreg_EnablePCIeGen3:int 2024-11-01T16:49:15.2072840Z parm: NVreg_EnableMSI:int 2024-11-01T16:49:15.2073292Z parm: NVreg_TCEBypassMode:int 2024-11-01T16:49:15.2073885Z parm: NVreg_EnableStreamMemOPs:int 2024-11-01T16:49:15.2074483Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-11-01T16:49:15.2075139Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-11-01T16:49:15.2075771Z parm: NVreg_EnableS0ixPowerManagement:int 2024-11-01T16:49:15.2076460Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-11-01T16:49:15.2077137Z parm: NVreg_DynamicPowerManagement:int 2024-11-01T16:49:15.2077814Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-11-01T16:49:15.2078483Z parm: NVreg_EnableGpuFirmware:int 2024-11-01T16:49:15.2079027Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-11-01T16:49:15.2079625Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-11-01T16:49:15.2080236Z parm: NVreg_EnableUserNUMAManagement:int 2024-11-01T16:49:15.2080784Z parm: NVreg_MemoryPoolSize:int 2024-11-01T16:49:15.2081349Z parm: NVreg_KMallocHeapMaxSize:int 2024-11-01T16:49:15.2081883Z parm: NVreg_VMallocHeapMaxSize:int 2024-11-01T16:49:15.2082404Z parm: NVreg_IgnoreMMIOCheck:int 2024-11-01T16:49:15.2082893Z parm: NVreg_NvLinkDisable:int 2024-11-01T16:49:15.2083452Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-11-01T16:49:15.2084020Z parm: NVreg_RegisterPCIDriver:int 2024-11-01T16:49:15.2084550Z parm: NVreg_EnableResizableBar:int 2024-11-01T16:49:15.2085098Z parm: NVreg_EnableDbgBreakpoint:int 2024-11-01T16:49:15.2085649Z parm: NVreg_EnableNonblockingOpen:int 2024-11-01T16:49:15.2086184Z parm: NVreg_RegistryDwords:charp 2024-11-01T16:49:15.2086720Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-11-01T16:49:15.2087251Z parm: NVreg_RmMsg:charp 2024-11-01T16:49:15.2087692Z parm: NVreg_GpuBlacklist:charp 2024-11-01T16:49:15.2088199Z parm: NVreg_TemporaryFilePath:charp 2024-11-01T16:49:15.2088718Z parm: NVreg_ExcludedGpus:charp 2024-11-01T16:49:15.2089217Z parm: NVreg_DmaRemapPeerMmio:int 2024-11-01T16:49:15.2089728Z parm: NVreg_RmNvlinkBandwidth:charp 2024-11-01T16:49:15.2090250Z parm: NVreg_ImexChannelCount:int 2024-11-01T16:49:15.2090748Z parm: rm_firmware_active:charp 2024-11-01T16:49:15.2091184Z + set +e 2024-11-01T16:49:15.2091504Z + nvidia-smi 2024-11-01T16:49:15.2548100Z Fri Nov 1 16:49:15 2024 2024-11-01T16:49:15.2548964Z +-----------------------------------------------------------------------------------------+ 2024-11-01T16:49:15.2549932Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-11-01T16:49:15.2550856Z |-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:49:15.2551786Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-11-01T16:49:15.2552819Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-11-01T16:49:15.2553633Z | | | MIG M. | 2024-11-01T16:49:15.2554240Z |=========================================+========================+======================| 2024-11-01T16:49:15.2718522Z | 0 Tesla T4 On | 00000000:00:1B.0 Off | 0 | 2024-11-01T16:49:15.2719368Z | N/A 28C P8 9W / 70W | 0MiB / 15360MiB | 0% Default | 2024-11-01T16:49:15.2720094Z | | | N/A | 2024-11-01T16:49:15.2720924Z +-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:49:15.2875485Z | 1 Tesla T4 On | 00000000:00:1C.0 Off | 0 | 2024-11-01T16:49:15.2876337Z | N/A 29C P8 10W / 70W | 0MiB / 15360MiB | 0% Default | 2024-11-01T16:49:15.2877281Z | | | N/A | 2024-11-01T16:49:15.2878111Z +-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:49:15.3033208Z | 2 Tesla T4 On | 00000000:00:1D.0 Off | 0 | 2024-11-01T16:49:15.3034056Z | N/A 27C P8 9W / 70W | 0MiB / 15360MiB | 0% Default | 2024-11-01T16:49:15.3034784Z | | | N/A | 2024-11-01T16:49:15.3035621Z +-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:49:15.3185183Z | 3 Tesla T4 On | 00000000:00:1E.0 Off | 0 | 2024-11-01T16:49:15.3186050Z | N/A 27C P8 9W / 70W | 0MiB / 15360MiB | 0% Default | 2024-11-01T16:49:15.3186794Z | | | N/A | 2024-11-01T16:49:15.3187853Z +-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:49:15.3203300Z 2024-11-01T16:49:15.3204105Z +-----------------------------------------------------------------------------------------+ 2024-11-01T16:49:15.3204835Z | Processes: | 2024-11-01T16:49:15.3205648Z | GPU GI CI PID Type Process name GPU Memory | 2024-11-01T16:49:15.3206437Z | ID ID Usage | 2024-11-01T16:49:15.3207059Z |=========================================================================================| 2024-11-01T16:49:15.3232365Z | No running processes found | 2024-11-01T16:49:15.3233371Z +-----------------------------------------------------------------------------------------+ 2024-11-01T16:49:15.9128418Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-11-01T16:49:15.9334045Z Tesla T4 2024-11-01T16:49:15.9410620Z + NVIDIA_SMI_STATUS=0 2024-11-01T16:49:15.9411140Z + '[' 0 -eq 0 ']' 2024-11-01T16:49:15.9411574Z + echo 'INFO: Ignoring allowed status 0' 2024-11-01T16:49:15.9412038Z + set -e 2024-11-01T16:49:15.9412346Z INFO: Ignoring allowed status 0 2024-11-01T16:49:15.9418285Z == Installing nvidia container toolkit for amzn2023 == 2024-11-01T16:49:15.9421662Z + sudo yum install -y yum-utils 2024-11-01T16:49:16.4098255Z Last metadata expiration check: 0:32:36 ago on Fri Nov 1 16:16:40 2024. 2024-11-01T16:49:16.4359588Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-11-01T16:49:16.4755708Z Dependencies resolved. 2024-11-01T16:49:16.4938211Z Nothing to do. 2024-11-01T16:49:16.4938579Z Complete! 2024-11-01T16:49:16.6474469Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-11-01T16:49:16.6475761Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-11-01T16:49:16.6477343Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-11-01T16:49:16.9393238Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-11-01T16:49:17.0091632Z + sudo yum install -y nvidia-docker2 2024-11-01T16:49:17.5491170Z nvidia-container-toolkit 9.8 kB/s | 833 B 00:00 2024-11-01T16:49:17.5758779Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-11-01T16:49:17.6143372Z Dependencies resolved. 2024-11-01T16:49:17.6329355Z Nothing to do. 2024-11-01T16:49:17.6330241Z Complete! 2024-11-01T16:49:17.7580782Z + sudo systemctl restart docker 2024-11-01T16:50:58.3153890Z nvidia-persistenced failed to initialize. Check syslog for more details. 2024-11-01T16:50:58.3663568Z Fri Nov 1 16:50:58 2024 2024-11-01T16:50:58.3664496Z +-----------------------------------------------------------------------------------------+ 2024-11-01T16:50:58.3666737Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-11-01T16:50:58.3667743Z |-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:50:58.3668816Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-11-01T16:50:58.3669856Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-11-01T16:50:58.3670648Z | | | MIG M. | 2024-11-01T16:50:58.3671255Z |=========================================+========================+======================| 2024-11-01T16:50:58.3832029Z | 0 Tesla T4 On | 00000000:00:1B.0 Off | 0 | 2024-11-01T16:50:58.3834191Z | N/A 28C P8 9W / 70W | 0MiB / 15360MiB | 0% Default | 2024-11-01T16:50:58.3834932Z | | | N/A | 2024-11-01T16:50:58.3835786Z +-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:50:58.3981648Z | 1 Tesla T4 On | 00000000:00:1C.0 Off | 0 | 2024-11-01T16:50:58.3982551Z | N/A 29C P8 9W / 70W | 0MiB / 15360MiB | 0% Default | 2024-11-01T16:50:58.3983319Z | | | N/A | 2024-11-01T16:50:58.3984178Z +-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:50:58.4135143Z | 2 Tesla T4 On | 00000000:00:1D.0 Off | 0 | 2024-11-01T16:50:58.4136146Z | N/A 27C P8 9W / 70W | 0MiB / 15360MiB | 0% Default | 2024-11-01T16:50:58.4137085Z | | | N/A | 2024-11-01T16:50:58.4138759Z +-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:50:58.4285234Z | 3 Tesla T4 On | 00000000:00:1E.0 Off | 0 | 2024-11-01T16:50:58.4286219Z | N/A 27C P8 9W / 70W | 0MiB / 15360MiB | 0% Default | 2024-11-01T16:50:58.4286958Z | | | N/A | 2024-11-01T16:50:58.4287805Z +-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:50:58.4298839Z 2024-11-01T16:50:58.4299671Z +-----------------------------------------------------------------------------------------+ 2024-11-01T16:50:58.4300457Z | Processes: | 2024-11-01T16:50:58.4301449Z | GPU GI CI PID Type Process name GPU Memory | 2024-11-01T16:50:58.4302255Z | ID ID Usage | 2024-11-01T16:50:58.4302927Z |=========================================================================================| 2024-11-01T16:50:58.4328787Z | No running processes found | 2024-11-01T16:50:58.4329853Z +-----------------------------------------------------------------------------------------+ 2024-11-01T16:50:59.5282172Z Command completed after 1 attempt(s). 2024-11-01T16:50:59.5373624Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-11-01T16:50:59.5374499Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-11-01T16:50:59.5375298Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-11-01T16:50:59.5376311Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:50:59.5386997Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:50:59.5387700Z env: 2024-11-01T16:50:59.5387997Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:50:59.5388555Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:50:59.5389048Z ##[endgroup] 2024-11-01T16:50:59.7981729Z Defaulting to user installation because normal site-packages is not writeable 2024-11-01T16:50:59.8155356Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.9/site-packages (5.9.1) 2024-11-01T16:50:59.8160092Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.9/site-packages (11.525.84) 2024-11-01T16:50:59.9422877Z Prepare all required actions 2024-11-01T16:50:59.9423459Z Getting action download info 2024-11-01T16:51:00.0718151Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-11-01T16:51:00.2560543Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-11-01T16:51:00.3850815Z ##[group]Run ./.github/actions/download-build-artifacts 2024-11-01T16:51:00.3851349Z with: 2024-11-01T16:51:00.3851680Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-11-01T16:51:00.3852242Z s3-bucket: gha-artifacts 2024-11-01T16:51:00.3852595Z env: 2024-11-01T16:51:00.3852869Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:00.3853346Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:00.3853865Z ##[endgroup] 2024-11-01T16:51:00.3893222Z ##[group]Run seemethere/download-artifact-s3@v4 2024-11-01T16:51:00.3893701Z with: 2024-11-01T16:51:00.3894068Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-11-01T16:51:00.3894540Z s3-bucket: gha-artifacts 2024-11-01T16:51:00.3894890Z region: us-east-1 2024-11-01T16:51:00.3895214Z env: 2024-11-01T16:51:00.3895486Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:00.3896248Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:00.3896807Z ##[endgroup] 2024-11-01T16:51:00.8924467Z (node:99451) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-11-01T16:51:00.8925284Z 2024-11-01T16:51:00.8925582Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-11-01T16:51:00.8926406Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-11-01T16:51:00.8927424Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-11-01T16:51:00.9717912Z Found 1 objects with prefix pytorch/pytorch/11632514903/linux-focal-cuda11.8-py3.10-gcc9/ 2024-11-01T16:51:00.9719174Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-11-01T16:51:21.8079165Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-11-01T16:51:21.8084639Z Artifact download has finished successfully 2024-11-01T16:51:21.8324321Z ##[group]Run unzip -o artifacts.zip 2024-11-01T16:51:21.8324756Z unzip -o artifacts.zip 2024-11-01T16:51:21.8331922Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:21.8332449Z env: 2024-11-01T16:51:21.8332736Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:21.8333216Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:21.8333729Z ##[endgroup] 2024-11-01T16:51:21.8379193Z Archive: artifacts.zip 2024-11-01T16:51:21.8380393Z creating: dist/ 2024-11-01T16:51:23.8522600Z inflating: dist/torch-2.6.0a0+gitd1aa4ef-cp310-cp310-linux_x86_64.whl 2024-11-01T16:51:23.8523436Z creating: build/custom_test_artifacts/ 2024-11-01T16:51:23.8524143Z creating: build/custom_test_artifacts/custom-op-build/ 2024-11-01T16:51:23.8527381Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-11-01T16:51:23.8528414Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-11-01T16:51:23.8529592Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-11-01T16:51:23.8530697Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-11-01T16:51:23.8531810Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-11-01T16:51:23.8532970Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-11-01T16:51:23.8534105Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-11-01T16:51:23.8535417Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-11-01T16:51:23.8537330Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-11-01T16:51:23.8538572Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-11-01T16:51:23.8539920Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-11-01T16:51:23.8541355Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-11-01T16:51:23.8542777Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-11-01T16:51:23.8544201Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-11-01T16:51:23.8545610Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-11-01T16:51:23.8547045Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-11-01T16:51:23.8548619Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-11-01T16:51:23.8549850Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-11-01T16:51:23.8551050Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-11-01T16:51:23.8594670Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-11-01T16:51:23.8596417Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-11-01T16:51:23.8598163Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-11-01T16:51:23.8599971Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-11-01T16:51:23.8601956Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-11-01T16:51:23.8603649Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-11-01T16:51:23.8605342Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-11-01T16:51:23.8607043Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-11-01T16:51:23.8608729Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-11-01T16:51:23.8641135Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-11-01T16:51:23.8682676Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-11-01T16:51:23.8684404Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-11-01T16:51:23.8685967Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-11-01T16:51:23.8687464Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-11-01T16:51:23.8688935Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-11-01T16:51:23.8690415Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-11-01T16:51:23.8691844Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-11-01T16:51:23.8693303Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-11-01T16:51:23.8752645Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-11-01T16:51:23.8816339Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-11-01T16:51:23.8818231Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-11-01T16:51:23.8819471Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-11-01T16:51:23.8820536Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-11-01T16:51:23.8821638Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-11-01T16:51:23.8822768Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-11-01T16:51:23.8824038Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-11-01T16:51:23.8825450Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-11-01T16:51:23.8826798Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-11-01T16:51:23.8828072Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-11-01T16:51:23.8829503Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-11-01T16:51:23.8830791Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-11-01T16:51:23.8832074Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-11-01T16:51:23.8833354Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-11-01T16:51:23.8834604Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-11-01T16:51:23.8843801Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-11-01T16:51:23.8990922Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-11-01T16:51:23.8992327Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-11-01T16:51:23.8993642Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-11-01T16:51:23.8995098Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-11-01T16:51:23.8996486Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-11-01T16:51:23.8997795Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-11-01T16:51:23.8999131Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-11-01T16:51:23.9000487Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-11-01T16:51:23.9002072Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-11-01T16:51:23.9003445Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-11-01T16:51:23.9004774Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-11-01T16:51:23.9016385Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-11-01T16:51:23.9100605Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-11-01T16:51:23.9102305Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-11-01T16:51:23.9103631Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-11-01T16:51:23.9105018Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-11-01T16:51:23.9106149Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-11-01T16:51:23.9107335Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-11-01T16:51:23.9108540Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-11-01T16:51:23.9109542Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-11-01T16:51:23.9110450Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-11-01T16:51:23.9111381Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-11-01T16:51:23.9234005Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-11-01T16:51:23.9297631Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-11-01T16:51:23.9298550Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-11-01T16:51:23.9299396Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-11-01T16:51:23.9300382Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-11-01T16:51:23.9304995Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-11-01T16:51:23.9306116Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-11-01T16:51:23.9307227Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-11-01T16:51:23.9308527Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-11-01T16:51:23.9309650Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-11-01T16:51:23.9310939Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-11-01T16:51:23.9312249Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-11-01T16:51:23.9313414Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-11-01T16:51:23.9314576Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-11-01T16:51:23.9315938Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-11-01T16:51:23.9317299Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-11-01T16:51:23.9318650Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-11-01T16:51:23.9319996Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-11-01T16:51:23.9321363Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-11-01T16:51:23.9322733Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-11-01T16:51:23.9323951Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-11-01T16:51:23.9325120Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-11-01T16:51:23.9369226Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-11-01T16:51:23.9370966Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-11-01T16:51:23.9372702Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-11-01T16:51:23.9374479Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-11-01T16:51:23.9376507Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-11-01T16:51:23.9378444Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-11-01T16:51:23.9380262Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-11-01T16:51:23.9381996Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-11-01T16:51:23.9383694Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-11-01T16:51:23.9416083Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-11-01T16:51:23.9457000Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-11-01T16:51:23.9458779Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-11-01T16:51:23.9460379Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-11-01T16:51:23.9461912Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-11-01T16:51:23.9463420Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-11-01T16:51:23.9464940Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-11-01T16:51:23.9466409Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-11-01T16:51:23.9467900Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-11-01T16:51:23.9527360Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-11-01T16:51:23.9591330Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-11-01T16:51:23.9592782Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-11-01T16:51:23.9593977Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-11-01T16:51:23.9594997Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-11-01T16:51:23.9596047Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-11-01T16:51:23.9597157Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-11-01T16:51:23.9598399Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-11-01T16:51:23.9599810Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-11-01T16:51:23.9601400Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-11-01T16:51:23.9602670Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-11-01T16:51:23.9604005Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-11-01T16:51:23.9605330Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-11-01T16:51:23.9606658Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-11-01T16:51:23.9607969Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-11-01T16:51:23.9609267Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-11-01T16:51:23.9618577Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-11-01T16:51:23.9685552Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-11-01T16:51:23.9687021Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-11-01T16:51:23.9688392Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-11-01T16:51:23.9689541Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-11-01T16:51:23.9690601Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-11-01T16:51:23.9691644Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-11-01T16:51:23.9692695Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-11-01T16:51:23.9693670Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-11-01T16:51:23.9694558Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-11-01T16:51:23.9695492Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-11-01T16:51:23.9743490Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-11-01T16:51:23.9744449Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-11-01T16:51:23.9745352Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-11-01T16:51:23.9746433Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-11-01T16:51:23.9751461Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-11-01T16:51:23.9752643Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-11-01T16:51:23.9753833Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-11-01T16:51:23.9755101Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-11-01T16:51:23.9756326Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-11-01T16:51:23.9757705Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-11-01T16:51:23.9759116Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-11-01T16:51:23.9760385Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-11-01T16:51:23.9761638Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-11-01T16:51:23.9763083Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-11-01T16:51:23.9764537Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-11-01T16:51:23.9765969Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-11-01T16:51:23.9767407Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-11-01T16:51:23.9768856Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-11-01T16:51:23.9770337Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-11-01T16:51:23.9771636Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-11-01T16:51:23.9772891Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-11-01T16:51:23.9817927Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-11-01T16:51:23.9819828Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-11-01T16:51:23.9821908Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-11-01T16:51:23.9823942Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-11-01T16:51:23.9825840Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-11-01T16:51:23.9827651Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-11-01T16:51:23.9829551Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-11-01T16:51:23.9831326Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-11-01T16:51:23.9833085Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-11-01T16:51:23.9862280Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-11-01T16:51:23.9902517Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-11-01T16:51:23.9904368Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-11-01T16:51:23.9906057Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-11-01T16:51:23.9907678Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-11-01T16:51:23.9909346Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-11-01T16:51:23.9910912Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-11-01T16:51:23.9912422Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-11-01T16:51:23.9913980Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-11-01T16:51:23.9973605Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-11-01T16:51:24.0039031Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-11-01T16:51:24.0040574Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-11-01T16:51:24.0041839Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-11-01T16:51:24.0042918Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-11-01T16:51:24.0044070Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-11-01T16:51:24.0045264Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-11-01T16:51:24.0046635Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-11-01T16:51:24.0048148Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-11-01T16:51:24.0049615Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-11-01T16:51:24.0050989Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-11-01T16:51:24.0052382Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-11-01T16:51:24.0053986Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-11-01T16:51:24.0055434Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-11-01T16:51:24.0057246Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-11-01T16:51:24.0058707Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-11-01T16:51:24.0060254Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-11-01T16:51:24.0174190Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-11-01T16:51:24.0175622Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-11-01T16:51:24.0177349Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-11-01T16:51:24.0179012Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-11-01T16:51:24.0180590Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-11-01T16:51:24.0182078Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-11-01T16:51:24.0183623Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-11-01T16:51:24.0185176Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-11-01T16:51:24.0186730Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-11-01T16:51:24.0188377Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-11-01T16:51:24.0189933Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-11-01T16:51:24.0199453Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-11-01T16:51:24.0258181Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-11-01T16:51:24.0259842Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-11-01T16:51:24.0261274Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-11-01T16:51:24.0262545Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-11-01T16:51:24.0263724Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-11-01T16:51:24.0264905Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-11-01T16:51:24.0266087Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-11-01T16:51:24.0267200Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-11-01T16:51:24.0268314Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-11-01T16:51:24.0269320Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-11-01T16:51:24.0369903Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-11-01T16:51:24.0413635Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-11-01T16:51:24.0414395Z creating: build/lib/ 2024-11-01T16:51:24.0423944Z inflating: build/lib/libpthreadpool.a 2024-11-01T16:51:24.0424833Z inflating: build/lib/libclog.a 2024-11-01T16:51:24.0433423Z inflating: build/lib/libcpuinfo_internals.a 2024-11-01T16:51:24.0441925Z inflating: build/lib/libcpuinfo.a 2024-11-01T16:51:24.0444332Z inflating: build/lib/libnnpack_reference_layers.a 2024-11-01T16:51:24.0537453Z inflating: build/lib/libprotobuf-lite.a 2024-11-01T16:51:24.0604098Z inflating: build/lib/libgtest.a 2024-11-01T16:51:24.0684069Z inflating: build/lib/libbenchmark.a 2024-11-01T16:51:24.0690401Z inflating: build/lib/libittnotify.a 2024-11-01T16:51:24.1161519Z inflating: build/lib/libprotobuf.a 2024-11-01T16:51:24.1191260Z inflating: build/lib/libtensorpipe_uv.a 2024-11-01T16:51:24.1257569Z inflating: build/lib/libasmjit.a 2024-11-01T16:51:24.1386897Z inflating: build/lib/libgloo.a 2024-11-01T16:51:24.1408670Z inflating: build/lib/libfmt.a 2024-11-01T16:51:24.1410206Z inflating: build/lib/libcaffe2_nvrtc.so 2024-11-01T16:51:24.1515206Z inflating: build/lib/libc10.so 2024-11-01T16:51:24.1515941Z inflating: build/lib/libtorch_global_deps.so 2024-11-01T16:51:24.1537321Z inflating: build/lib/libpytorch_qnnpack.a 2024-11-01T16:51:24.1556536Z inflating: build/lib/libgmock.a 2024-11-01T16:51:24.2074225Z inflating: build/lib/libprotoc.a 2024-11-01T16:51:24.2074905Z inflating: build/lib/libgtest_main.a 2024-11-01T16:51:24.2075441Z inflating: build/lib/libbenchmark_main.a 2024-11-01T16:51:24.2094412Z inflating: build/lib/libnnpack.a 2024-11-01T16:51:25.2439026Z inflating: build/lib/libdnnl.a 2024-11-01T16:51:25.3019967Z inflating: build/lib/libtensorpipe.a 2024-11-01T16:51:25.3082202Z inflating: build/lib/libc10_cuda.so 2024-11-01T16:51:25.4341987Z inflating: build/lib/libfbgemm.a 2024-11-01T16:51:25.4342540Z inflating: build/lib/libgmock_main.a 2024-11-01T16:51:25.4867155Z inflating: build/lib/libkineto.a 2024-11-01T16:51:25.5131063Z inflating: build/lib/libtensorpipe_cuda.a 2024-11-01T16:51:25.5332846Z inflating: build/lib/libXNNPACK.a 2024-11-01T16:51:25.5720248Z inflating: build/lib/libgloo_cuda.a 2024-11-01T16:51:25.5764401Z inflating: build/lib/libonnx_proto.a 2024-11-01T16:51:25.6506058Z inflating: build/lib/libonnx.a 2024-11-01T16:51:28.1897998Z inflating: build/lib/libtorch_cpu.so 2024-11-01T16:51:28.1901214Z inflating: build/lib/libshm.so 2024-11-01T16:51:28.1906510Z inflating: build/lib/libunbox_lib.a 2024-11-01T16:51:29.8976818Z inflating: build/lib/libtorch_cuda.so 2024-11-01T16:51:29.8977428Z inflating: build/lib/libtorch.so 2024-11-01T16:51:29.8979286Z inflating: build/lib/libc10d_cuda_test.so 2024-11-01T16:51:30.8559326Z inflating: build/lib/libtorch_cuda_linalg.so 2024-11-01T16:51:30.8579839Z inflating: build/lib/libjitbackend_test.so 2024-11-01T16:51:30.8653136Z inflating: build/lib/libtorchbind_test.so 2024-11-01T16:51:30.8678510Z inflating: build/lib/libaoti_custom_ops.so 2024-11-01T16:51:30.8704504Z inflating: build/lib/libbackend_with_compiler.so 2024-11-01T16:51:31.0780885Z inflating: build/lib/libtorch_python.so 2024-11-01T16:51:31.0818049Z inflating: build/lib/libnnapi_backend.so 2024-11-01T16:51:31.0818617Z creating: build/bin/ 2024-11-01T16:51:31.0872806Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-11-01T16:51:31.0926760Z inflating: build/bin/c10_DeviceGuard_test 2024-11-01T16:51:31.0980089Z inflating: build/bin/c10_Device_test 2024-11-01T16:51:31.1043441Z inflating: build/bin/c10_DispatchKeySet_test 2024-11-01T16:51:31.1097934Z inflating: build/bin/c10_Scalar_test 2024-11-01T16:51:31.1150929Z inflating: build/bin/c10_StreamGuard_test 2024-11-01T16:51:31.1203145Z inflating: build/bin/c10_SymInt_test 2024-11-01T16:51:31.1260215Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-11-01T16:51:31.1319631Z inflating: build/bin/c10_InlineStreamGuard_test 2024-11-01T16:51:31.1377479Z inflating: build/bin/c10_SizesAndStrides_test 2024-11-01T16:51:31.1452273Z inflating: build/bin/c10_cow_test 2024-11-01T16:51:31.1506603Z inflating: build/bin/c10_Bitset_test 2024-11-01T16:51:31.1558758Z inflating: build/bin/c10_ConstexprCrc_test 2024-11-01T16:51:31.1610403Z inflating: build/bin/c10_DeadlockDetection_test 2024-11-01T16:51:31.1670635Z inflating: build/bin/c10_LeftRight_test 2024-11-01T16:51:31.1723487Z inflating: build/bin/c10_Half_test 2024-11-01T16:51:31.1780109Z inflating: build/bin/c10_Metaprogramming_test 2024-11-01T16:51:31.1838874Z inflating: build/bin/c10_NetworkFlow_test 2024-11-01T16:51:31.1890827Z inflating: build/bin/c10_Synchronized_test 2024-11-01T16:51:31.1950716Z inflating: build/bin/c10_ThreadLocal_test 2024-11-01T16:51:31.2003161Z inflating: build/bin/c10_TypeIndex_test 2024-11-01T16:51:31.2057465Z inflating: build/bin/c10_TypeList_test 2024-11-01T16:51:31.2107445Z inflating: build/bin/c10_TypeTraits_test 2024-11-01T16:51:31.2162484Z inflating: build/bin/c10_accumulate_test 2024-11-01T16:51:31.2220384Z inflating: build/bin/c10_bfloat16_test 2024-11-01T16:51:31.2274103Z inflating: build/bin/c10_bit_cast_test 2024-11-01T16:51:31.2332707Z inflating: build/bin/c10_complex_math_test 2024-11-01T16:51:31.2391102Z inflating: build/bin/c10_complex_test 2024-11-01T16:51:31.2447183Z inflating: build/bin/c10_exception_test 2024-11-01T16:51:31.2498482Z inflating: build/bin/c10_flags_test 2024-11-01T16:51:31.2552187Z inflating: build/bin/c10_generic_math_test 2024-11-01T16:51:31.2720457Z inflating: build/bin/c10_intrusive_ptr_test 2024-11-01T16:51:31.2773593Z inflating: build/bin/c10_irange_test 2024-11-01T16:51:31.2831750Z inflating: build/bin/c10_lazy_test 2024-11-01T16:51:31.2891324Z inflating: build/bin/c10_logging_test 2024-11-01T16:51:31.2969450Z inflating: build/bin/c10_optional_test 2024-11-01T16:51:31.3036329Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-11-01T16:51:31.3091813Z inflating: build/bin/c10_registry_test 2024-11-01T16:51:31.3248100Z inflating: build/bin/c10_small_vector_test 2024-11-01T16:51:31.3300778Z inflating: build/bin/c10_ssize_test 2024-11-01T16:51:31.3356599Z inflating: build/bin/c10_string_util_test 2024-11-01T16:51:31.3418727Z inflating: build/bin/c10_string_view_test 2024-11-01T16:51:31.3472414Z inflating: build/bin/c10_tempfile_test 2024-11-01T16:51:31.3530909Z inflating: build/bin/c10_typeid_test 2024-11-01T16:51:31.3580543Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-11-01T16:51:31.4047003Z inflating: build/bin/protoc-3.13.0.0 2024-11-01T16:51:31.4510986Z inflating: build/bin/protoc 2024-11-01T16:51:31.4565650Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-11-01T16:51:31.4621277Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-11-01T16:51:31.4678102Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-11-01T16:51:31.4733436Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-11-01T16:51:31.4787003Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-11-01T16:51:31.4840893Z inflating: build/bin/c10_cuda_CUDATest 2024-11-01T16:51:31.4896680Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-11-01T16:51:31.4952171Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-11-01T16:51:31.5340791Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-11-01T16:51:31.5742869Z inflating: build/bin/vec_test_all_types_AVX512 2024-11-01T16:51:31.6158809Z inflating: build/bin/vec_test_all_types_AVX2 2024-11-01T16:51:31.6217436Z inflating: build/bin/TCPStoreTest 2024-11-01T16:51:31.6272986Z inflating: build/bin/BackoffTest 2024-11-01T16:51:31.6328644Z inflating: build/bin/HashStoreTest 2024-11-01T16:51:31.6384266Z inflating: build/bin/FileStoreTest 2024-11-01T16:51:31.6399889Z inflating: build/bin/ProcessGroupMPITest 2024-11-01T16:51:31.6403408Z inflating: build/bin/torch_shm_manager 2024-11-01T16:51:31.6460275Z inflating: build/bin/test_edge_op_registration 2024-11-01T16:51:31.6463093Z inflating: build/bin/example_allreduce 2024-11-01T16:51:31.6518957Z inflating: build/bin/Dimname_test 2024-11-01T16:51:31.6595766Z inflating: build/bin/Dict_test 2024-11-01T16:51:31.6665137Z inflating: build/bin/MaybeOwned_test 2024-11-01T16:51:31.6727772Z inflating: build/bin/apply_utils_test 2024-11-01T16:51:31.6786896Z inflating: build/bin/NamedTensor_test 2024-11-01T16:51:31.6850657Z inflating: build/bin/atest 2024-11-01T16:51:31.6917646Z inflating: build/bin/basic 2024-11-01T16:51:31.6974368Z inflating: build/bin/broadcast_test 2024-11-01T16:51:31.7030780Z inflating: build/bin/cpu_allocator_test 2024-11-01T16:51:31.7089987Z inflating: build/bin/cpu_generator_test 2024-11-01T16:51:31.7145573Z inflating: build/bin/cpu_profiling_allocator_test 2024-11-01T16:51:31.7244052Z inflating: build/bin/cpu_rng_test 2024-11-01T16:51:31.7296065Z inflating: build/bin/dispatch_key_set_test 2024-11-01T16:51:31.7349778Z inflating: build/bin/dlconvertor_test 2024-11-01T16:51:31.7407430Z inflating: build/bin/half_test 2024-11-01T16:51:31.7467820Z inflating: build/bin/extension_backend_test 2024-11-01T16:51:31.7521138Z inflating: build/bin/lazy_tensor_test 2024-11-01T16:51:31.7621367Z inflating: build/bin/ivalue_test 2024-11-01T16:51:31.7679314Z inflating: build/bin/math_kernel_test 2024-11-01T16:51:31.7736087Z inflating: build/bin/memory_format_test 2024-11-01T16:51:31.7791768Z inflating: build/bin/memory_overlapping_test 2024-11-01T16:51:31.7848644Z inflating: build/bin/mobile_memory_cleanup 2024-11-01T16:51:31.7900067Z inflating: build/bin/operator_name_test 2024-11-01T16:51:31.7961229Z inflating: build/bin/native_test 2024-11-01T16:51:31.8016170Z inflating: build/bin/operators_test 2024-11-01T16:51:31.8071655Z inflating: build/bin/packedtensoraccessor_test 2024-11-01T16:51:31.8141244Z inflating: build/bin/pow_test 2024-11-01T16:51:31.8195345Z inflating: build/bin/reduce_ops_test 2024-11-01T16:51:31.8255181Z inflating: build/bin/quantized_test 2024-11-01T16:51:31.8308021Z inflating: build/bin/reportMemoryUsage_test 2024-11-01T16:51:31.8367588Z inflating: build/bin/scalar_tensor_test 2024-11-01T16:51:31.8422841Z inflating: build/bin/StorageUtils_test 2024-11-01T16:51:31.8484601Z inflating: build/bin/scalar_test 2024-11-01T16:51:31.8539124Z inflating: build/bin/stride_properties_test 2024-11-01T16:51:31.8621654Z inflating: build/bin/tensor_iterator_test 2024-11-01T16:51:31.8680181Z inflating: build/bin/test_parallel 2024-11-01T16:51:31.8682022Z inflating: build/bin/thread_init_test 2024-11-01T16:51:31.8742473Z inflating: build/bin/type_ptr_test 2024-11-01T16:51:31.8804670Z inflating: build/bin/type_test 2024-11-01T16:51:31.8860224Z inflating: build/bin/undefined_tensor_test 2024-11-01T16:51:31.8861380Z inflating: build/bin/verify_api_visibility 2024-11-01T16:51:31.8917456Z inflating: build/bin/weakref_test 2024-11-01T16:51:31.8990507Z inflating: build/bin/legacy_vmap_test 2024-11-01T16:51:31.9044967Z inflating: build/bin/wrapdim_test 2024-11-01T16:51:31.9106952Z inflating: build/bin/IListRef_test 2024-11-01T16:51:31.9217589Z inflating: build/bin/List_test 2024-11-01T16:51:31.9272496Z inflating: build/bin/xla_tensor_test 2024-11-01T16:51:31.9398478Z inflating: build/bin/kernel_function_legacy_test 2024-11-01T16:51:31.9497590Z inflating: build/bin/kernel_function_test 2024-11-01T16:51:31.9567111Z inflating: build/bin/KernelFunction_test 2024-11-01T16:51:31.9701284Z inflating: build/bin/kernel_lambda_legacy_test 2024-11-01T16:51:31.9765367Z inflating: build/bin/kernel_stackbased_test 2024-11-01T16:51:31.9875139Z inflating: build/bin/kernel_lambda_test 2024-11-01T16:51:31.9977056Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-11-01T16:51:32.0032167Z inflating: build/bin/CppSignature_test 2024-11-01T16:51:32.0090226Z inflating: build/bin/backend_fallback_test 2024-11-01T16:51:32.0140969Z inflating: build/bin/op_allowlist_test 2024-11-01T16:51:32.0207300Z inflating: build/bin/inline_container_test 2024-11-01T16:51:32.0516236Z inflating: build/bin/op_registration_test 2024-11-01T16:51:32.0571886Z inflating: build/bin/cuda_apply_test 2024-11-01T16:51:32.0626921Z inflating: build/bin/cuda_allocator_test 2024-11-01T16:51:32.0690842Z inflating: build/bin/cuda_atomic_ops_test 2024-11-01T16:51:32.0747827Z inflating: build/bin/cuda_caching_host_allocator_test 2024-11-01T16:51:32.0821444Z inflating: build/bin/cuda_complex_math_test 2024-11-01T16:51:32.0875989Z inflating: build/bin/cuda_device_test 2024-11-01T16:51:32.0937584Z inflating: build/bin/cuda_complex_test 2024-11-01T16:51:32.0998859Z inflating: build/bin/cuda_cub_test 2024-11-01T16:51:32.1052369Z inflating: build/bin/cuda_dlconvertor_test 2024-11-01T16:51:32.1104935Z inflating: build/bin/cuda_integer_divider_test 2024-11-01T16:51:32.1164318Z inflating: build/bin/cuda_generator_test 2024-11-01T16:51:32.1234736Z inflating: build/bin/cuda_distributions_test 2024-11-01T16:51:32.1286439Z inflating: build/bin/cuda_half_test 2024-11-01T16:51:32.1337782Z inflating: build/bin/cuda_optional_test 2024-11-01T16:51:32.1390763Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-11-01T16:51:32.1446110Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-11-01T16:51:32.1510051Z inflating: build/bin/cuda_stream_test 2024-11-01T16:51:32.1563526Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-11-01T16:51:32.1616670Z inflating: build/bin/cuda_cudnn_test 2024-11-01T16:51:32.1671153Z inflating: build/bin/cuda_vectorized_test 2024-11-01T16:51:32.1739244Z inflating: build/bin/ProcessGroupGlooTest 2024-11-01T16:51:32.1755633Z inflating: build/bin/tutorial_tensorexpr 2024-11-01T16:51:32.1823648Z inflating: build/bin/ProcessGroupNCCLTest 2024-11-01T16:51:32.1885164Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-11-01T16:51:32.1951345Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-11-01T16:51:32.2005059Z inflating: build/bin/ProcessGroupUCCTest 2024-11-01T16:51:32.2062746Z inflating: build/bin/test_dist_autograd 2024-11-01T16:51:32.2065056Z inflating: build/bin/parallel_benchmark 2024-11-01T16:51:32.2074413Z inflating: build/bin/aot_model_compiler_test 2024-11-01T16:51:32.2146865Z inflating: build/bin/test_cpp_rpc 2024-11-01T16:51:32.2217769Z inflating: build/bin/test_mobile_nnc 2024-11-01T16:51:32.2569667Z inflating: build/bin/test_lazy 2024-11-01T16:51:32.3429467Z inflating: build/bin/test_tensorexpr 2024-11-01T16:51:32.4633994Z inflating: build/bin/test_api 2024-11-01T16:51:32.5232087Z inflating: build/bin/test_jit 2024-11-01T16:51:32.5232696Z creating: .additional_ci_files/ 2024-11-01T16:51:32.5312399Z inflating: .additional_ci_files/test-times.json 2024-11-01T16:51:32.5637068Z inflating: .additional_ci_files/test-class-times.json 2024-11-01T16:51:32.5673322Z ##[group]Run rm artifacts.zip 2024-11-01T16:51:32.5673883Z rm artifacts.zip 2024-11-01T16:51:32.5680182Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:32.5680688Z env: 2024-11-01T16:51:32.5680960Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:32.5681415Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:32.5681909Z ##[endgroup] 2024-11-01T16:51:32.6324763Z ##[group]Run df -H 2024-11-01T16:51:32.6325072Z df -H 2024-11-01T16:51:32.6330921Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:32.6331475Z env: 2024-11-01T16:51:32.6331772Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:32.6332265Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:32.6332899Z ##[endgroup] 2024-11-01T16:51:32.6376946Z Filesystem Size Used Avail Use% Mounted on 2024-11-01T16:51:32.6377700Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-11-01T16:51:32.6378275Z tmpfs 101G 0 101G 0% /dev/shm 2024-11-01T16:51:32.6379105Z tmpfs 41G 697k 41G 1% /run 2024-11-01T16:51:32.6379619Z /dev/nvme0n1p1 161G 47G 115G 30% / 2024-11-01T16:51:32.6380282Z tmpfs 101G 8.2k 101G 1% /tmp 2024-11-01T16:51:32.6380835Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-11-01T16:51:32.6381424Z tmpfs 21G 0 21G 0% /run/user/0 2024-11-01T16:51:32.6427976Z Prepare all required actions 2024-11-01T16:51:32.6428700Z Getting action download info 2024-11-01T16:51:32.7622639Z ##[group]Run ./.github/actions/download-td-artifacts 2024-11-01T16:51:32.7623168Z with: 2024-11-01T16:51:32.7623426Z env: 2024-11-01T16:51:32.7623722Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:32.7624230Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:32.7624776Z ##[endgroup] 2024-11-01T16:51:32.7662548Z ##[group]Run seemethere/download-artifact-s3@v4 2024-11-01T16:51:32.7663050Z with: 2024-11-01T16:51:32.7663334Z name: td_results 2024-11-01T16:51:32.7663684Z s3-bucket: gha-artifacts 2024-11-01T16:51:32.7664053Z region: us-east-1 2024-11-01T16:51:32.7664381Z env: 2024-11-01T16:51:32.7664672Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:32.7665190Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:32.7665739Z ##[endgroup] 2024-11-01T16:51:33.2658618Z (node:99482) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-11-01T16:51:33.2660030Z 2024-11-01T16:51:33.2660496Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-11-01T16:51:33.2661993Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-11-01T16:51:33.2663683Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-11-01T16:51:33.3897143Z Found 1 objects with prefix pytorch/pytorch/11632514903/td_results/ 2024-11-01T16:51:33.3898332Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-11-01T16:51:33.4784768Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-11-01T16:51:33.4791246Z Artifact download has finished successfully 2024-11-01T16:51:33.5026183Z ##[group]Run mkdir -p .additional_ci_files 2024-11-01T16:51:33.5026728Z mkdir -p .additional_ci_files 2024-11-01T16:51:33.5027387Z mv td_results.json .additional_ci_files/td_results.json 2024-11-01T16:51:33.5034147Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:33.5034689Z env: 2024-11-01T16:51:33.5034973Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:33.5035456Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:33.5035963Z ##[endgroup] 2024-11-01T16:51:33.5123303Z ##[group]Run .github/scripts/parse_ref.py 2024-11-01T16:51:33.5123788Z .github/scripts/parse_ref.py 2024-11-01T16:51:33.5129524Z shell: /usr/bin/bash -e {0} 2024-11-01T16:51:33.5129892Z env: 2024-11-01T16:51:33.5130161Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:33.5130643Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:33.5131163Z ##[endgroup] 2024-11-01T16:51:33.5423253Z Prepare all required actions 2024-11-01T16:51:33.5486606Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-11-01T16:51:33.5487060Z with: 2024-11-01T16:51:33.5487646Z github-token: *** 2024-11-01T16:51:33.5487945Z env: 2024-11-01T16:51:33.5488210Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:33.5488664Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:33.5489142Z ##[endgroup] 2024-11-01T16:51:33.5509394Z ##[group]Run set -eux 2024-11-01T16:51:33.5509752Z set -eux 2024-11-01T16:51:33.5510409Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-11-01T16:51:33.5516436Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:33.5516944Z env: 2024-11-01T16:51:33.5517214Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:33.5517666Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:33.5518332Z GITHUB_TOKEN: *** 2024-11-01T16:51:33.5518636Z ##[endgroup] 2024-11-01T16:51:33.5543463Z + python3 .github/scripts/get_workflow_job_id.py 11632514903 i-06c394753ea076091 2024-11-01T16:51:35.6410525Z setting job-id=32397034850 2024-11-01T16:51:35.6411576Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-11-01T16:51:35.6649431Z Prepare all required actions 2024-11-01T16:51:35.6649885Z Getting action download info 2024-11-01T16:51:35.7886010Z ##[group]Run ./.github/actions/filter-test-configs 2024-11-01T16:51:35.7886502Z with: 2024-11-01T16:51:35.7886985Z github-token: *** 2024-11-01T16:51:35.7888737Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}]} 2024-11-01T16:51:35.7890982Z job-name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-11-01T16:51:35.7891807Z env: 2024-11-01T16:51:35.7892099Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:35.7892568Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:35.7893082Z ##[endgroup] 2024-11-01T16:51:35.7955535Z ##[group]Run nick-fields/retry@v3.0.0 2024-11-01T16:51:35.7955942Z with: 2024-11-01T16:51:35.7956205Z shell: bash 2024-11-01T16:51:35.7956498Z timeout_minutes: 10 2024-11-01T16:51:35.7956821Z max_attempts: 5 2024-11-01T16:51:35.7957119Z retry_wait_seconds: 30 2024-11-01T16:51:35.7958307Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-11-01T16:51:35.7959562Z polling_interval_seconds: 1 2024-11-01T16:51:35.7959935Z warning_on_retry: true 2024-11-01T16:51:35.7960278Z continue_on_error: false 2024-11-01T16:51:35.7960609Z env: 2024-11-01T16:51:35.7960855Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:35.7961306Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:35.7961989Z GITHUB_TOKEN: *** 2024-11-01T16:51:35.7962294Z ##[endgroup] 2024-11-01T16:51:35.8944986Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-11-01T16:51:36.1489501Z Defaulting to user installation because normal site-packages is not writeable 2024-11-01T16:51:36.1658524Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.9/site-packages (2.27.1) 2024-11-01T16:51:36.1661938Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.9/site-packages (6.0.1) 2024-11-01T16:51:36.1788036Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-11-01T16:51:36.1796969Z Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2.0.12) 2024-11-01T16:51:36.1801268Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2024.8.30) 2024-11-01T16:51:36.1807824Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-11-01T16:51:36.8767256Z Command completed after 1 attempt(s). 2024-11-01T16:51:36.8826661Z ##[group]Run set -x 2024-11-01T16:51:36.8827170Z set -x 2024-11-01T16:51:36.8827482Z  2024-11-01T16:51:36.8828089Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-11-01T16:51:36.8828852Z # in runner workspace 2024-11-01T16:51:36.8829437Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-11-01T16:51:36.8836084Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:36.8836580Z env: 2024-11-01T16:51:36.8836849Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:36.8837301Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:36.8837788Z ##[endgroup] 2024-11-01T16:51:36.8864353Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-11-01T16:51:36.9121457Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-11-01T16:51:36.9122168Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-11-01T16:51:36.9122650Z echo "Job name: ${JOB_NAME}" 2024-11-01T16:51:36.9123056Z  2024-11-01T16:51:36.9123594Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-11-01T16:51:36.9124295Z # in runner workspace 2024-11-01T16:51:36.9124884Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-11-01T16:51:36.9125545Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-11-01T16:51:36.9126003Z  --job-name "${JOB_NAME}" \ 2024-11-01T16:51:36.9127794Z  --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}]}" \ 2024-11-01T16:51:36.9129601Z  --selected-test-configs "" \ 2024-11-01T16:51:36.9130058Z  --pr-number "${PR_NUMBER}" \ 2024-11-01T16:51:36.9130464Z  --tag "${TAG}" \ 2024-11-01T16:51:36.9131033Z  --event-name "${EVENT_NAME}" \ 2024-11-01T16:51:36.9131507Z  --schedule "${SCHEDULE}" \ 2024-11-01T16:51:36.9131956Z  --branch "${HEAD_BRANCH}" 2024-11-01T16:51:36.9138323Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:36.9138894Z env: 2024-11-01T16:51:36.9139183Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:36.9139692Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:36.9140461Z GITHUB_TOKEN: *** 2024-11-01T16:51:36.9141255Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-11-01T16:51:36.9142254Z PR_NUMBER: 138766 2024-11-01T16:51:36.9142662Z TAG: 2024-11-01T16:51:36.9142916Z EVENT_NAME: pull_request 2024-11-01T16:51:36.9143259Z SCHEDULE: 2024-11-01T16:51:36.9143549Z HEAD_BRANCH: 2024-11-01T16:51:36.9143837Z ##[endgroup] 2024-11-01T16:51:36.9169262Z Workflow: pull 2024-11-01T16:51:36.9170710Z Job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-11-01T16:51:37.1565170Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-11-01T16:51:37.3031247Z ##[group]Run echo "Filtered matrix:" 2024-11-01T16:51:37.3031717Z echo "Filtered matrix:" 2024-11-01T16:51:37.3033742Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}]}" 2024-11-01T16:51:37.3035611Z  2024-11-01T16:51:37.3035894Z echo 2024-11-01T16:51:37.3036296Z echo "Is the current job unstable? False" 2024-11-01T16:51:37.3036799Z  2024-11-01T16:51:37.3037079Z echo 2024-11-01T16:51:37.3037460Z echo "Is keep-going label set? False" 2024-11-01T16:51:37.3037939Z  2024-11-01T16:51:37.3038215Z echo 2024-11-01T16:51:37.3038531Z echo "Renabled issues? " 2024-11-01T16:51:37.3044912Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:37.3045463Z env: 2024-11-01T16:51:37.3045754Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:37.3046248Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:37.3046776Z ##[endgroup] 2024-11-01T16:51:37.3083897Z Filtered matrix: 2024-11-01T16:51:37.3085651Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: linux.g4dn.12xlarge.nvidia.gpu}, {config: distributed, shard: 2, num_shards: 3, runner: linux.g4dn.12xlarge.nvidia.gpu}, {config: distributed, shard: 3, num_shards: 3, runner: linux.g4dn.12xlarge.nvidia.gpu}]} 2024-11-01T16:51:37.3087553Z 2024-11-01T16:51:37.3087874Z Is the current job unstable? False 2024-11-01T16:51:37.3088196Z 2024-11-01T16:51:37.3088537Z Is keep-going label set? False 2024-11-01T16:51:37.3088819Z 2024-11-01T16:51:37.3088953Z Renabled issues? 2024-11-01T16:51:37.3136385Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-11-01T16:51:37.3137222Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-11-01T16:51:37.3143493Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:37.3144064Z env: 2024-11-01T16:51:37.3144360Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:37.3144869Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:37.3145412Z JOB_TIMEOUT: 360 2024-11-01T16:51:37.3145746Z ##[endgroup] 2024-11-01T16:51:37.3254805Z ##[group]Run # Fetch aws credential from IMDs 2024-11-01T16:51:37.3255407Z # Fetch aws credential from IMDs 2024-11-01T16:51:37.3256286Z eval "$(python3 .github/scripts/get_aws_session_tokens.py)" 2024-11-01T16:51:37.3256908Z set -x 2024-11-01T16:51:37.3257211Z  2024-11-01T16:51:37.3257595Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-11-01T16:51:37.3258212Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-11-01T16:51:37.3258851Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-11-01T16:51:37.3259426Z  TEST_COMMAND=.ci/onnx/test.sh 2024-11-01T16:51:37.3259885Z else 2024-11-01T16:51:37.3260245Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-11-01T16:51:37.3260718Z fi 2024-11-01T16:51:37.3261009Z  2024-11-01T16:51:37.3261533Z # detached container should get cleaned up by teardown_ec2_linux 2024-11-01T16:51:37.3262458Z # TODO: Stop building test binaries as part of the build phase 2024-11-01T16:51:37.3263247Z # Used for GPU_FLAG since that doesn't play nice 2024-11-01T16:51:37.3263867Z # shellcheck disable=SC2086,SC2090 2024-11-01T16:51:37.3264370Z container_name=$(docker run \ 2024-11-01T16:51:37.3264834Z  ${GPU_FLAG:-} \ 2024-11-01T16:51:37.3265297Z  ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \ 2024-11-01T16:51:37.3265834Z  -e BUILD_ENVIRONMENT \ 2024-11-01T16:51:37.3266280Z  -e PR_NUMBER \ 2024-11-01T16:51:37.3266663Z  -e GITHUB_ACTIONS \ 2024-11-01T16:51:37.3267098Z  -e GITHUB_REPOSITORY \ 2024-11-01T16:51:37.3267545Z  -e GITHUB_WORKFLOW \ 2024-11-01T16:51:37.3267973Z  -e GITHUB_JOB \ 2024-11-01T16:51:37.3268464Z  -e GITHUB_RUN_ID \ 2024-11-01T16:51:37.3268945Z  -e GITHUB_RUN_NUMBER \ 2024-11-01T16:51:37.3269334Z  -e GITHUB_RUN_ATTEMPT \ 2024-11-01T16:51:37.3269733Z  -e JOB_ID \ 2024-11-01T16:51:37.3270061Z  -e JOB_NAME \ 2024-11-01T16:51:37.3270408Z  -e BASE_SHA \ 2024-11-01T16:51:37.3270738Z  -e BRANCH \ 2024-11-01T16:51:37.3271052Z  -e SHA1 \ 2024-11-01T16:51:37.3271380Z  -e AWS_DEFAULT_REGION \ 2024-11-01T16:51:37.3271784Z  -e AWS_ACCESS_KEY_ID \ 2024-11-01T16:51:37.3272186Z  -e AWS_SECRET_ACCESS_KEY \ 2024-11-01T16:51:37.3272603Z  -e AWS_SESSION_TOKEN \ 2024-11-01T16:51:37.3272980Z  -e IN_WHEEL_TEST \ 2024-11-01T16:51:37.3273354Z  -e SHARD_NUMBER \ 2024-11-01T16:51:37.3273723Z  -e TEST_CONFIG \ 2024-11-01T16:51:37.3274085Z  -e NUM_TEST_SHARDS \ 2024-11-01T16:51:37.3274468Z  -e REENABLED_ISSUES \ 2024-11-01T16:51:37.3274876Z  -e CONTINUE_THROUGH_ERROR \ 2024-11-01T16:51:37.3275289Z  -e VERBOSE_TEST_LOGS \ 2024-11-01T16:51:37.3275683Z  -e TEST_SHOWLOCALS \ 2024-11-01T16:51:37.3276076Z  -e NO_TEST_TIMEOUT \ 2024-11-01T16:51:37.3276443Z  -e NO_TD \ 2024-11-01T16:51:37.3276927Z  -e TD_DISTRIBUTED \ 2024-11-01T16:51:37.3277288Z  -e PR_LABELS \ 2024-11-01T16:51:37.3277688Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-11-01T16:51:37.3278148Z  -e SCCACHE_BUCKET \ 2024-11-01T16:51:37.3278529Z  -e SCCACHE_REGION \ 2024-11-01T16:51:37.3278920Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-11-01T16:51:37.3279325Z  -e XLA_CUDA \ 2024-11-01T16:51:37.3279706Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-11-01T16:51:37.3280221Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-11-01T16:51:37.3280748Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-11-01T16:51:37.3281268Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-11-01T16:51:37.3281741Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-11-01T16:51:37.3282194Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-11-01T16:51:37.3282724Z  -e DASHBOARD_TAG \ 2024-11-01T16:51:37.3283104Z  -e IS_A100_RUNNER \ 2024-11-01T16:51:37.3283499Z  -e ARTIFACTS_FILE_SUFFIX \ 2024-11-01T16:51:37.3284016Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-11-01T16:51:37.3284576Z  --security-opt seccomp=unconfined \ 2024-11-01T16:51:37.3285030Z  --cap-add=SYS_PTRACE \ 2024-11-01T16:51:37.3285419Z  --ipc=host \ 2024-11-01T16:51:37.3285766Z  --shm-size="${SHM_SIZE}" \ 2024-11-01T16:51:37.3286155Z  --tty \ 2024-11-01T16:51:37.3286459Z  --detach \ 2024-11-01T16:51:37.3286807Z  --name="${container_name}" \ 2024-11-01T16:51:37.3287275Z  --user jenkins \ 2024-11-01T16:51:37.3287762Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-11-01T16:51:37.3288334Z  -w /var/lib/jenkins/workspace \ 2024-11-01T16:51:37.3288770Z  "${DOCKER_IMAGE}" 2024-11-01T16:51:37.3289117Z ) 2024-11-01T16:51:37.3289500Z # Propagate download.pytorch.org IP to container 2024-11-01T16:51:37.3290468Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-11-01T16:51:37.3291491Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-11-01T16:51:37.3292453Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-11-01T16:51:37.3298731Z shell: /usr/bin/bash -e {0} 2024-11-01T16:51:37.3299121Z env: 2024-11-01T16:51:37.3299418Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:37.3299914Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:37.3300592Z BUILD_ENVIRONMENT: linux-focal-cuda11.8-py3.10-gcc9 2024-11-01T16:51:37.3301357Z PR_NUMBER: 138766 2024-11-01T16:51:37.3301735Z GITHUB_REPOSITORY: pytorch/pytorch 2024-11-01T16:51:37.3302196Z GITHUB_WORKFLOW: pull 2024-11-01T16:51:37.3302579Z GITHUB_JOB: test 2024-11-01T16:51:37.3302909Z GITHUB_RUN_ID: 11632514903 2024-11-01T16:51:37.3303316Z GITHUB_RUN_NUMBER: 261361 2024-11-01T16:51:37.3303706Z GITHUB_RUN_ATTEMPT: 1 2024-11-01T16:51:37.3304066Z JOB_ID: 32397034850 2024-11-01T16:51:37.3304857Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-11-01T16:51:37.3305735Z BRANCH: pull/138766 2024-11-01T16:51:37.3306165Z SHA1: d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:51:37.3306768Z BASE_SHA: 2055d3c8dc8b93c52eda6b9c07d716448d9e28c9 2024-11-01T16:51:37.3307305Z TEST_CONFIG: distributed 2024-11-01T16:51:37.3307682Z SHARD_NUMBER: 2 2024-11-01T16:51:37.3308130Z NUM_TEST_SHARDS: 3 2024-11-01T16:51:37.3308457Z REENABLED_ISSUES: 2024-11-01T16:51:37.3308813Z CONTINUE_THROUGH_ERROR: False 2024-11-01T16:51:37.3309232Z VERBOSE_TEST_LOGS: False 2024-11-01T16:51:37.3309611Z TEST_SHOWLOCALS: False 2024-11-01T16:51:37.3309977Z NO_TEST_TIMEOUT: False 2024-11-01T16:51:37.3310319Z NO_TD: False 2024-11-01T16:51:37.3310634Z TD_DISTRIBUTED: False 2024-11-01T16:51:37.3311217Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-11-01T16:51:37.3311750Z SCCACHE_REGION: us-east-1 2024-11-01T16:51:37.3312142Z SCCACHE_S3_KEY_PREFIX: pull 2024-11-01T16:51:37.3312510Z SHM_SIZE: 2g 2024-11-01T16:51:37.3313632Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:51:37.3314731Z XLA_CUDA: 2024-11-01T16:51:37.3315218Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-11-01T16:51:37.3315856Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-11-01T16:51:37.3316286Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-11-01T16:51:37.3316678Z DASHBOARD_TAG: 2024-11-01T16:51:37.3317195Z HUGGING_FACE_HUB_TOKEN: *** 2024-11-01T16:51:37.3317749Z SCRIBE_GRAPHQL_ACCESS_TOKEN: *** 2024-11-01T16:51:37.3318137Z IS_A100_RUNNER: 0 2024-11-01T16:51:37.3318874Z ARTIFACTS_FILE_SUFFIX: test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_32397034850 2024-11-01T16:51:37.3319595Z ##[endgroup] 2024-11-01T16:51:37.3527486Z Traceback (most recent call last): 2024-11-01T16:51:37.3528928Z File "/home/ec2-user/actions-runner/_work/pytorch/pytorch/.github/scripts/get_aws_session_tokens.py", line 2, in 2024-11-01T16:51:37.3529946Z import boto3 # type: ignore[import] 2024-11-01T16:51:37.3530529Z ModuleNotFoundError: No module named 'boto3' 2024-11-01T16:51:37.3556689Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-11-01T16:51:37.3557644Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *onnx* ]] 2024-11-01T16:51:37.3558281Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-11-01T16:51:37.3565756Z +++ nproc --ignore=2 2024-11-01T16:51:37.3583951Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=46 -e SCCACHE_BUCKET -e SCCACHE_REGION -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e IS_A100_RUNNER -e ARTIFACTS_FILE_SUFFIX --env-file=/tmp/github_env_11632514903 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:51:48.4824468Z + container_name=e581e04e5ab141ca02b611c02d76c89f1cb6a25e20d241d526180d97229d0e5e 2024-11-01T16:51:48.4826409Z + grep download.pytorch.org /etc/hosts 2024-11-01T16:51:48.4828782Z + docker exec -i e581e04e5ab141ca02b611c02d76c89f1cb6a25e20d241d526180d97229d0e5e sudo bash -c '/bin/cat >> /etc/hosts' 2024-11-01T16:51:48.6221506Z + echo DOCKER_CONTAINER_ID=e581e04e5ab141ca02b611c02d76c89f1cb6a25e20d241d526180d97229d0e5e 2024-11-01T16:51:48.6225200Z ++ echo dist/torch-2.6.0a0+gitd1aa4ef-cp310-cp310-linux_x86_64.whl 2024-11-01T16:51:48.6227650Z + docker exec -t e581e04e5ab141ca02b611c02d76c89f1cb6a25e20d241d526180d97229d0e5e sh -c 'pip install dist/torch-2.6.0a0+gitd1aa4ef-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-11-01T16:51:49.0185784Z Processing ./dist/torch-2.6.0a0+gitd1aa4ef-cp310-cp310-linux_x86_64.whl (from torch==2.6.0a0+gitd1aa4ef) 2024-11-01T16:51:49.8982533Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (3.16.1) 2024-11-01T16:51:49.8985391Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (4.12.2) 2024-11-01T16:51:49.8988201Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (2.8.8) 2024-11-01T16:51:49.8991712Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (3.1.4) 2024-11-01T16:51:49.8995080Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (2024.10.0) 2024-11-01T16:51:49.9001550Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (1.13.1) 2024-11-01T16:51:49.9020808Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (1.3.0) 2024-11-01T16:51:49.9039933Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (3.3.0) 2024-11-01T16:51:49.9059067Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (1.22.4) 2024-11-01T16:51:49.9453657Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (3.0.2) 2024-11-01T16:51:50.2802892Z Installing collected packages: torch 2024-11-01T16:52:01.4378521Z Successfully installed torch-2.6.0a0+gitd1aa4ef 2024-11-01T16:52:01.5117080Z + export TERM=vt100 2024-11-01T16:52:01.5117543Z + TERM=vt100 2024-11-01T16:52:01.5120137Z ++ dirname .ci/pytorch/test.sh 2024-11-01T16:52:01.5124761Z + source .ci/pytorch/common.sh 2024-11-01T16:52:01.5128793Z +++ dirname .ci/pytorch/common.sh 2024-11-01T16:52:01.5136844Z ++ source .ci/pytorch/common_utils.sh 2024-11-01T16:52:01.5138715Z +++ declare -f -t trap_add 2024-11-01T16:52:01.5144974Z ++ set -ex 2024-11-01T16:52:01.5145508Z ++ [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-11-01T16:52:01.5146048Z ++ BUILD_TEST_LIBTORCH=0 2024-11-01T16:52:01.5146589Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-11-01T16:52:01.5149341Z ++ stat -c %u /var/lib/jenkins/workspace 2024-11-01T16:52:01.5168377Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-11-01T16:52:01.5168901Z + trap_add cleanup_workspace EXIT 2024-11-01T16:52:01.5169528Z + trap_add_cmd=cleanup_workspace 2024-11-01T16:52:01.5169932Z + shift 2024-11-01T16:52:01.5170262Z + for trap_add_name in "$@" 2024-11-01T16:52:01.5173770Z +++ trap -p EXIT 2024-11-01T16:52:01.5175530Z ++ eval 'extract_trap_cmd ' 2024-11-01T16:52:01.5176036Z +++ extract_trap_cmd 2024-11-01T16:52:01.5176433Z +++ printf '%s\n' '' 2024-11-01T16:52:01.5176868Z ++ printf '%s\n' cleanup_workspace 2024-11-01T16:52:01.5177828Z + trap -- ' 2024-11-01T16:52:01.5178192Z cleanup_workspace' EXIT 2024-11-01T16:52:01.5178724Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-11-01T16:52:02.0267117Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-11-01T16:52:02.0289955Z + echo 'Environment variables:' 2024-11-01T16:52:02.0290430Z Environment variables: 2024-11-01T16:52:02.0290783Z + env 2024-11-01T16:52:02.0298480Z INSTALLED_DB=yes 2024-11-01T16:52:02.0298903Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-11-01T16:52:02.0299349Z NVIDIA_VISIBLE_DEVICES=all 2024-11-01T16:52:02.0299776Z NV_NVML_DEV_VERSION=11.8.86-1 2024-11-01T16:52:02.0300553Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-11-01T16:52:02.0301391Z CONTINUE_THROUGH_ERROR=False 2024-11-01T16:52:02.0302229Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-11-01T16:52:02.0302827Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-11-01T16:52:02.0303480Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-11-01T16:52:02.0304025Z HOSTNAME=e581e04e5ab1 2024-11-01T16:52:02.0305039Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_6b59c2d6-10ce-4869-92e7-36a56a58367a 2024-11-01T16:52:02.0305996Z GITHUB_ACTION=__self 2024-11-01T16:52:02.0306375Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-11-01T16:52:02.0309095Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-11-01T16:52:02.0311962Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-11-01T16:52:02.0312552Z NV_NVTX_VERSION=11.8.86-1 2024-11-01T16:52:02.0312937Z GITHUB_RUN_NUMBER=261361 2024-11-01T16:52:02.0313319Z TEST_CONFIG=distributed 2024-11-01T16:52:02.0313829Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-11-01T16:52:02.0314339Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-11-01T16:52:02.0314784Z IS_A100_RUNNER=0 2024-11-01T16:52:02.0315152Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-11-01T16:52:02.0315630Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-11-01T16:52:02.0316409Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2024-11-01T16:52:02.0316916Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-11-01T16:52:02.0317323Z GITHUB_TRIGGERING_ACTOR=c00w 2024-11-01T16:52:02.0317780Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-11-01T16:52:02.0318296Z GITHUB_REF_TYPE=branch 2024-11-01T16:52:02.0318660Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-11-01T16:52:02.0319064Z NCCL_VERSION=2.16.2-1 2024-11-01T16:52:02.0319492Z BASE_SHA=2055d3c8dc8b93c52eda6b9c07d716448d9e28c9 2024-11-01T16:52:02.0319974Z XLA_CUDA= 2024-11-01T16:52:02.0320442Z HUGGING_FACE_HUB_TOKEN=*** 2024-11-01T16:52:02.0322833Z *** 2024-11-01T16:52:02.0323136Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-11-01T16:52:02.0323569Z GITHUB_REPOSITORY_ID=65600975 2024-11-01T16:52:02.0323964Z GITHUB_ACTIONS=true 2024-11-01T16:52:02.0324303Z NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:52:02.0324872Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-11-01T16:52:02.0325465Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-11-01T16:52:02.0325987Z SHA1=d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:52:02.0326565Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-11-01T16:52:02.0327093Z GITHUB_SHA=368da7510f5d755034681777a4e5ae6d33c07b38 2024-11-01T16:52:02.0327882Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/138766/merge 2024-11-01T16:52:02.0328601Z UCC_HOME=/usr 2024-11-01T16:52:02.0328967Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-11-01T16:52:02.0329407Z VERBOSE_TEST_LOGS=False 2024-11-01T16:52:02.0329780Z NVIDIA_PRODUCT_NAME=CUDA 2024-11-01T16:52:02.0330271Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-11-01T16:52:02.0330788Z GITHUB_REF=refs/pull/138766/merge 2024-11-01T16:52:02.0331239Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-11-01T16:52:02.0331641Z SHARD_NUMBER=2 2024-11-01T16:52:02.0331961Z GITHUB_REF_PROTECTED=false 2024-11-01T16:52:02.0332322Z HOME=/var/lib/jenkins 2024-11-01T16:52:02.0332707Z GITHUB_API_URL=https://api.github.com 2024-11-01T16:52:02.0333185Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-11-01T16:52:02.0333782Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-11-01T16:52:02.0334310Z SCCACHE_S3_KEY_PREFIX=pull 2024-11-01T16:52:02.0334670Z CUDA_VERSION=11.8.0 2024-11-01T16:52:02.0335132Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-11-01T16:52:02.0335613Z NUM_TEST_SHARDS=3 2024-11-01T16:52:02.0336028Z UCX_HOME=/usr 2024-11-01T16:52:02.0336816Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-11-01T16:52:02.0338145Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_6b59c2d6-10ce-4869-92e7-36a56a58367a 2024-11-01T16:52:02.0339790Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-11-01T16:52:02.0341352Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_6b59c2d6-10ce-4869-92e7-36a56a58367a 2024-11-01T16:52:02.0342713Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-11-01T16:52:02.0343496Z GITHUB_EVENT_NAME=pull_request 2024-11-01T16:52:02.0343897Z DASHBOARD_TAG= 2024-11-01T16:52:02.0344221Z GITHUB_RUN_ID=11632514903 2024-11-01T16:52:02.0344730Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-11-01T16:52:02.0345349Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-11-01T16:52:02.0346651Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_6b59c2d6-10ce-4869-92e7-36a56a58367a 2024-11-01T16:52:02.0347809Z GITHUB_ACTOR=c00w 2024-11-01T16:52:02.0348191Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-11-01T16:52:02.0348586Z PR_NUMBER=138766 2024-11-01T16:52:02.0348913Z GITHUB_RUN_ATTEMPT=1 2024-11-01T16:52:02.0349272Z ANACONDA_PYTHON_VERSION=3.10 2024-11-01T16:52:02.0349754Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-11-01T16:52:02.0350257Z TERM=vt100 2024-11-01T16:52:02.0350593Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-11-01T16:52:02.0351040Z INSTALLED_VISION=yes 2024-11-01T16:52:02.0351382Z BRANCH=pull/138766 2024-11-01T16:52:02.0351764Z SCCACHE_REGION=us-east-1 2024-11-01T16:52:02.0352143Z OPENSSL_ROOT_DIR=/opt/openssl 2024-11-01T16:52:02.0352576Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-11-01T16:52:02.0353024Z CUDA_PATH=/usr/local/cuda 2024-11-01T16:52:02.0353886Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-11-01T16:52:02.0354745Z GITHUB_SERVER_URL=https://github.com 2024-11-01T16:52:02.0355286Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-11-01T16:52:02.0355811Z REENABLED_ISSUES= 2024-11-01T16:52:02.0356110Z SHLVL=1 2024-11-01T16:52:02.0356376Z MAX_JOBS=46 2024-11-01T16:52:02.0356702Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-11-01T16:52:02.0357075Z NVARCH=x86_64 2024-11-01T16:52:02.0357380Z GITHUB_ACTOR_ID=486199 2024-11-01T16:52:02.0357848Z GITHUB_WORKFLOW_SHA=368da7510f5d755034681777a4e5ae6d33c07b38 2024-11-01T16:52:02.0358411Z GITHUB_REF_NAME=138766/merge 2024-11-01T16:52:02.0358881Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-11-01T16:52:02.0359641Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-11-01T16:52:02.0360287Z GITHUB_JOB=test 2024-11-01T16:52:02.0360701Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-11-01T16:52:02.0361312Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-11-01T16:52:02.0361880Z NO_TEST_TIMEOUT=False 2024-11-01T16:52:02.0362224Z TD_DISTRIBUTED=False 2024-11-01T16:52:02.0362634Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-11-01T16:52:02.0363111Z GITHUB_REPOSITORY=pytorch/pytorch 2024-11-01T16:52:02.0363555Z NV_NVPROF_VERSION=11.8.87-1 2024-11-01T16:52:02.0363940Z GITHUB_RETENTION_DAYS=90 2024-11-01T16:52:02.0364309Z OPENSSL_DIR=/opt/openssl 2024-11-01T16:52:02.0364682Z GITHUB_ACTION_REPOSITORY= 2024-11-01T16:52:02.0365884Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-11-01T16:52:02.0367138Z GITHUB_BASE_REF=gh/c00w/2/base 2024-11-01T16:52:02.0367981Z ARTIFACTS_FILE_SUFFIX=test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_32397034850 2024-11-01T16:52:02.0368825Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-11-01T16:52:02.0369220Z CI=true 2024-11-01T16:52:02.0369554Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-11-01T16:52:02.0369990Z GITHUB_REPOSITORY_OWNER=pytorch 2024-11-01T16:52:02.0370389Z JOB_ID=32397034850 2024-11-01T16:52:02.0370717Z INSTALLED_PROTOBUF=yes 2024-11-01T16:52:02.0371085Z GITHUB_HEAD_REF=gh/c00w/2/head 2024-11-01T16:52:02.0371475Z GITHUB_ACTION_REF= 2024-11-01T16:52:02.0372055Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-11-01T16:52:02.0372579Z TEST_SHOWLOCALS=False 2024-11-01T16:52:02.0372929Z GITHUB_WORKFLOW=pull 2024-11-01T16:52:02.0373289Z DEBIAN_FRONTEND=noninteractive 2024-11-01T16:52:02.0374355Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_6b59c2d6-10ce-4869-92e7-36a56a58367a 2024-11-01T16:52:02.0375292Z NO_TD=False 2024-11-01T16:52:02.0375595Z SKIP_SCCACHE_INITIALIZATION=1 2024-11-01T16:52:02.0376082Z _=/usr/bin/env 2024-11-01T16:52:02.0376797Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-11-01T16:52:02.0545218Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-11-01T16:52:02.0546289Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-11-01T16:52:02.0547498Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-11-01T16:52:02.0548561Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-11-01T16:52:02.0549285Z + BUILD_DIR=build 2024-11-01T16:52:02.0549645Z + BUILD_RENAMED_DIR=build_renamed 2024-11-01T16:52:02.0550079Z + BUILD_BIN_DIR=build/bin 2024-11-01T16:52:02.0550436Z + SHARD_NUMBER=2 2024-11-01T16:52:02.0550765Z + NUM_TEST_SHARDS=3 2024-11-01T16:52:02.0551108Z + export VALGRIND=ON 2024-11-01T16:52:02.0551451Z + VALGRIND=ON 2024-11-01T16:52:02.0551939Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *clang9* ]] 2024-11-01T16:52:02.0552457Z + [[ 0 == \1 ]] 2024-11-01T16:52:02.0552773Z + [[ False == \1 ]] 2024-11-01T16:52:02.0553385Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *bazel* ]] 2024-11-01T16:52:02.0553939Z ++ realpath build/custom_test_artifacts 2024-11-01T16:52:02.0560265Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-11-01T16:52:02.0561046Z + [[ -n '' ]] 2024-11-01T16:52:02.0561409Z + echo 'Environment variables' 2024-11-01T16:52:02.0561827Z Environment variables 2024-11-01T16:52:02.0562157Z + env 2024-11-01T16:52:02.0568941Z INSTALLED_DB=yes 2024-11-01T16:52:02.0569312Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-11-01T16:52:02.0569931Z NVIDIA_VISIBLE_DEVICES=all 2024-11-01T16:52:02.0570346Z NV_NVML_DEV_VERSION=11.8.86-1 2024-11-01T16:52:02.0571033Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-11-01T16:52:02.0571694Z CONTINUE_THROUGH_ERROR=False 2024-11-01T16:52:02.0572256Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-11-01T16:52:02.0572849Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-11-01T16:52:02.0573487Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-11-01T16:52:02.0574043Z HOSTNAME=e581e04e5ab1 2024-11-01T16:52:02.0575062Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_6b59c2d6-10ce-4869-92e7-36a56a58367a 2024-11-01T16:52:02.0576120Z GITHUB_ACTION=__self 2024-11-01T16:52:02.0576507Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-11-01T16:52:02.0579097Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-11-01T16:52:02.0581882Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-11-01T16:52:02.0582451Z NV_NVTX_VERSION=11.8.86-1 2024-11-01T16:52:02.0582839Z GITHUB_RUN_NUMBER=261361 2024-11-01T16:52:02.0583223Z TEST_CONFIG=distributed 2024-11-01T16:52:02.0583618Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-11-01T16:52:02.0584153Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-11-01T16:52:02.0584611Z IS_A100_RUNNER=0 2024-11-01T16:52:02.0584977Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-11-01T16:52:02.0585473Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-11-01T16:52:02.0586256Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2024-11-01T16:52:02.0586718Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-11-01T16:52:02.0587449Z GITHUB_TRIGGERING_ACTOR=c00w 2024-11-01T16:52:02.0587922Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-11-01T16:52:02.0588443Z GITHUB_REF_TYPE=branch 2024-11-01T16:52:02.0588791Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-11-01T16:52:02.0589200Z NCCL_VERSION=2.16.2-1 2024-11-01T16:52:02.0589627Z BASE_SHA=2055d3c8dc8b93c52eda6b9c07d716448d9e28c9 2024-11-01T16:52:02.0590120Z XLA_CUDA= 2024-11-01T16:52:02.0590630Z HUGGING_FACE_HUB_TOKEN=*** 2024-11-01T16:52:02.0591050Z *** 2024-11-01T16:52:02.0591326Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-11-01T16:52:02.0591755Z GITHUB_REPOSITORY_ID=65600975 2024-11-01T16:52:02.0592145Z GITHUB_ACTIONS=true 2024-11-01T16:52:02.0592494Z NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:52:02.0593029Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-11-01T16:52:02.0593611Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-11-01T16:52:02.0594133Z SHA1=d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:52:02.0594786Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-11-01T16:52:02.0595320Z GITHUB_SHA=368da7510f5d755034681777a4e5ae6d33c07b38 2024-11-01T16:52:02.0596119Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/138766/merge 2024-11-01T16:52:02.0596841Z UCC_HOME=/usr 2024-11-01T16:52:02.0597192Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-11-01T16:52:02.0597631Z VERBOSE_TEST_LOGS=False 2024-11-01T16:52:02.0598001Z NVIDIA_PRODUCT_NAME=CUDA 2024-11-01T16:52:02.0598502Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-11-01T16:52:02.0599064Z GITHUB_REF=refs/pull/138766/merge 2024-11-01T16:52:02.0599504Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-11-01T16:52:02.0599904Z SHARD_NUMBER=2 2024-11-01T16:52:02.0600226Z GITHUB_REF_PROTECTED=false 2024-11-01T16:52:02.0600599Z HOME=/var/lib/jenkins 2024-11-01T16:52:02.0601174Z GITHUB_API_URL=https://api.github.com 2024-11-01T16:52:02.0601826Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-11-01T16:52:02.0602380Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-11-01T16:52:02.0602929Z SCCACHE_S3_KEY_PREFIX=pull 2024-11-01T16:52:02.0603318Z CUDA_VERSION=11.8.0 2024-11-01T16:52:02.0603799Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-11-01T16:52:02.0604283Z NUM_TEST_SHARDS=3 2024-11-01T16:52:02.0604609Z UCX_HOME=/usr 2024-11-01T16:52:02.0605194Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-11-01T16:52:02.0606501Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_6b59c2d6-10ce-4869-92e7-36a56a58367a 2024-11-01T16:52:02.0608026Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-11-01T16:52:02.0609529Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_6b59c2d6-10ce-4869-92e7-36a56a58367a 2024-11-01T16:52:02.0610864Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-11-01T16:52:02.0611656Z GITHUB_EVENT_NAME=pull_request 2024-11-01T16:52:02.0612066Z DASHBOARD_TAG= 2024-11-01T16:52:02.0612392Z GITHUB_RUN_ID=11632514903 2024-11-01T16:52:02.0612914Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-11-01T16:52:02.0613665Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-11-01T16:52:02.0614815Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_6b59c2d6-10ce-4869-92e7-36a56a58367a 2024-11-01T16:52:02.0615880Z GITHUB_ACTOR=c00w 2024-11-01T16:52:02.0616431Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-11-01T16:52:02.0616857Z PR_NUMBER=138766 2024-11-01T16:52:02.0617194Z GITHUB_RUN_ATTEMPT=1 2024-11-01T16:52:02.0617526Z VALGRIND=ON 2024-11-01T16:52:02.0617848Z ANACONDA_PYTHON_VERSION=3.10 2024-11-01T16:52:02.0618349Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-11-01T16:52:02.0618867Z TERM=vt100 2024-11-01T16:52:02.0619226Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-11-01T16:52:02.0619672Z INSTALLED_VISION=yes 2024-11-01T16:52:02.0620022Z BRANCH=pull/138766 2024-11-01T16:52:02.0620415Z SCCACHE_REGION=us-east-1 2024-11-01T16:52:02.0620804Z OPENSSL_ROOT_DIR=/opt/openssl 2024-11-01T16:52:02.0621394Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-11-01T16:52:02.0621853Z CUDA_PATH=/usr/local/cuda 2024-11-01T16:52:02.0622748Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-11-01T16:52:02.0623628Z GITHUB_SERVER_URL=https://github.com 2024-11-01T16:52:02.0624182Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-11-01T16:52:02.0624715Z REENABLED_ISSUES= 2024-11-01T16:52:02.0625032Z SHLVL=1 2024-11-01T16:52:02.0625297Z MAX_JOBS=46 2024-11-01T16:52:02.0625631Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-11-01T16:52:02.0626021Z NVARCH=x86_64 2024-11-01T16:52:02.0626340Z GITHUB_ACTOR_ID=486199 2024-11-01T16:52:02.0626824Z GITHUB_WORKFLOW_SHA=368da7510f5d755034681777a4e5ae6d33c07b38 2024-11-01T16:52:02.0627415Z GITHUB_REF_NAME=138766/merge 2024-11-01T16:52:02.0628042Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-11-01T16:52:02.0628893Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-11-01T16:52:02.0629543Z GITHUB_JOB=test 2024-11-01T16:52:02.0629985Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-11-01T16:52:02.0630588Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-11-01T16:52:02.0631153Z NO_TEST_TIMEOUT=False 2024-11-01T16:52:02.0631506Z TD_DISTRIBUTED=False 2024-11-01T16:52:02.0631913Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-11-01T16:52:02.0632400Z GITHUB_REPOSITORY=pytorch/pytorch 2024-11-01T16:52:02.0632842Z NV_NVPROF_VERSION=11.8.87-1 2024-11-01T16:52:02.0633232Z GITHUB_RETENTION_DAYS=90 2024-11-01T16:52:02.0633605Z OPENSSL_DIR=/opt/openssl 2024-11-01T16:52:02.0633979Z GITHUB_ACTION_REPOSITORY= 2024-11-01T16:52:02.0635184Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-11-01T16:52:02.0636452Z GITHUB_BASE_REF=gh/c00w/2/base 2024-11-01T16:52:02.0637290Z ARTIFACTS_FILE_SUFFIX=test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_32397034850 2024-11-01T16:52:02.0638096Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-11-01T16:52:02.0638492Z CI=true 2024-11-01T16:52:02.0638826Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-11-01T16:52:02.0639275Z GITHUB_REPOSITORY_OWNER=pytorch 2024-11-01T16:52:02.0639665Z JOB_ID=32397034850 2024-11-01T16:52:02.0639997Z INSTALLED_PROTOBUF=yes 2024-11-01T16:52:02.0640362Z GITHUB_HEAD_REF=gh/c00w/2/head 2024-11-01T16:52:02.0640759Z GITHUB_ACTION_REF= 2024-11-01T16:52:02.0641236Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-11-01T16:52:02.0641741Z TEST_SHOWLOCALS=False 2024-11-01T16:52:02.0642087Z GITHUB_WORKFLOW=pull 2024-11-01T16:52:02.0642449Z DEBIAN_FRONTEND=noninteractive 2024-11-01T16:52:02.0643506Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_6b59c2d6-10ce-4869-92e7-36a56a58367a 2024-11-01T16:52:02.0644444Z NO_TD=False 2024-11-01T16:52:02.0644747Z SKIP_SCCACHE_INITIALIZATION=1 2024-11-01T16:52:02.0645138Z _=/usr/bin/env 2024-11-01T16:52:02.0645500Z + echo 'Testing pytorch' 2024-11-01T16:52:02.0645866Z Testing pytorch 2024-11-01T16:52:02.0646202Z + export LANG=C.UTF-8 2024-11-01T16:52:02.0646578Z + LANG=C.UTF-8 2024-11-01T16:52:02.0646915Z + PR_NUMBER=138766 2024-11-01T16:52:02.0647272Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-11-01T16:52:02.0647763Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-11-01T16:52:02.0648390Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-11-01T16:52:02.0648917Z + [[ distributed == \s\l\o\w ]] 2024-11-01T16:52:02.0649530Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *slow-gradcheck* ]] 2024-11-01T16:52:02.0650245Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-11-01T16:52:02.0650818Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-11-01T16:52:02.0651334Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-11-01T16:52:02.0651795Z + [[ distributed == *crossref* ]] 2024-11-01T16:52:02.0652340Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-11-01T16:52:02.0652994Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *xpu* ]] 2024-11-01T16:52:02.0653659Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-11-01T16:52:02.0654350Z + pip_install --user ninja==1.10.2 2024-11-01T16:52:02.0654938Z + pip install --progress-bar off --user ninja==1.10.2 2024-11-01T16:52:03.8700353Z Collecting ninja==1.10.2 2024-11-01T16:52:03.8909231Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-11-01T16:52:03.9022942Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-11-01T16:52:04.2357955Z Installing collected packages: ninja 2024-11-01T16:52:04.2438586Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-11-01T16:52:04.2440104Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-11-01T16:52:04.2498151Z Successfully installed ninja-1.10.2 2024-11-01T16:52:04.3162377Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-11-01T16:52:04.3165240Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-11-01T16:52:04.3166997Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-11-01T16:52:04.3167535Z + install_tlparse 2024-11-01T16:52:04.3167937Z + pip_install --user tlparse==0.3.25 2024-11-01T16:52:04.3168564Z + pip install --progress-bar off --user tlparse==0.3.25 2024-11-01T16:52:04.7141989Z Collecting tlparse==0.3.25 2024-11-01T16:52:04.7358721Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-11-01T16:52:04.7462221Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-11-01T16:52:05.1024888Z Installing collected packages: tlparse 2024-11-01T16:52:05.1407950Z Successfully installed tlparse-0.3.25 2024-11-01T16:52:05.2066368Z ++ python -m site --user-base 2024-11-01T16:52:05.2322093Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-11-01T16:52:05.2323984Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *asan* ]] 2024-11-01T16:52:05.2324686Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-debug* ]] 2024-11-01T16:52:05.2325386Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-11-01T16:52:05.2326420Z + echo 'We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass' 2024-11-01T16:52:05.2327680Z We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass 2024-11-01T16:52:05.2328443Z + cd test 2024-11-01T16:52:05.2329056Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-11-01T16:52:06.9476566Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-11-01T16:52:06.9477280Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-11-01T16:52:06.9477803Z + DYNAMO_BENCHMARK_FLAGS=() 2024-11-01T16:52:06.9479169Z + [[ distributed == *pr_time_benchmarks* ]] 2024-11-01T16:52:06.9479670Z + [[ distributed == *dynamo_eager* ]] 2024-11-01T16:52:06.9480147Z + [[ distributed == *aot_eager* ]] 2024-11-01T16:52:06.9480597Z + [[ distributed == *aot_inductor* ]] 2024-11-01T16:52:06.9481054Z + [[ distributed == *inductor* ]] 2024-11-01T16:52:06.9481488Z + [[ distributed == *dynamic* ]] 2024-11-01T16:52:06.9481911Z + [[ distributed == *cpu* ]] 2024-11-01T16:52:06.9482564Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-11-01T16:52:06.9520012Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-11-01T16:52:06.9520853Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-bazel-* ]] 2024-11-01T16:52:06.9521375Z + cd test 2024-11-01T16:52:06.9521878Z + python -c 'import torch; print(torch.__config__.show())' 2024-11-01T16:52:08.9103057Z PyTorch built with: 2024-11-01T16:52:08.9103766Z - GCC 9.4 2024-11-01T16:52:08.9104134Z - C++ Version: 201703 2024-11-01T16:52:08.9105180Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-11-01T16:52:08.9106516Z - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-11-01T16:52:08.9107286Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-11-01T16:52:08.9107876Z - LAPACK is enabled (usually provided by MKL) 2024-11-01T16:52:08.9108426Z - NNPACK is enabled 2024-11-01T16:52:08.9108857Z - CPU capability usage: AVX512 2024-11-01T16:52:08.9128691Z - CUDA Runtime 11.8 2024-11-01T16:52:08.9129497Z - NVCC architecture flags: -gencode;arch=compute_75,code=sm_75 2024-11-01T16:52:08.9130118Z - CuDNN 90.1 2024-11-01T16:52:08.9130450Z - Magma 2.6.1 2024-11-01T16:52:08.9139733Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.6.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-11-01T16:52:08.9147931Z 2024-11-01T16:52:09.3299868Z + cd test 2024-11-01T16:52:09.3300725Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-11-01T16:52:10.7937425Z ATen/Parallel: 2024-11-01T16:52:10.7937910Z at::get_num_threads() : 24 2024-11-01T16:52:10.7938350Z at::get_num_interop_threads() : 24 2024-11-01T16:52:10.7938830Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-11-01T16:52:10.7939274Z omp_get_max_threads() : 24 2024-11-01T16:52:10.7940465Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-11-01T16:52:10.7941423Z mkl_get_max_threads() : 24 2024-11-01T16:52:10.7942161Z Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-11-01T16:52:10.7942915Z std::thread::hardware_concurrency() : 48 2024-11-01T16:52:10.7943423Z Environment variables: 2024-11-01T16:52:10.7943809Z OMP_NUM_THREADS : [not set] 2024-11-01T16:52:10.7944241Z MKL_NUM_THREADS : [not set] 2024-11-01T16:52:10.7944663Z ATen parallel backend: OpenMP 2024-11-01T16:52:10.7944943Z 2024-11-01T16:52:11.0339647Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-11-01T16:52:11.0340328Z + [[ distributed == *backward* ]] 2024-11-01T16:52:11.0340796Z + [[ distributed == *xla* ]] 2024-11-01T16:52:11.0341255Z + [[ distributed == *executorch* ]] 2024-11-01T16:52:11.0341761Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-11-01T16:52:11.0342464Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-11-01T16:52:11.0343039Z + [[ distributed == distributed ]] 2024-11-01T16:52:11.0343477Z + test_distributed 2024-11-01T16:52:11.0343930Z + echo 'Testing distributed python tests' 2024-11-01T16:52:11.0344437Z Testing distributed python tests 2024-11-01T16:52:11.0345148Z + python test/run_test.py --distributed-tests --shard 2 3 --verbose 2024-11-01T16:52:11.1398842Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-11-01T16:52:11.1400477Z import pkg_resources 2024-11-01T16:52:16.2749402Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-11-01T16:52:16.3289496Z Ignoring disabled issues: [''] 2024-11-01T16:52:16.3446616Z Found test times from artifacts 2024-11-01T16:52:16.4069914Z Found test times from artifacts 2024-11-01T16:52:16.4091126Z Running 25% of tests based on TD 2024-11-01T16:52:16.4201381Z Running parallel tests on 2 processes 2024-11-01T16:52:16.4202078Z Name: tests to run (est. time: 7.56min) 2024-11-01T16:52:16.4203002Z Serial tests (6): 2024-11-01T16:52:16.4203741Z distributed/test_dynamo_distributed 1/1 2024-11-01T16:52:16.4204457Z distributed/pipelining/test_microbatch 1/1 2024-11-01T16:52:16.4205344Z distributed/_tensor/test_experimental_ops 1/1 2024-11-01T16:52:16.4205918Z distributed/_tensor/test_tensor_ops 1/1 2024-11-01T16:52:16.4206544Z distributed/_composable/fsdp/test_fully_shard_logging 1/1 2024-11-01T16:52:16.4207206Z distributed/checkpoint/test_format_utils 1/1 2024-11-01T16:52:16.4207730Z Parallel tests (0): 2024-11-01T16:52:16.4208121Z Name: excluded (est. time: 102.54min) 2024-11-01T16:52:16.4208577Z Serial tests (57): 2024-11-01T16:52:16.4208941Z distributed/test_c10d_gloo 1/1 2024-11-01T16:52:16.4209430Z distributed/test_distributed_spawn 3/10 2024-11-01T16:52:16.4209970Z distributed/test_distributed_spawn 6/10 2024-11-01T16:52:16.4210507Z distributed/test_distributed_spawn 9/10 2024-11-01T16:52:16.4211055Z distributed/fsdp/test_fsdp_state_dict 2/2 2024-11-01T16:52:16.4211603Z distributed/test_c10d_nccl 1/3 2024-11-01T16:52:16.4212185Z distributed/_composable/fsdp/test_fully_shard_training 1/1 2024-11-01T16:52:16.4212849Z distributed/fsdp/test_fsdp_mixed_precision 1/1 2024-11-01T16:52:16.4213513Z distributed/test_c10d_ucc 1/1 2024-11-01T16:52:16.4214021Z distributed/fsdp/test_fsdp_freezing_weights 1/1 2024-11-01T16:52:16.4214555Z distributed/test_c10d_pypg 1/1 2024-11-01T16:52:16.4215115Z distributed/_composable/test_replicate_with_compiler 1/1 2024-11-01T16:52:16.4215868Z distributed/_composable/fsdp/test_fully_shard_comm 1/1 2024-11-01T16:52:16.4216822Z distributed/_composable/test_composability/test_2d_composability 1/1 2024-11-01T16:52:16.4217553Z distributed/_composable/test_replicate 1/1 2024-11-01T16:52:16.4218091Z distributed/fsdp/test_fsdp_misc 1/1 2024-11-01T16:52:16.4218629Z distributed/fsdp/test_fsdp_hybrid_shard 1/1 2024-11-01T16:52:16.4219197Z distributed/_tensor/test_redistribute 1/1 2024-11-01T16:52:16.4219720Z distributed/algorithms/test_join 1/1 2024-11-01T16:52:16.4220223Z distributed/test_c10d_spawn_nccl 1/1 2024-11-01T16:52:16.4220765Z distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 2024-11-01T16:52:16.4221318Z distributed/fsdp/test_fsdp_meta 1/1 2024-11-01T16:52:16.4221938Z distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 2024-11-01T16:52:16.4222564Z distributed/test_c10d_spawn_gloo 1/1 2024-11-01T16:52:16.4223170Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 2024-11-01T16:52:16.4223914Z distributed/_composable/fully_shard/test_fully_shard_init 1/1 2024-11-01T16:52:16.4224649Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 2024-11-01T16:52:16.4225271Z distributed/_tensor/test_random_ops 1/1 2024-11-01T16:52:16.4225906Z distributed/_composable/fsdp/test_fully_shard_autograd 1/1 2024-11-01T16:52:16.4226609Z distributed/checkpoint/test_file_system_checkpoint 1/1 2024-11-01T16:52:16.4227212Z distributed/test_c10d_spawn_ucc 1/1 2024-11-01T16:52:16.4227811Z distributed/test_c10d_ops_nccl 1/1 2024-11-01T16:52:16.4228450Z distributed/fsdp/test_hsdp_dtensor_state_dict 1/1 2024-11-01T16:52:16.4229001Z distributed/_tensor/test_view_ops 1/1 2024-11-01T16:52:16.4229488Z distributed/_tensor/test_embedding_ops 1/1 2024-11-01T16:52:16.4230237Z distributed/_tensor/experimental/test_local_map 1/1 2024-11-01T16:52:16.4230811Z distributed/fsdp/test_fsdp_pure_fp16 1/1 2024-11-01T16:52:16.4231402Z distributed/_tensor/experimental/test_tp_transform 1/1 2024-11-01T16:52:16.4231967Z distributed/_shard/test_sharder 1/1 2024-11-01T16:52:16.4232444Z distributed/fsdp/test_shard_utils 1/1 2024-11-01T16:52:16.4232953Z distributed/checkpoint/test_tp_checkpoint 1/1 2024-11-01T16:52:16.4233518Z distributed/checkpoint/e2e/test_pipeline 1/1 2024-11-01T16:52:16.4234099Z distributed/fsdp/test_fsdp_multiple_wrapping 1/1 2024-11-01T16:52:16.4234851Z distributed/elastic/utils/distributed_test 1/1 2024-11-01T16:52:16.4235426Z distributed/elastic/test_control_plane 1/1 2024-11-01T16:52:16.4235978Z distributed/checkpoint/e2e/test_fine_tuning 1/1 2024-11-01T16:52:16.4236677Z distributed/tensor/parallel/test_tp_random_state 1/1 2024-11-01T16:52:16.4237235Z distributed/test_nccl 1/1 2024-11-01T16:52:16.4237657Z distributed/test_fake_pg 1/1 2024-11-01T16:52:16.4238134Z distributed/_tensor/test_pointwise_ops 1/1 2024-11-01T16:52:16.4238642Z distributed/fsdp/test_utils 1/1 2024-11-01T16:52:16.4239142Z distributed/checkpoint/test_compatibility 1/1 2024-11-01T16:52:16.4239696Z distributed/elastic/events/lib_test 1/1 2024-11-01T16:52:16.4240231Z distributed/test_control_collectives 1/1 2024-11-01T16:52:16.4240766Z distributed/_tensor/test_op_strategy 1/1 2024-11-01T16:52:16.4241367Z distributed/optim/test_apply_optimizer_in_backward 1/1 2024-11-01T16:52:16.4241965Z distributed/rpc/test_tensorpipe_agent 1/1 2024-11-01T16:52:16.4242460Z Parallel tests (0): 2024-11-01T16:52:16.4261824Z Running distributed/test_dynamo_distributed 1/1 ... [2024-11-01 16:52:16.425895] 2024-11-01T16:52:16.4262576Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:52:16.4266212Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_dynamo_distributed.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:52:16.426328] 2024-11-01T16:57:24.1213745Z 2024-11-01T16:57:24.1215642Z distributed/test_dynamo_distributed 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_dynamo_distributed_1.1_4e217fbad2522b58_.log 2024-11-01T16:57:24.1250684Z Running 56 items in this shard: test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_call_method_forward, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_issue90375, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_symbol_splitting, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_direct, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_indirect, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_no_binding, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_torture_multi, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_asymmetric_compilation, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_asymmetric_compilation_with_fx_cache, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_scalar, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_speculation_divergence, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_tensor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_dim_mismatch, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_graph_break_empty_graph_still_collective, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_missing_source, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_scalar_missing_source, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_type_mismatch, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_baseline_aot_eager_multiprocess, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_setattr, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_unspecialized_forced_getattr_inline, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_unspecialized_forced_getattr_no_inline, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager_static_graph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor_static_graph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_aot_autograd, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_async_subclass_no_specialize, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_compiled_flex_attention_full_model_ddp, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_compiled_flex_attention_local_ddp, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_custom_layer, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_aot_eager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_empty_graph_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_diff_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_same_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_orig_params_assert, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_guards, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_register_attr_or_module, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_staticmethod, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_ctx_manager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_inference, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_training, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_transpose, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_higher_order_op, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ignored_parameters, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_no_split 2024-11-01T16:57:24.1283989Z 2024-11-01T16:57:24.1284595Z Running distributed/pipelining/test_microbatch 1/1 ... [2024-11-01 16:57:24.121754] 2024-11-01T16:57:24.1285352Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:57:24.1289306Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_microbatch.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:57:24.122132] 2024-11-01T16:57:28.6979524Z 2024-11-01T16:57:28.6981594Z distributed/pipelining/test_microbatch 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_microbatch_1.1_c9674f305f760b06_.log 2024-11-01T16:57:28.6984162Z Running 2 items in this shard: test/distributed/pipelining/test_microbatch.py::MicrobatchTests::test_chunk_spec, test/distributed/pipelining/test_microbatch.py::MicrobatchTests::test_split_and_merge 2024-11-01T16:57:28.6987574Z 2024-11-01T16:57:28.6989530Z Running distributed/_tensor/test_experimental_ops 1/1 ... [2024-11-01 16:57:28.698251] 2024-11-01T16:57:28.6990323Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:57:28.6992392Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_experimental_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:57:28.698611] 2024-11-01T16:57:45.8565404Z 2024-11-01T16:57:45.8567284Z distributed/_tensor/test_experimental_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_experimental_ops_1.1_e4290363901e788e_.log 2024-11-01T16:57:45.8570245Z Running 3 items in this shard: test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_bernoulli, test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_nll, test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_slice 2024-11-01T16:57:45.8571999Z 2024-11-01T16:57:45.8572532Z Running distributed/_tensor/test_tensor_ops 1/1 ... [2024-11-01 16:57:45.856375] 2024-11-01T16:57:45.8573276Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:57:45.8575280Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_tensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:57:45.856729] 2024-11-01T16:59:48.3887122Z 2024-11-01T16:59:48.3889024Z distributed/_tensor/test_tensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_tensor_ops_1.1_49ef08eb08d07087_.log 2024-11-01T16:59:48.3903138Z Running 25 items in this shard: test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_aten_contiguous, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_clone, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_contiguous, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_detach, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_dtensor_dtype_conversion, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_empty_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_equal, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_fill_inplace, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_fill_inplace_partial_sum, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_full_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_gather, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_index, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_inplace_op, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_new_empty_strided, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_new_full, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_ones_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_ones_like_partial_sum, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_op_out_variant, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_scatter, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_slice, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_stack, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_where_type_promotion, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zero_inplace, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zeros_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zeros_like_partial_sum 2024-11-01T16:59:48.3916092Z 2024-11-01T16:59:48.3916727Z Running distributed/_composable/fsdp/test_fully_shard_logging 1/1 ... [2024-11-01 16:59:48.388796] 2024-11-01T16:59:48.3917512Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:59:48.3919698Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_logging.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:59:48.389147] 2024-11-01T16:59:52.4520082Z 2024-11-01T16:59:52.4522130Z distributed/_composable/fsdp/test_fully_shard_logging 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_logging_1.1_3ddb2db7a7b6c949_.log 2024-11-01T16:59:52.4523766Z Running 0 items in this shard: 2024-11-01T16:59:52.4524081Z 2024-11-01T16:59:52.4524633Z Running distributed/checkpoint/test_format_utils 1/1 ... [2024-11-01 16:59:52.451992] 2024-11-01T16:59:52.4525403Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:59:52.4527490Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_format_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:59:52.452386] 2024-11-01T17:00:12.7204454Z 2024-11-01T17:00:12.7206489Z distributed/checkpoint/test_format_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_format_utils_1.1_d7e886c3ecd32850_.log 2024-11-01T17:00:12.7209617Z Running 3 items in this shard: test/distributed/checkpoint/test_format_utils.py::TestFormatUtils::test_dcp_to_torch_save, test/distributed/checkpoint/test_format_utils.py::TestFormatUtils::test_online_torch_save_to_dcp, test/distributed/checkpoint/test_format_utils.py::TestFormatUtils::test_torch_save_to_dcp 2024-11-01T17:00:12.7211589Z 2024-11-01T17:00:13.4222514Z Running test batch 'tests to run' cost 477.0 seconds 2024-11-01T17:00:14.2145620Z 2024-11-01T17:00:14.2146063Z real 8m3.181s 2024-11-01T17:00:14.2146446Z user 21m37.756s 2024-11-01T17:00:14.2150289Z sys 7m15.185s 2024-11-01T17:00:14.2150663Z + assert_git_not_dirty 2024-11-01T17:00:14.2151371Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-11-01T17:00:14.2152046Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *xla* ]] 2024-11-01T17:00:14.2152611Z ++ git status --porcelain 2024-11-01T17:00:14.2153038Z ++ grep -v '?? third_party' 2024-11-01T17:00:16.7857734Z ++ true 2024-11-01T17:00:16.7858121Z + git_status= 2024-11-01T17:00:16.7858672Z + [[ -n '' ]] 2024-11-01T17:00:16.7859188Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-11-01T17:00:16.7859712Z + [[ 2 == 1 ]] 2024-11-01T17:00:16.7860026Z + [[ 2 == 1 ]] 2024-11-01T17:00:16.7860343Z + cleanup_workspace 2024-11-01T17:00:16.7861281Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-11-01T17:00:16.7862564Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-11-01T17:00:16.7863685Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-11-01T17:00:16.7864414Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-11-01T17:00:16.7865659Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-11-01T17:00:16.7866713Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-11-01T17:00:16.7867521Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-11-01T17:00:17.2829364Z ##[group]Run cat test/**/*_toprint.log || true 2024-11-01T17:00:17.2829934Z cat test/**/*_toprint.log || true 2024-11-01T17:00:17.2839925Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:00:17.2840448Z env: 2024-11-01T17:00:17.2840721Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:00:17.2841178Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:00:17.2841950Z DOCKER_CONTAINER_ID: e581e04e5ab141ca02b611c02d76c89f1cb6a25e20d241d526180d97229d0e5e 2024-11-01T17:00:17.2842632Z ##[endgroup] 2024-11-01T17:00:17.2924789Z cat: 'test/**/*_toprint.log': No such file or directory 2024-11-01T17:00:17.2959709Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-11-01T17:00:17.2960231Z kill "$MONITOR_SCRIPT_PID" 2024-11-01T17:00:17.2966944Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:00:17.2967476Z env: 2024-11-01T17:00:17.2967758Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:00:17.2968244Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:00:17.2969067Z DOCKER_CONTAINER_ID: e581e04e5ab141ca02b611c02d76c89f1cb6a25e20d241d526180d97229d0e5e 2024-11-01T17:00:17.2969797Z MONITOR_SCRIPT_PID: 99406 2024-11-01T17:00:17.2970154Z ##[endgroup] 2024-11-01T17:00:17.3191869Z Prepare all required actions 2024-11-01T17:00:17.3192331Z Getting action download info 2024-11-01T17:00:17.5955116Z Download action repository 'actions/upload-artifact@v3' (SHA:ff15f0306b3f739f7b6fd43fb5d26cd321bd4de5) 2024-11-01T17:00:17.7587660Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-11-01T17:00:17.7588293Z with: 2024-11-01T17:00:17.7588876Z file-suffix: test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_32397034850 2024-11-01T17:00:17.7589637Z s3-bucket: gha-artifacts 2024-11-01T17:00:17.7589995Z env: 2024-11-01T17:00:17.7590280Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:00:17.7590762Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:00:17.7591586Z DOCKER_CONTAINER_ID: e581e04e5ab141ca02b611c02d76c89f1cb6a25e20d241d526180d97229d0e5e 2024-11-01T17:00:17.7592319Z ##[endgroup] 2024-11-01T17:00:17.7626217Z ##[group]Run # Remove any previous test jsons if they exist 2024-11-01T17:00:17.7627001Z # Remove any previous test jsons if they exist 2024-11-01T17:00:17.7627587Z rm -f test-jsons-*.zip 2024-11-01T17:00:17.7628382Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json' 2024-11-01T17:00:17.7634942Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:00:17.7635489Z env: 2024-11-01T17:00:17.7635763Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:00:17.7636255Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:00:17.7637209Z DOCKER_CONTAINER_ID: e581e04e5ab141ca02b611c02d76c89f1cb6a25e20d241d526180d97229d0e5e 2024-11-01T17:00:17.7638267Z FILE_SUFFIX: test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_32397034850 2024-11-01T17:00:17.7638922Z ##[endgroup] 2024-11-01T17:00:17.7688016Z adding: test/test-reports/td_exclusions-4bd7bc79f913127d94c6.json (deflated 86%) 2024-11-01T17:00:17.7719153Z ##[group]Run # Remove any previous test reports if they exist 2024-11-01T17:00:17.7719857Z # Remove any previous test reports if they exist 2024-11-01T17:00:17.7720433Z rm -f test-reports-*.zip 2024-11-01T17:00:17.7721141Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv' 2024-11-01T17:00:17.7727393Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:00:17.7727916Z env: 2024-11-01T17:00:17.7728198Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:00:17.7728673Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:00:17.7729622Z DOCKER_CONTAINER_ID: e581e04e5ab141ca02b611c02d76c89f1cb6a25e20d241d526180d97229d0e5e 2024-11-01T17:00:17.7730622Z FILE_SUFFIX: test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_32397034850 2024-11-01T17:00:17.7731415Z ##[endgroup] 2024-11-01T17:00:17.7780370Z adding: test/test-reports/python-pytest/distributed.test_dynamo_distributed/distributed.test_dynamo_distributed-855fdbb61d431263.xml (deflated 91%) 2024-11-01T17:00:17.7782598Z adding: test/test-reports/python-pytest/distributed.pipelining.test_microbatch/distributed.pipelining.test_microbatch-d9a91d811d01d79f.xml (deflated 54%) 2024-11-01T17:00:17.7784933Z adding: test/test-reports/python-pytest/distributed._tensor.test_experimental_ops/distributed._tensor.test_experimental_ops-710636b0121ed67b.xml (deflated 57%) 2024-11-01T17:00:17.7787042Z adding: test/test-reports/python-pytest/distributed._tensor.test_tensor_ops/distributed._tensor.test_tensor_ops-8a49486979d3f066.xml (deflated 86%) 2024-11-01T17:00:17.7789414Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_logging/distributed._composable.fsdp.test_fully_shard_logging-d572a39ac70ece99.xml (deflated 28%) 2024-11-01T17:00:17.7791739Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_format_utils/distributed.checkpoint.test_format_utils-687929863558e943.xml (deflated 57%) 2024-11-01T17:00:17.7819884Z ##[group]Run # Remove any previous usage logs if they exist 2024-11-01T17:00:17.7820612Z # Remove any previous usage logs if they exist 2024-11-01T17:00:17.7821306Z rm -f logs-*.zip 2024-11-01T17:00:17.7822079Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-11-01T17:00:17.7823005Z # so check to see if the file exists first 2024-11-01T17:00:17.7823578Z if [ -f 'usage_log.txt' ]; then 2024-11-01T17:00:17.7824175Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-11-01T17:00:17.7824727Z fi 2024-11-01T17:00:17.7825301Z if find "test/test-reports" -name "*.log" 2>/dev/null | grep -q .; then 2024-11-01T17:00:17.7826208Z  zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' 2024-11-01T17:00:17.7826841Z fi 2024-11-01T17:00:17.7832778Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:00:17.7833309Z env: 2024-11-01T17:00:17.7833586Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:00:17.7834048Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:00:17.7834872Z DOCKER_CONTAINER_ID: e581e04e5ab141ca02b611c02d76c89f1cb6a25e20d241d526180d97229d0e5e 2024-11-01T17:00:17.7835877Z FILE_SUFFIX: test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_32397034850 2024-11-01T17:00:17.7836573Z ##[endgroup] 2024-11-01T17:00:17.7884267Z adding: usage_log.txt (deflated 92%) 2024-11-01T17:00:17.7920348Z adding: test/test-reports/distributed.test_dynamo_distributed_1.1_4e217fbad2522b58_.log (deflated 87%) 2024-11-01T17:00:17.7921798Z adding: test/test-reports/distributed.pipelining.test_microbatch_1.1_c9674f305f760b06_.log (deflated 58%) 2024-11-01T17:00:17.7923256Z adding: test/test-reports/distributed._tensor.test_experimental_ops_1.1_e4290363901e788e_.log (deflated 66%) 2024-11-01T17:00:17.7924647Z adding: test/test-reports/distributed._tensor.test_tensor_ops_1.1_49ef08eb08d07087_.log (deflated 87%) 2024-11-01T17:00:17.7926140Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_logging_1.1_3ddb2db7a7b6c949_.log (deflated 53%) 2024-11-01T17:00:17.7928349Z adding: test/test-reports/distributed.checkpoint.test_format_utils_1.1_d7e886c3ecd32850_.log (deflated 82%) 2024-11-01T17:00:17.7955971Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-11-01T17:00:17.7956710Z # Remove any previous debugging artifacts if they exist 2024-11-01T17:00:17.7957272Z rm -f debug-*.zip 2024-11-01T17:00:17.7957651Z if [ -d 'test/debug' ]; then 2024-11-01T17:00:17.7958160Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-11-01T17:00:17.7958781Z fi 2024-11-01T17:00:17.7965431Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:00:17.7966473Z env: 2024-11-01T17:00:17.7967007Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:00:17.7967852Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:00:17.7969369Z DOCKER_CONTAINER_ID: e581e04e5ab141ca02b611c02d76c89f1cb6a25e20d241d526180d97229d0e5e 2024-11-01T17:00:17.7971352Z FILE_SUFFIX: test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_32397034850 2024-11-01T17:00:17.7972707Z ##[endgroup] 2024-11-01T17:00:17.8071584Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-11-01T17:00:17.8072024Z with: 2024-11-01T17:00:17.8072275Z s3-bucket: gha-artifacts 2024-11-01T17:00:17.8072702Z s3-prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T17:00:17.8073169Z retention-days: 14 2024-11-01T17:00:17.8073489Z if-no-files-found: warn 2024-11-01T17:00:17.8073837Z path: test-jsons-*.zip 2024-11-01T17:00:17.8074167Z name: artifact 2024-11-01T17:00:17.8074440Z region: us-east-1 2024-11-01T17:00:17.8074725Z env: 2024-11-01T17:00:17.8074979Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:00:17.8075424Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:00:17.8076189Z DOCKER_CONTAINER_ID: e581e04e5ab141ca02b611c02d76c89f1cb6a25e20d241d526180d97229d0e5e 2024-11-01T17:00:17.8076850Z ##[endgroup] 2024-11-01T17:00:18.1710853Z NOTE: s3-prefix specified, ignoring name parameter 2024-11-01T17:00:18.1711852Z With the provided path, there will be 1 file uploaded 2024-11-01T17:00:18.1712584Z Uploading to s3 prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T17:00:18.1755333Z Starting upload of test-jsons-test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_32397034850.zip 2024-11-01T17:00:18.2604378Z Finished upload of test-jsons-test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_32397034850.zip 2024-11-01T17:00:18.3299638Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-11-01T17:00:18.3300156Z with: 2024-11-01T17:00:18.3300459Z s3-bucket: gha-artifacts 2024-11-01T17:00:18.3301142Z s3-prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T17:00:18.3301692Z retention-days: 14 2024-11-01T17:00:18.3302049Z if-no-files-found: error 2024-11-01T17:00:18.3302454Z path: test-reports-*.zip 2024-11-01T17:00:18.3302839Z name: artifact 2024-11-01T17:00:18.3303168Z region: us-east-1 2024-11-01T17:00:18.3303492Z env: 2024-11-01T17:00:18.3303770Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:00:18.3304285Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:00:18.3305160Z DOCKER_CONTAINER_ID: e581e04e5ab141ca02b611c02d76c89f1cb6a25e20d241d526180d97229d0e5e 2024-11-01T17:00:18.3305918Z ##[endgroup] 2024-11-01T17:00:18.6925560Z NOTE: s3-prefix specified, ignoring name parameter 2024-11-01T17:00:18.6926423Z With the provided path, there will be 1 file uploaded 2024-11-01T17:00:18.6927120Z Uploading to s3 prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T17:00:18.6968132Z Starting upload of test-reports-test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_32397034850.zip 2024-11-01T17:00:18.7815541Z Finished upload of test-reports-test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_32397034850.zip 2024-11-01T17:00:18.7989134Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-11-01T17:00:18.7989583Z with: 2024-11-01T17:00:18.7989860Z s3-bucket: gha-artifacts 2024-11-01T17:00:18.7990284Z s3-prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T17:00:18.7990773Z retention-days: 14 2024-11-01T17:00:18.7991118Z if-no-files-found: ignore 2024-11-01T17:00:18.7991485Z path: logs-*.zip 2024-11-01T17:00:18.7991796Z name: artifact 2024-11-01T17:00:18.7992079Z region: us-east-1 2024-11-01T17:00:18.7992373Z env: 2024-11-01T17:00:18.7992646Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:00:18.7993104Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:00:18.7993888Z DOCKER_CONTAINER_ID: e581e04e5ab141ca02b611c02d76c89f1cb6a25e20d241d526180d97229d0e5e 2024-11-01T17:00:18.7994702Z ##[endgroup] 2024-11-01T17:00:19.1597744Z NOTE: s3-prefix specified, ignoring name parameter 2024-11-01T17:00:19.1598588Z With the provided path, there will be 1 file uploaded 2024-11-01T17:00:19.1599292Z Uploading to s3 prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T17:00:19.1640866Z Starting upload of logs-test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_32397034850.zip 2024-11-01T17:00:19.2820266Z Finished upload of logs-test-distributed-2-3-linux.g4dn.12xlarge.nvidia.gpu_32397034850.zip 2024-11-01T17:00:19.2998087Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-11-01T17:00:19.2998559Z with: 2024-11-01T17:00:19.2998834Z s3-bucket: gha-artifacts 2024-11-01T17:00:19.2999293Z s3-prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T17:00:19.2999797Z retention-days: 14 2024-11-01T17:00:19.3000146Z if-no-files-found: ignore 2024-11-01T17:00:19.3000519Z path: debug-*.zip 2024-11-01T17:00:19.3000838Z name: artifact 2024-11-01T17:00:19.3001547Z region: us-east-1 2024-11-01T17:00:19.3001936Z env: 2024-11-01T17:00:19.3002222Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:00:19.3002711Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:00:19.3003556Z DOCKER_CONTAINER_ID: e581e04e5ab141ca02b611c02d76c89f1cb6a25e20d241d526180d97229d0e5e 2024-11-01T17:00:19.3004278Z ##[endgroup] 2024-11-01T17:00:19.6546477Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-11-01T17:00:19.6726406Z ##[group]Run # shellcheck disable=SC2156 2024-11-01T17:00:19.6726888Z # shellcheck disable=SC2156 2024-11-01T17:00:19.6727730Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-11-01T17:00:19.6734054Z shell: /usr/bin/bash -e {0} 2024-11-01T17:00:19.6734407Z env: 2024-11-01T17:00:19.6734681Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:00:19.6735126Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:00:19.6736031Z DOCKER_CONTAINER_ID: e581e04e5ab141ca02b611c02d76c89f1cb6a25e20d241d526180d97229d0e5e 2024-11-01T17:00:19.6736942Z ##[endgroup] 2024-11-01T17:00:19.9194770Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-11-01T17:00:19.9195349Z with: 2024-11-01T17:00:19.9195595Z env: 2024-11-01T17:00:19.9195849Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:00:19.9196309Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:00:19.9197088Z DOCKER_CONTAINER_ID: e581e04e5ab141ca02b611c02d76c89f1cb6a25e20d241d526180d97229d0e5e 2024-11-01T17:00:19.9197768Z ##[endgroup] 2024-11-01T17:00:19.9222869Z ##[group]Run set -eou pipefail 2024-11-01T17:00:19.9223320Z set -eou pipefail 2024-11-01T17:00:19.9223707Z  2024-11-01T17:00:19.9224271Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-11-01T17:00:19.9225015Z for _ in $(seq 1440); do 2024-11-01T17:00:19.9225540Z  # Break if no ssh session exists anymore 2024-11-01T17:00:19.9226116Z  if [ "$(who)" = "" ]; then 2024-11-01T17:00:19.9226567Z  break 2024-11-01T17:00:19.9226942Z  fi 2024-11-01T17:00:19.9227266Z  echo "." 2024-11-01T17:00:19.9227607Z  sleep 5 2024-11-01T17:00:19.9227946Z done 2024-11-01T17:00:19.9233681Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:00:19.9234193Z env: 2024-11-01T17:00:19.9234466Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:00:19.9234902Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:00:19.9235681Z DOCKER_CONTAINER_ID: e581e04e5ab141ca02b611c02d76c89f1cb6a25e20d241d526180d97229d0e5e 2024-11-01T17:00:19.9236358Z ##[endgroup] 2024-11-01T17:00:19.9260981Z Holding runner for 2 hours until all ssh sessions have logged out 2024-11-01T17:00:19.9315301Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-11-01T17:00:19.9316242Z # ignore expansion of "docker ps -q" since it could be empty 2024-11-01T17:00:19.9316975Z # shellcheck disable=SC2046 2024-11-01T17:00:19.9317454Z docker stop $(docker ps -q) || true 2024-11-01T17:00:19.9317940Z # Prune all of the docker images 2024-11-01T17:00:19.9318401Z docker system prune -af 2024-11-01T17:00:19.9323844Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:00:19.9324356Z env: 2024-11-01T17:00:19.9324615Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:00:19.9325075Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:00:19.9325858Z DOCKER_CONTAINER_ID: e581e04e5ab141ca02b611c02d76c89f1cb6a25e20d241d526180d97229d0e5e 2024-11-01T17:00:19.9326539Z ##[endgroup] 2024-11-01T17:00:20.4010670Z e581e04e5ab1 2024-11-01T17:00:20.8216016Z Deleted Containers: 2024-11-01T17:00:20.8216640Z e581e04e5ab141ca02b611c02d76c89f1cb6a25e20d241d526180d97229d0e5e 2024-11-01T17:00:25.6043891Z 2024-11-01T17:00:25.6044333Z Deleted Images: 2024-11-01T17:00:25.6046659Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T17:00:25.6049794Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9@sha256:7e55ca35c7b3a515f97404dcaef8e9d5373f6e358ac63c424d2d80dda1e618f7 2024-11-01T17:00:25.6051493Z deleted: sha256:f4475432de42c25077a2ffe0f2f56efb42124dedca9504f01daf084f9103394b 2024-11-01T17:00:25.6052494Z deleted: sha256:0a3b02914acc05dcfc3fdaea37e1a4336aaa696c207e6e993e87815010457060 2024-11-01T17:00:25.6053490Z deleted: sha256:287a6b6bfe47fa7d50e7409829056a1cf37ad073cb6890163e779734ff296fb2 2024-11-01T17:00:25.6054451Z deleted: sha256:53fa61dc822c6716c85f6363bb6a83cd7541c8879d5f9b93dea1064af2d6463c 2024-11-01T17:00:25.6055420Z deleted: sha256:1a8f08fa472d2fb692f63de804eaaa893862346178152979127e0609eb12d696 2024-11-01T17:00:25.6056688Z deleted: sha256:a1b5599bc63b031b0d8a037d7dd06b8878dc70ed0692585d42de476263f63af3 2024-11-01T17:00:25.6057696Z deleted: sha256:f8784943057bdd6a3eec2f0816741a058296e19c92a8ec86595e225509b9d244 2024-11-01T17:00:25.6059012Z deleted: sha256:f62e2601c7c54fd6b496f2b9162da63bc6078f0b98bcc590c8438df3191843b0 2024-11-01T17:00:25.6060018Z deleted: sha256:429b19ab4950a882a2573ce19266e5e84f52dd115bf27e8a23188bae420b5ec2 2024-11-01T17:00:25.6060981Z deleted: sha256:2a955783b858a5e071968bf711c0c32f0932a3ca65a4421074551c69d0bbf98f 2024-11-01T17:00:25.6061987Z deleted: sha256:de6fbaec1030affac12cf8d0d4a3728e048f86288dd421ff999b7f3ee058f0b6 2024-11-01T17:00:25.6063010Z deleted: sha256:381f67e4868ab208466a6d548b250ab4f4a05944a6b27ff3bbee4afef20473ca 2024-11-01T17:00:25.6084926Z deleted: sha256:8c153bbd827867cdf94b1c4237be87df7196b4b4b2043e4eb08fffa5af35245d 2024-11-01T17:00:25.6085935Z deleted: sha256:0e72c2f47638416e5761d4a2b6723d0c2176a3232a9d65650d6f2774c1b1a7bb 2024-11-01T17:00:25.6086919Z deleted: sha256:1af9aaf89749d8b264269a0da14092f9bb6ec46db7db6ab6c9edb2b7eacc1e36 2024-11-01T17:00:25.6087921Z deleted: sha256:b7fe349ffe9a77b7cd8df359c0d4f447eab8c3c5fe1d1f8474d5421fae0b52e7 2024-11-01T17:00:25.6088940Z deleted: sha256:89d4a9d4808254c489a6774ebe1a7fbe7a921a48e2867b0370db5ae6643ef591 2024-11-01T17:00:25.6089923Z deleted: sha256:a3e0a41d4550f3243e4b20bdb204324bcbcea605d9ecd04e8f3f1af843f57e86 2024-11-01T17:00:25.6090909Z deleted: sha256:019e53db4314f4ba6148551c897d2d50cdceda1a96a4f03554208af0e99de4b8 2024-11-01T17:00:25.6091878Z deleted: sha256:383583432ea93d7b02c0844ca27319cab73bfc40c49d3f0e1e04295ccbcd9662 2024-11-01T17:00:25.6092867Z deleted: sha256:af4dadd55bfef20fdd6e7e3ce936c5930b3a4030c6dd380e41c57ac57fa6d351 2024-11-01T17:00:25.6093867Z deleted: sha256:df578b0221ae1fdd0a5b9b02c806ab3ee607208c6c0e8b75876caff585bc6177 2024-11-01T17:00:25.6094853Z deleted: sha256:29baf54d9e67bc5615a74704ba363598e29ae1314b34fa4ca5c843f2b5fb1265 2024-11-01T17:00:25.6095946Z deleted: sha256:7b401ef5b5879bdc37e726876c721fbbbe5f4f7721c236560a37cccb8f8ddb32 2024-11-01T17:00:25.6097131Z deleted: sha256:3c65fa1a83a01d453087f56b23a9cdda9218e1cdfe05500cbfcb1e753a3ab978 2024-11-01T17:00:25.6098378Z deleted: sha256:008280bac2ca227cafab11733314d6f662de2f70bb060483e181e5634e8a347f 2024-11-01T17:00:25.6099384Z deleted: sha256:5d4a60ee9e3f8449e187468813913fda8f5408c95b59600b5508ac77126c045e 2024-11-01T17:00:25.6100388Z deleted: sha256:f0b9826ef432a31e2e43af6d32313a2a4ae3557a3636f010eb150f5fb5dfbbff 2024-11-01T17:00:25.6101598Z deleted: sha256:7e4497566413d4a3599e7034506c24a6ab96193b4647a7c214f50295649f5291 2024-11-01T17:00:25.6102596Z deleted: sha256:930abad7c169e40236fa09a9608a0ba7e8e9a966686a6133b78bd660bdfb9a59 2024-11-01T17:00:25.6103586Z deleted: sha256:77f2520779046086794a6a4f24626f4f688c329e9cee7de2d22254477299c457 2024-11-01T17:00:25.6104568Z deleted: sha256:67e6ef1843c893be2880063dee3735bf00e40894ef866be07f61b8d59cf3755a 2024-11-01T17:00:25.6105563Z deleted: sha256:2ddf875758f00a6403c9e203e755cc185eca0f9e423849e5700fdb076dfc562d 2024-11-01T17:00:25.6106570Z deleted: sha256:68d0c60c82c5f5afa713dd4784be9baa1f588998b8e108ce0ece68f6e976d122 2024-11-01T17:00:25.6107585Z deleted: sha256:c3d898e584b844147299b6b3eaf2ea2cb63b4e93f3640a7b52baa64e770bb5f3 2024-11-01T17:00:25.6108587Z deleted: sha256:b46bb13fe8c4577e03ee179b7042226178e6cd61eaeb80445a16d25b1bb1f35f 2024-11-01T17:00:25.6109580Z deleted: sha256:42cc15f51fb1309300408cfaf0056e28b11de1a0f6dc033c99a943473619e1fb 2024-11-01T17:00:25.6110575Z deleted: sha256:4418b6a425adc3a7406a74e6b47bd7f3788ad275193db6716e23729b892ce0a8 2024-11-01T17:00:25.6111576Z deleted: sha256:7c58b3261a79180e66c7932cc232845b4dbdaffc908c0f0a2a2cde84950ad553 2024-11-01T17:00:25.6112597Z deleted: sha256:ea3f9a836caadfa9bd915005c979b0febc5334ae0a607daae43e85f77cc72474 2024-11-01T17:00:25.6113789Z deleted: sha256:decc9ca252ac120ee1858488ad90a3f707afd34238019bd1c51e97b4d09c45f6 2024-11-01T17:00:25.6114746Z deleted: sha256:e033e0e1b98c30aaad1bf6c6c2434254968b11262e9c6e46b56eb4ebdb74ab51 2024-11-01T17:00:25.6115698Z deleted: sha256:f8a7f2c4c34a3c04a982d753851c220e47a6f1049b57f2a42b4144d0b6a17b9e 2024-11-01T17:00:25.6116640Z deleted: sha256:88766d7f595185d3bc2ff7a2476931bda2cdac996c0d13db9a95b301b2db75a9 2024-11-01T17:00:25.6117735Z deleted: sha256:5bf533e94db3a248573dcbfdf6a58089e558f5f0f912277788bce3b54545d01c 2024-11-01T17:00:25.6118705Z deleted: sha256:8d77e0f9c2de0b3a7b5bdc2992ad1034b843b2eb67d9f262f43160f55884bab8 2024-11-01T17:00:25.6119749Z deleted: sha256:54bdbdf9f9e33647140f0c0f8c1dc00c526479f42964765338b8f79a1049ee9c 2024-11-01T17:00:25.6120647Z deleted: sha256:2359a12b67991edef98665db87a5192ba02e183ee2232e283e6b58c94f8d3826 2024-11-01T17:00:25.6121533Z deleted: sha256:7727c4d3952195e6aaf87b225a3f67b89642a424451e601618fa1e7b52325cd7 2024-11-01T17:00:25.6122405Z deleted: sha256:6f9f382dd9bcb89f97553031ee64253426902b4910f2556a2bea7881309c376c 2024-11-01T17:00:25.6123302Z deleted: sha256:c35ad5f713dc248c630d50c7520cd35ba1ab936116cf355e419c3fbe74909a04 2024-11-01T17:00:25.6124197Z deleted: sha256:4a5efa02522a748ca3722e8b7c0142d4d2291584b8d195e63da582862a967528 2024-11-01T17:00:25.6125086Z deleted: sha256:a30107d370bc5fa598ef737c0c53419acc51af7ab8ef64d75696fae37670abcb 2024-11-01T17:00:25.6125979Z deleted: sha256:672ec49fd713d9e0cdfb23310a314ebf467b0a33d23f64599816ca4684cb57f5 2024-11-01T17:00:25.6126880Z deleted: sha256:b4de36d9c64ae16e75798019674dcb2fade5e6653e5095c2dfd2846384786772 2024-11-01T17:00:25.6127782Z deleted: sha256:20d847bb7af881db42903f7d4166e9e372334ebdfb3d1f34b8ef669623d0da58 2024-11-01T17:00:25.6128662Z deleted: sha256:c34468395058a30a439e859a7bd5133e780faa762a11e025e0fdeb88627550e4 2024-11-01T17:00:25.6129539Z deleted: sha256:27f5680f66a0d01ec1d74ae03418fcb2e03db96605d55e0d8933a40acf0def6c 2024-11-01T17:00:25.6130431Z deleted: sha256:9a974b67ee0b37916f8a38a233be94f2fa8205ec477fdee5d06905af1cb34a9c 2024-11-01T17:00:25.6131307Z deleted: sha256:a7e98e6ff938d43862e03ca9edb22288389b8a503b60d8b72b26562de329e745 2024-11-01T17:00:25.6132196Z deleted: sha256:f8dfc84a271d94297e3a777f3455db814bebc4c56d31dc2b589320a70c5b3e01 2024-11-01T17:00:25.6133106Z deleted: sha256:cb9ee3913530d2683b184f99aaeaafade4ee059d8d83ca203fe890c9b71a39ce 2024-11-01T17:00:25.6134099Z deleted: sha256:cf8424e79315a3b2ffb71729abf912d62e257dfe818f3c47b09812733723a948 2024-11-01T17:00:25.6134998Z deleted: sha256:22de19c76bb02aa83af513603ea7b11c598a75ea3d120371286253e6ce67f0f0 2024-11-01T17:00:25.6135963Z deleted: sha256:e3bb91f1c5e44b1d35320ec7e5220af244c47d23a888f50508efbb919f0b7b29 2024-11-01T17:00:25.6137091Z deleted: sha256:08c5645ed9673839aa7a6eae9119ab5e407ecce85b3b5466607c5a651c3f2920 2024-11-01T17:00:25.6138113Z deleted: sha256:32c37789f4e3eaa19d0b7d08bfd1b4a1fed982cbeedccde408a3e3e2afaa1f9e 2024-11-01T17:00:25.6139149Z deleted: sha256:8b0aee0f411b6479db05dfa28910475b3cbbb2e08c3b048b78a43939c48c7419 2024-11-01T17:00:25.6140174Z deleted: sha256:2f7d309060abdcfff184b9086398b192de12a03af2a8e018c807cc6fa89ec403 2024-11-01T17:00:25.6141193Z deleted: sha256:fa738ffc39690ea6d3e51036aaa9acff34100a51d635e157817603eb79dee259 2024-11-01T17:00:25.6142204Z deleted: sha256:07a2164b82cf81c6f9f6fc6859a6d498203b5e0fb9ecd9878bf00de3c4c9949d 2024-11-01T17:00:25.6143198Z deleted: sha256:89edf8c906678c217b9a88160e1561991f53909c3fd27a4a5e1f0ca0f7ff7888 2024-11-01T17:00:25.6144219Z deleted: sha256:d5ba67f8f9b0c2b6da06d2d5d9d5fedc9f17cbe1a5e13a1a4033a4619f4528ba 2024-11-01T17:00:25.6145226Z deleted: sha256:5d61311e1a293c0c9642e34871719525dfeec7028e4b45e08344a469e3fe24bf 2024-11-01T17:00:25.6146216Z deleted: sha256:b89bb70f745a94c0edd5e974aee294e160735f76fd666a1a06a0b9b1be3caf63 2024-11-01T17:00:25.6147233Z deleted: sha256:9e5477f1f21a38c3dfe21a56a7d77d1b1a42453aed3e11729afcbb9d5731770b 2024-11-01T17:00:25.6148266Z deleted: sha256:b76f4af91cffe2bb7b3f09960f5bdbb26f741419defda65c451c4407fc5b0a92 2024-11-01T17:00:25.6149295Z deleted: sha256:3f8748a00ebd394c0e9d9d481972d764dff58896841ef0150b3f7efaac0f4a4a 2024-11-01T17:00:25.6150165Z deleted: sha256:13989c771e3ba4d40f618344e55e5cb7062a35041b5efd69e2f740ba819f46ee 2024-11-01T17:00:25.6151068Z deleted: sha256:9ebb27837f6c67ad8c17e9af4e63c008faae03fa226c3641dd8bbbfeefb106c4 2024-11-01T17:00:25.6151962Z deleted: sha256:8c16907d735950f8093582bc3cf392b70e002456bf9c4224de5a5b86b78b3ea2 2024-11-01T17:00:25.6152922Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-11-01T17:00:25.6153457Z 2024-11-01T17:00:25.6153615Z Total reclaimed space: 34.51GB 2024-11-01T17:00:25.6242307Z Post job cleanup. 2024-11-01T17:00:25.6300559Z Post job cleanup. 2024-11-01T17:00:25.7206802Z [command]/usr/bin/git version 2024-11-01T17:00:25.7254417Z git version 2.40.1 2024-11-01T17:00:25.7297805Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/7081a594-7e38-4187-81e2-5e1ffa30f7ff' before making global git config changes 2024-11-01T17:00:25.7299327Z Adding repository directory to the temporary git global config as a safe directory 2024-11-01T17:00:25.7301802Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-11-01T17:00:25.7342020Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-11-01T17:00:25.7378174Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-11-01T17:00:25.7701134Z Entering 'android/libs/fbjni' 2024-11-01T17:00:25.7758561Z Entering 'third_party/FP16' 2024-11-01T17:00:25.7817663Z Entering 'third_party/FXdiv' 2024-11-01T17:00:25.7876206Z Entering 'third_party/NNPACK' 2024-11-01T17:00:25.7933696Z Entering 'third_party/NVTX' 2024-11-01T17:00:25.7992840Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T17:00:25.8050864Z Entering 'third_party/XNNPACK' 2024-11-01T17:00:25.8128226Z Entering 'third_party/benchmark' 2024-11-01T17:00:25.8186477Z Entering 'third_party/composable_kernel' 2024-11-01T17:00:25.8250087Z Entering 'third_party/cpp-httplib' 2024-11-01T17:00:25.8307049Z Entering 'third_party/cpuinfo' 2024-11-01T17:00:25.8364006Z Entering 'third_party/cudnn_frontend' 2024-11-01T17:00:25.8418512Z Entering 'third_party/cutlass' 2024-11-01T17:00:25.8484429Z Entering 'third_party/eigen' 2024-11-01T17:00:25.8542695Z Entering 'third_party/fbgemm' 2024-11-01T17:00:25.8599706Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T17:00:25.8656156Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T17:00:25.8716689Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T17:00:25.8778312Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T17:00:25.8834940Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T17:00:25.8895244Z Entering 'third_party/flatbuffers' 2024-11-01T17:00:25.8957391Z Entering 'third_party/fmt' 2024-11-01T17:00:25.9015336Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T17:00:25.9073006Z Entering 'third_party/gloo' 2024-11-01T17:00:25.9130439Z Entering 'third_party/googletest' 2024-11-01T17:00:25.9192631Z Entering 'third_party/ideep' 2024-11-01T17:00:25.9249796Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T17:00:25.9323774Z Entering 'third_party/ittapi' 2024-11-01T17:00:25.9378641Z Entering 'third_party/kineto' 2024-11-01T17:00:25.9436494Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T17:00:25.9498076Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T17:00:25.9556286Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T17:00:25.9618196Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T17:00:25.9676039Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T17:00:25.9730332Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T17:00:25.9788433Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T17:00:25.9845544Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T17:00:25.9904313Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T17:00:25.9961996Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T17:00:26.0021183Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T17:00:26.0076958Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T17:00:26.0136346Z Entering 'third_party/mimalloc' 2024-11-01T17:00:26.0195914Z Entering 'third_party/nccl/nccl' 2024-11-01T17:00:26.0253677Z Entering 'third_party/nlohmann' 2024-11-01T17:00:26.0311921Z Entering 'third_party/onnx' 2024-11-01T17:00:26.0385202Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T17:00:26.0447886Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T17:00:26.0510169Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T17:00:26.0572548Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T17:00:26.0627051Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T17:00:26.0681721Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T17:00:26.0745972Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T17:00:26.0802669Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T17:00:26.0862072Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T17:00:26.0918225Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T17:00:26.0977991Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T17:00:26.1037962Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T17:00:26.1120394Z Entering 'third_party/pocketfft' 2024-11-01T17:00:26.1176705Z Entering 'third_party/protobuf' 2024-11-01T17:00:26.1237249Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T17:00:26.1297954Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T17:00:26.1360561Z Entering 'third_party/psimd' 2024-11-01T17:00:26.1416893Z Entering 'third_party/pthreadpool' 2024-11-01T17:00:26.1476258Z Entering 'third_party/pybind11' 2024-11-01T17:00:26.1535079Z Entering 'third_party/python-peachpy' 2024-11-01T17:00:26.1593936Z Entering 'third_party/sleef' 2024-11-01T17:00:26.1652229Z Entering 'third_party/tensorpipe' 2024-11-01T17:00:26.1707683Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T17:00:26.1764661Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T17:00:26.1819850Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T17:00:26.1875401Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T17:00:26.1930803Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T17:00:26.2013818Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-11-01T17:00:26.2042735Z http.https://github.com/.extraheader 2024-11-01T17:00:26.2050690Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-11-01T17:00:26.2086420Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-11-01T17:00:26.2406270Z Entering 'android/libs/fbjni' 2024-11-01T17:00:26.2446027Z http.https://github.com/.extraheader 2024-11-01T17:00:26.2481623Z Entering 'third_party/FP16' 2024-11-01T17:00:26.2521888Z http.https://github.com/.extraheader 2024-11-01T17:00:26.2557330Z Entering 'third_party/FXdiv' 2024-11-01T17:00:26.2596003Z http.https://github.com/.extraheader 2024-11-01T17:00:26.2633277Z Entering 'third_party/NNPACK' 2024-11-01T17:00:26.2671738Z http.https://github.com/.extraheader 2024-11-01T17:00:26.2707650Z Entering 'third_party/NVTX' 2024-11-01T17:00:26.2745396Z http.https://github.com/.extraheader 2024-11-01T17:00:26.2779902Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T17:00:26.2818450Z http.https://github.com/.extraheader 2024-11-01T17:00:26.2856841Z Entering 'third_party/XNNPACK' 2024-11-01T17:00:26.2894139Z http.https://github.com/.extraheader 2024-11-01T17:00:26.2946534Z Entering 'third_party/benchmark' 2024-11-01T17:00:26.2983738Z http.https://github.com/.extraheader 2024-11-01T17:00:26.3020524Z Entering 'third_party/composable_kernel' 2024-11-01T17:00:26.3058359Z http.https://github.com/.extraheader 2024-11-01T17:00:26.3100073Z Entering 'third_party/cpp-httplib' 2024-11-01T17:00:26.3138107Z http.https://github.com/.extraheader 2024-11-01T17:00:26.3175348Z Entering 'third_party/cpuinfo' 2024-11-01T17:00:26.3213244Z http.https://github.com/.extraheader 2024-11-01T17:00:26.3251867Z Entering 'third_party/cudnn_frontend' 2024-11-01T17:00:26.3289369Z http.https://github.com/.extraheader 2024-11-01T17:00:26.3327466Z Entering 'third_party/cutlass' 2024-11-01T17:00:26.3367193Z http.https://github.com/.extraheader 2024-11-01T17:00:26.3411067Z Entering 'third_party/eigen' 2024-11-01T17:00:26.3450237Z http.https://github.com/.extraheader 2024-11-01T17:00:26.3490552Z Entering 'third_party/fbgemm' 2024-11-01T17:00:26.3530090Z http.https://github.com/.extraheader 2024-11-01T17:00:26.3567104Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T17:00:26.3603969Z http.https://github.com/.extraheader 2024-11-01T17:00:26.3640562Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T17:00:26.3678648Z http.https://github.com/.extraheader 2024-11-01T17:00:26.3716311Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T17:00:26.3753431Z http.https://github.com/.extraheader 2024-11-01T17:00:26.3798393Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T17:00:26.3837265Z http.https://github.com/.extraheader 2024-11-01T17:00:26.3875375Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T17:00:26.3910802Z http.https://github.com/.extraheader 2024-11-01T17:00:26.3947910Z Entering 'third_party/flatbuffers' 2024-11-01T17:00:26.3987276Z http.https://github.com/.extraheader 2024-11-01T17:00:26.4026192Z Entering 'third_party/fmt' 2024-11-01T17:00:26.4063862Z http.https://github.com/.extraheader 2024-11-01T17:00:26.4099429Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T17:00:26.4138040Z http.https://github.com/.extraheader 2024-11-01T17:00:26.4173861Z Entering 'third_party/gloo' 2024-11-01T17:00:26.4213604Z http.https://github.com/.extraheader 2024-11-01T17:00:26.4257574Z Entering 'third_party/googletest' 2024-11-01T17:00:26.4294858Z http.https://github.com/.extraheader 2024-11-01T17:00:26.4331089Z Entering 'third_party/ideep' 2024-11-01T17:00:26.4368650Z http.https://github.com/.extraheader 2024-11-01T17:00:26.4402865Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T17:00:26.4440633Z http.https://github.com/.extraheader 2024-11-01T17:00:26.4485823Z Entering 'third_party/ittapi' 2024-11-01T17:00:26.4524961Z http.https://github.com/.extraheader 2024-11-01T17:00:26.4560102Z Entering 'third_party/kineto' 2024-11-01T17:00:26.4598424Z http.https://github.com/.extraheader 2024-11-01T17:00:26.4635791Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T17:00:26.4674855Z http.https://github.com/.extraheader 2024-11-01T17:00:26.4709803Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T17:00:26.4746655Z http.https://github.com/.extraheader 2024-11-01T17:00:26.4787786Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T17:00:26.4826131Z http.https://github.com/.extraheader 2024-11-01T17:00:26.4861718Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T17:00:26.4898026Z http.https://github.com/.extraheader 2024-11-01T17:00:26.4937950Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T17:00:26.4975307Z http.https://github.com/.extraheader 2024-11-01T17:00:26.5017853Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T17:00:26.5054855Z http.https://github.com/.extraheader 2024-11-01T17:00:26.5099328Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T17:00:26.5137583Z http.https://github.com/.extraheader 2024-11-01T17:00:26.5177325Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T17:00:26.5215072Z http.https://github.com/.extraheader 2024-11-01T17:00:26.5252988Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T17:00:26.5289994Z http.https://github.com/.extraheader 2024-11-01T17:00:26.5328090Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T17:00:26.5368746Z http.https://github.com/.extraheader 2024-11-01T17:00:26.5404604Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T17:00:26.5443638Z http.https://github.com/.extraheader 2024-11-01T17:00:26.5478059Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T17:00:26.5516255Z http.https://github.com/.extraheader 2024-11-01T17:00:26.5558497Z Entering 'third_party/mimalloc' 2024-11-01T17:00:26.5597181Z http.https://github.com/.extraheader 2024-11-01T17:00:26.5635852Z Entering 'third_party/nccl/nccl' 2024-11-01T17:00:26.5673484Z http.https://github.com/.extraheader 2024-11-01T17:00:26.5709531Z Entering 'third_party/nlohmann' 2024-11-01T17:00:26.5746058Z http.https://github.com/.extraheader 2024-11-01T17:00:26.5788083Z Entering 'third_party/onnx' 2024-11-01T17:00:26.5830634Z http.https://github.com/.extraheader 2024-11-01T17:00:26.5887484Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T17:00:26.5925332Z http.https://github.com/.extraheader 2024-11-01T17:00:26.5963640Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T17:00:26.6001234Z http.https://github.com/.extraheader 2024-11-01T17:00:26.6039029Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T17:00:26.6075721Z http.https://github.com/.extraheader 2024-11-01T17:00:26.6119789Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T17:00:26.6157120Z http.https://github.com/.extraheader 2024-11-01T17:00:26.6196601Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T17:00:26.6235054Z http.https://github.com/.extraheader 2024-11-01T17:00:26.6275727Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T17:00:26.6311197Z http.https://github.com/.extraheader 2024-11-01T17:00:26.6350764Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T17:00:26.6386055Z http.https://github.com/.extraheader 2024-11-01T17:00:26.6419864Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T17:00:26.6456959Z http.https://github.com/.extraheader 2024-11-01T17:00:26.6498145Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T17:00:26.6536093Z http.https://github.com/.extraheader 2024-11-01T17:00:26.6576671Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T17:00:26.6612430Z http.https://github.com/.extraheader 2024-11-01T17:00:26.6655025Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T17:00:26.6691537Z http.https://github.com/.extraheader 2024-11-01T17:00:26.6729917Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T17:00:26.6766892Z http.https://github.com/.extraheader 2024-11-01T17:00:26.6824102Z Entering 'third_party/pocketfft' 2024-11-01T17:00:26.6861048Z http.https://github.com/.extraheader 2024-11-01T17:00:26.6898442Z Entering 'third_party/protobuf' 2024-11-01T17:00:26.6937564Z http.https://github.com/.extraheader 2024-11-01T17:00:26.6976293Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T17:00:26.7012283Z http.https://github.com/.extraheader 2024-11-01T17:00:26.7057835Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T17:00:26.7094349Z http.https://github.com/.extraheader 2024-11-01T17:00:26.7133289Z Entering 'third_party/psimd' 2024-11-01T17:00:26.7170661Z http.https://github.com/.extraheader 2024-11-01T17:00:26.7205443Z Entering 'third_party/pthreadpool' 2024-11-01T17:00:26.7244848Z http.https://github.com/.extraheader 2024-11-01T17:00:26.7279061Z Entering 'third_party/pybind11' 2024-11-01T17:00:26.7318190Z http.https://github.com/.extraheader 2024-11-01T17:00:26.7357056Z Entering 'third_party/python-peachpy' 2024-11-01T17:00:26.7394352Z http.https://github.com/.extraheader 2024-11-01T17:00:26.7430128Z Entering 'third_party/sleef' 2024-11-01T17:00:26.7468714Z http.https://github.com/.extraheader 2024-11-01T17:00:26.7507054Z Entering 'third_party/tensorpipe' 2024-11-01T17:00:26.7546680Z http.https://github.com/.extraheader 2024-11-01T17:00:26.7580831Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T17:00:26.7617680Z http.https://github.com/.extraheader 2024-11-01T17:00:26.7656153Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T17:00:26.7692675Z http.https://github.com/.extraheader 2024-11-01T17:00:26.7728439Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T17:00:26.7765830Z http.https://github.com/.extraheader 2024-11-01T17:00:26.7800528Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T17:00:26.7839500Z http.https://github.com/.extraheader 2024-11-01T17:00:26.7875136Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T17:00:26.7911100Z http.https://github.com/.extraheader 2024-11-01T17:00:26.8043072Z A job completed hook has been configured by the self-hosted runner administrator 2024-11-01T17:00:26.8065394Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-11-01T17:00:26.8071504Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:00:26.8072036Z ##[endgroup] 2024-11-01T17:00:33.5158727Z Cleaning up orphan processes