2024-08-20T21:47:47.2990971Z Current runner version: '2.319.1' 2024-08-20T21:47:47.2999803Z Runner name: 'i-0cc1c134a48939331' 2024-08-20T21:47:47.3001017Z Runner group name: 'Default' 2024-08-20T21:47:47.3002049Z Machine name: 'ip-10-0-2-62' 2024-08-20T21:47:47.3025108Z Testing runner upgrade compatibility 2024-08-20T21:47:47.4176704Z ##[group]GITHUB_TOKEN Permissions 2024-08-20T21:47:47.4179331Z Actions: read 2024-08-20T21:47:47.4179965Z Attestations: read 2024-08-20T21:47:47.4180743Z Checks: read 2024-08-20T21:47:47.4181348Z Contents: read 2024-08-20T21:47:47.4181914Z Deployments: read 2024-08-20T21:47:47.4182607Z Discussions: read 2024-08-20T21:47:47.4183236Z Issues: read 2024-08-20T21:47:47.4183764Z Metadata: read 2024-08-20T21:47:47.4184502Z Packages: read 2024-08-20T21:47:47.4185116Z Pages: read 2024-08-20T21:47:47.4185638Z PullRequests: read 2024-08-20T21:47:47.4186354Z RepositoryProjects: read 2024-08-20T21:47:47.4187091Z SecurityEvents: read 2024-08-20T21:47:47.4187682Z Statuses: read 2024-08-20T21:47:47.4188604Z ##[endgroup] 2024-08-20T21:47:47.4193046Z Secret source: Actions 2024-08-20T21:47:47.4194198Z Prepare workflow directory 2024-08-20T21:47:47.9118120Z Prepare all required actions 2024-08-20T21:47:47.9349138Z Getting action download info 2024-08-20T21:47:48.1509879Z Download action repository 'pytorch/test-infra@main' (SHA:0c3a2634aaa2f638c8f640e743f03d696ce1191f) 2024-08-20T21:47:48.6270711Z Download action repository 'pytorch/pytorch@main' (SHA:2e1830c7c8dfc1141c89232e0356671ac07aa979) 2024-08-20T21:47:52.8979277Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-20T21:47:53.0375246Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-20T21:47:53.4222322Z Getting action download info 2024-08-20T21:47:53.5261735Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-20T21:47:53.7491038Z Getting action download info 2024-08-20T21:47:53.8352225Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-20T21:47:53.9960264Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/133712/merge (f2fb9405c2fa9f9502a76363091cce6fd8179736) 2024-08-20T21:47:53.9963737Z ##[group] Inputs 2024-08-20T21:47:53.9964320Z build-environment: linux-focal-cuda11.8-py3.10-gcc9 2024-08-20T21:47:53.9966570Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}]} 2024-08-20T21:47:53.9970083Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:47:53.9971448Z sync-tag: 2024-08-20T21:47:53.9972674Z timeout-minutes: 360 2024-08-20T21:47:53.9973108Z use-gha: 2024-08-20T21:47:53.9973442Z dashboard-tag: 2024-08-20T21:47:53.9973830Z s3-bucket: gha-artifacts 2024-08-20T21:47:53.9974267Z aws-role-to-assume: 2024-08-20T21:47:53.9974668Z ##[endgroup] 2024-08-20T21:47:53.9975989Z Complete job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, amz2023.linux.8xlarge.nvidia.gpu) 2024-08-20T21:47:54.0972100Z A job started hook has been configured by the self-hosted runner administrator 2024-08-20T21:47:54.1170743Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-20T21:47:54.1182621Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:47:54.1183323Z ##[endgroup] 2024-08-20T21:47:56.0102013Z Runner Type: amz2023.linux.8xlarge.nvidia.gpu 2024-08-20T21:47:56.0102674Z Instance Type: g3.8xlarge 2024-08-20T21:47:56.0103554Z AMI Name: al2023-ami-2023.5.20240701.0-kernel-6.1-x86_64 2024-08-20T21:47:56.0104284Z AMI ID: ami-06c68f701d8090592 2024-08-20T21:48:03.1806406Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-20T21:48:03.1807202Z with: 2024-08-20T21:48:03.1808074Z github-secret: *** 2024-08-20T21:48:03.1809287Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T21:48:03.1810567Z activate-with-label: false 2024-08-20T21:48:03.1811017Z label: with-ssh 2024-08-20T21:48:03.1811394Z remove-existing-keys: true 2024-08-20T21:48:03.1811846Z fail-silently: true 2024-08-20T21:48:03.1812235Z env: 2024-08-20T21:48:03.1812553Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:48:03.1812980Z ##[endgroup] 2024-08-20T21:48:03.2994681Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-20T21:48:03.6630622Z Grabbing public ssh keys from https://github.com/pytorchmergebot.keys 2024-08-20T21:48:03.7363760Z No SSH keys found for user pytorchmergebot 2024-08-20T21:48:03.7364941Z Grabbing public ssh keys from https://github.com/XuehaiPan.keys 2024-08-20T21:48:03.8034296Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-20T21:48:03.8056453Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-20T21:48:03.8097276Z Login using: ssh ec2-user@ec2-3-81-48-61.compute-1.amazonaws.com 2024-08-20T21:48:03.8098704Z All testing is done inside the container, to start an interactive session run: 2024-08-20T21:48:03.8099790Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T21:48:03.8245543Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-20T21:48:03.8246241Z with: 2024-08-20T21:48:03.8246578Z submodules: recursive 2024-08-20T21:48:03.8246993Z fetch-depth: 0 2024-08-20T21:48:03.8247328Z env: 2024-08-20T21:48:03.8247655Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:48:03.8248076Z ##[endgroup] 2024-08-20T21:48:03.8511945Z ##[group]Run retry () { 2024-08-20T21:48:03.8512423Z retry () { 2024-08-20T21:48:03.8513046Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-20T21:48:03.8513742Z } 2024-08-20T21:48:03.8514100Z echo "${GITHUB_WORKSPACE}" 2024-08-20T21:48:03.8514614Z if [ -z "${NO_SUDO}" ]; then 2024-08-20T21:48:03.8515207Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T21:48:03.8515752Z else 2024-08-20T21:48:03.8516170Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T21:48:03.8516700Z fi 2024-08-20T21:48:03.8517119Z mkdir "${GITHUB_WORKSPACE}" 2024-08-20T21:48:03.8525908Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:48:03.8526553Z env: 2024-08-20T21:48:03.8526888Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:48:03.8527292Z NO_SUDO: 2024-08-20T21:48:03.8527627Z ##[endgroup] 2024-08-20T21:48:03.8558948Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T21:48:07.3375156Z ##[group]Run malfet/checkout@silent-checkout 2024-08-20T21:48:07.3375708Z with: 2024-08-20T21:48:07.3376092Z ref: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:48:07.3376644Z fetch-depth: 0 2024-08-20T21:48:07.3377026Z submodules: recursive 2024-08-20T21:48:07.3377424Z quiet-checkout: true 2024-08-20T21:48:07.3377848Z repository: pytorch/pytorch 2024-08-20T21:48:07.3378461Z token: *** 2024-08-20T21:48:07.3378799Z ssh-strict: true 2024-08-20T21:48:07.3379196Z persist-credentials: true 2024-08-20T21:48:07.3379634Z clean: true 2024-08-20T21:48:07.3380018Z sparse-checkout-cone-mode: true 2024-08-20T21:48:07.3380493Z lfs: false 2024-08-20T21:48:07.3380856Z set-safe-directory: true 2024-08-20T21:48:07.3381253Z env: 2024-08-20T21:48:07.3381575Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:48:07.3381995Z ##[endgroup] 2024-08-20T21:48:07.4705149Z Syncing repository: pytorch/pytorch 2024-08-20T21:48:07.4707471Z ##[group]Getting Git version info 2024-08-20T21:48:07.4708408Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T21:48:07.4709821Z [command]/usr/bin/git version 2024-08-20T21:48:07.4710273Z git version 2.40.1 2024-08-20T21:48:07.4719898Z ##[endgroup] 2024-08-20T21:48:07.4742540Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/a789acfe-b4b7-431f-8996-6a1a12a7efb6' before making global git config changes 2024-08-20T21:48:07.4744136Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T21:48:07.4750144Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T21:48:07.4799818Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T21:48:07.4806232Z ##[group]Initializing the repository 2024-08-20T21:48:07.4810577Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T21:48:07.4847311Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-20T21:48:07.4848585Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-20T21:48:07.4849584Z hint: of your new repositories, which will suppress this warning, call: 2024-08-20T21:48:07.4850263Z hint: 2024-08-20T21:48:07.4850816Z hint: git config --global init.defaultBranch 2024-08-20T21:48:07.4851391Z hint: 2024-08-20T21:48:07.4852043Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-20T21:48:07.4853113Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-20T21:48:07.4854087Z hint: 2024-08-20T21:48:07.4854502Z hint: git branch -m 2024-08-20T21:48:07.4855446Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-20T21:48:07.4863713Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-20T21:48:07.4900266Z ##[endgroup] 2024-08-20T21:48:07.4900968Z ##[group]Disabling automatic garbage collection 2024-08-20T21:48:07.4905153Z [command]/usr/bin/git config --local gc.auto 0 2024-08-20T21:48:07.4941544Z ##[endgroup] 2024-08-20T21:48:07.4942166Z ##[group]Setting up auth 2024-08-20T21:48:07.4951730Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T21:48:07.4992468Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T21:48:07.5315890Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T21:48:07.5350989Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T21:48:07.5672155Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T21:48:07.5725097Z ##[endgroup] 2024-08-20T21:48:07.5725753Z ##[group]Fetching the repository 2024-08-20T21:48:07.5734923Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-20T21:48:11.1906075Z remote: Enumerating objects: 1008510 2024-08-20T21:48:11.1906764Z remote: Enumerating objects: 1010161, done. 2024-08-20T21:48:11.1908389Z remote: Counting objects: 0% (1/1651) 2024-08-20T21:48:11.1909020Z remote: Counting objects: 1% (17/1651) 2024-08-20T21:48:11.1909667Z remote: Counting objects: 2% (34/1651) 2024-08-20T21:48:11.1910293Z remote: Counting objects: 3% (50/1651) 2024-08-20T21:48:11.1910940Z remote: Counting objects: 4% (67/1651) 2024-08-20T21:48:11.1911555Z remote: Counting objects: 5% (83/1651) 2024-08-20T21:48:11.1912162Z remote: Counting objects: 6% (100/1651) 2024-08-20T21:48:11.1912793Z remote: Counting objects: 7% (116/1651) 2024-08-20T21:48:11.1913787Z remote: Counting objects: 8% (133/1651) 2024-08-20T21:48:11.1914395Z remote: Counting objects: 9% (149/1651) 2024-08-20T21:48:11.1915024Z remote: Counting objects: 10% (166/1651) 2024-08-20T21:48:11.1915638Z remote: Counting objects: 11% (182/1651) 2024-08-20T21:48:11.1916238Z remote: Counting objects: 12% (199/1651) 2024-08-20T21:48:11.1916865Z remote: Counting objects: 13% (215/1651) 2024-08-20T21:48:11.1917486Z remote: Counting objects: 14% (232/1651) 2024-08-20T21:48:11.1918092Z remote: Counting objects: 15% (248/1651) 2024-08-20T21:48:11.1918709Z remote: Counting objects: 16% (265/1651) 2024-08-20T21:48:11.1919329Z remote: Counting objects: 17% (281/1651) 2024-08-20T21:48:11.1919945Z remote: Counting objects: 18% (298/1651) 2024-08-20T21:48:11.1920591Z remote: Counting objects: 19% (314/1651) 2024-08-20T21:48:11.1921206Z remote: Counting objects: 20% (331/1651) 2024-08-20T21:48:11.1921834Z remote: Counting objects: 21% (347/1651) 2024-08-20T21:48:11.1922432Z remote: Counting objects: 22% (364/1651) 2024-08-20T21:48:11.1923050Z remote: Counting objects: 23% (380/1651) 2024-08-20T21:48:11.1923663Z remote: Counting objects: 24% (397/1651) 2024-08-20T21:48:11.1924258Z remote: Counting objects: 25% (413/1651) 2024-08-20T21:48:11.1924876Z remote: Counting objects: 26% (430/1651) 2024-08-20T21:48:11.1925491Z remote: Counting objects: 27% (446/1651) 2024-08-20T21:48:11.1926261Z remote: Counting objects: 28% (463/1651) 2024-08-20T21:48:11.1926904Z remote: Counting objects: 29% (479/1651) 2024-08-20T21:48:11.1927520Z remote: Counting objects: 30% (496/1651) 2024-08-20T21:48:11.1928122Z remote: Counting objects: 31% (512/1651) 2024-08-20T21:48:11.1928738Z remote: Counting objects: 32% (529/1651) 2024-08-20T21:48:11.1929355Z remote: Counting objects: 33% (545/1651) 2024-08-20T21:48:11.1930002Z remote: Counting objects: 34% (562/1651) 2024-08-20T21:48:11.1930599Z remote: Counting objects: 35% (578/1651) 2024-08-20T21:48:11.1931214Z remote: Counting objects: 36% (595/1651) 2024-08-20T21:48:11.1931831Z remote: Counting objects: 37% (611/1651) 2024-08-20T21:48:11.1932423Z remote: Counting objects: 38% (628/1651) 2024-08-20T21:48:11.1933040Z remote: Counting objects: 39% (644/1651) 2024-08-20T21:48:11.1933657Z remote: Counting objects: 40% (661/1651) 2024-08-20T21:48:11.1934256Z remote: Counting objects: 41% (677/1651) 2024-08-20T21:48:11.1934873Z remote: Counting objects: 42% (694/1651) 2024-08-20T21:48:11.1935486Z remote: Counting objects: 43% (710/1651) 2024-08-20T21:48:11.1936081Z remote: Counting objects: 44% (727/1651) 2024-08-20T21:48:11.1936694Z remote: Counting objects: 45% (743/1651) 2024-08-20T21:48:11.1937309Z remote: Counting objects: 46% (760/1651) 2024-08-20T21:48:11.1937897Z remote: Counting objects: 47% (776/1651) 2024-08-20T21:48:11.1938507Z remote: Counting objects: 48% (793/1651) 2024-08-20T21:48:11.1939121Z remote: Counting objects: 49% (809/1651) 2024-08-20T21:48:11.1939731Z remote: Counting objects: 50% (826/1651) 2024-08-20T21:48:11.1940326Z remote: Counting objects: 51% (843/1651) 2024-08-20T21:48:11.1940942Z remote: Counting objects: 52% (859/1651) 2024-08-20T21:48:11.1941554Z remote: Counting objects: 53% (876/1651) 2024-08-20T21:48:11.1942154Z remote: Counting objects: 54% (892/1651) 2024-08-20T21:48:11.1942767Z remote: Counting objects: 55% (909/1651) 2024-08-20T21:48:11.1943377Z remote: Counting objects: 56% (925/1651) 2024-08-20T21:48:11.1943970Z remote: Counting objects: 57% (942/1651) 2024-08-20T21:48:11.1944586Z remote: Counting objects: 58% (958/1651) 2024-08-20T21:48:11.1945311Z remote: Counting objects: 59% (975/1651) 2024-08-20T21:48:11.1945902Z remote: Counting objects: 60% (991/1651) 2024-08-20T21:48:11.1946523Z remote: Counting objects: 61% (1008/1651) 2024-08-20T21:48:11.1947168Z remote: Counting objects: 62% (1024/1651) 2024-08-20T21:48:11.1947819Z remote: Counting objects: 63% (1041/1651) 2024-08-20T21:48:11.1948443Z remote: Counting objects: 64% (1057/1651) 2024-08-20T21:48:11.1949078Z remote: Counting objects: 65% (1074/1651) 2024-08-20T21:48:11.1949724Z remote: Counting objects: 66% (1090/1651) 2024-08-20T21:48:11.1950343Z remote: Counting objects: 67% (1107/1651) 2024-08-20T21:48:11.1950976Z remote: Counting objects: 68% (1123/1651) 2024-08-20T21:48:11.1951613Z remote: Counting objects: 69% (1140/1651) 2024-08-20T21:48:11.1952231Z remote: Counting objects: 70% (1156/1651) 2024-08-20T21:48:11.1952874Z remote: Counting objects: 71% (1173/1651) 2024-08-20T21:48:11.1953514Z remote: Counting objects: 72% (1189/1651) 2024-08-20T21:48:11.1954146Z remote: Counting objects: 73% (1206/1651) 2024-08-20T21:48:11.1954785Z remote: Counting objects: 74% (1222/1651) 2024-08-20T21:48:11.1955421Z remote: Counting objects: 75% (1239/1651) 2024-08-20T21:48:11.1956056Z remote: Counting objects: 76% (1255/1651) 2024-08-20T21:48:11.1956673Z remote: Counting objects: 77% (1272/1651) 2024-08-20T21:48:11.1957302Z remote: Counting objects: 78% (1288/1651) 2024-08-20T21:48:11.1958025Z remote: Counting objects: 79% (1305/1651) 2024-08-20T21:48:11.1958659Z remote: Counting objects: 80% (1321/1651) 2024-08-20T21:48:11.1959289Z remote: Counting objects: 81% (1338/1651) 2024-08-20T21:48:11.1959925Z remote: Counting objects: 82% (1354/1651) 2024-08-20T21:48:11.1960539Z remote: Counting objects: 83% (1371/1651) 2024-08-20T21:48:11.1961173Z remote: Counting objects: 84% (1387/1651) 2024-08-20T21:48:11.1961812Z remote: Counting objects: 85% (1404/1651) 2024-08-20T21:48:11.1962441Z remote: Counting objects: 86% (1420/1651) 2024-08-20T21:48:11.1963059Z remote: Counting objects: 87% (1437/1651) 2024-08-20T21:48:11.1963687Z remote: Counting objects: 88% (1453/1651) 2024-08-20T21:48:11.1964314Z remote: Counting objects: 89% (1470/1651) 2024-08-20T21:48:11.1964933Z remote: Counting objects: 90% (1486/1651) 2024-08-20T21:48:11.1965566Z remote: Counting objects: 91% (1503/1651) 2024-08-20T21:48:11.1966197Z remote: Counting objects: 92% (1519/1651) 2024-08-20T21:48:11.1966814Z remote: Counting objects: 93% (1536/1651) 2024-08-20T21:48:11.1967445Z remote: Counting objects: 94% (1552/1651) 2024-08-20T21:48:11.1968072Z remote: Counting objects: 95% (1569/1651) 2024-08-20T21:48:11.1968683Z remote: Counting objects: 96% (1585/1651) 2024-08-20T21:48:11.1969711Z remote: Counting objects: 97% (1602/1651) 2024-08-20T21:48:11.1970356Z remote: Counting objects: 98% (1618/1651) 2024-08-20T21:48:11.1970971Z remote: Counting objects: 99% (1635/1651) 2024-08-20T21:48:11.1971605Z remote: Counting objects: 100% (1651/1651) 2024-08-20T21:48:11.1972282Z remote: Counting objects: 100% (1651/1651), done. 2024-08-20T21:48:11.2151228Z remote: Compressing objects: 0% (1/846) 2024-08-20T21:48:11.2516616Z remote: Compressing objects: 1% (9/846) 2024-08-20T21:48:11.2877458Z remote: Compressing objects: 2% (17/846) 2024-08-20T21:48:11.3818332Z remote: Compressing objects: 3% (26/846) 2024-08-20T21:48:11.4664213Z remote: Compressing objects: 4% (34/846) 2024-08-20T21:48:11.5148008Z remote: Compressing objects: 5% (43/846) 2024-08-20T21:48:11.5490134Z remote: Compressing objects: 6% (51/846) 2024-08-20T21:48:11.5857180Z remote: Compressing objects: 7% (60/846) 2024-08-20T21:48:11.6309241Z remote: Compressing objects: 8% (68/846) 2024-08-20T21:48:11.6581222Z remote: Compressing objects: 9% (77/846) 2024-08-20T21:48:11.6903074Z remote: Compressing objects: 10% (85/846) 2024-08-20T21:48:11.7132540Z remote: Compressing objects: 11% (94/846) 2024-08-20T21:48:11.7355791Z remote: Compressing objects: 12% (102/846) 2024-08-20T21:48:11.7547726Z remote: Compressing objects: 13% (110/846) 2024-08-20T21:48:11.7651156Z remote: Compressing objects: 14% (119/846) 2024-08-20T21:48:11.7769103Z remote: Compressing objects: 15% (127/846) 2024-08-20T21:48:11.7861561Z remote: Compressing objects: 16% (136/846) 2024-08-20T21:48:11.7952687Z remote: Compressing objects: 17% (144/846) 2024-08-20T21:48:11.8020258Z remote: Compressing objects: 18% (153/846) 2024-08-20T21:48:11.8062545Z remote: Compressing objects: 19% (161/846) 2024-08-20T21:48:11.8091295Z remote: Compressing objects: 20% (170/846) 2024-08-20T21:48:11.8124273Z remote: Compressing objects: 21% (178/846) 2024-08-20T21:48:11.8127255Z remote: Compressing objects: 22% (187/846) 2024-08-20T21:48:11.8132124Z remote: Compressing objects: 23% (195/846) 2024-08-20T21:48:11.8140205Z remote: Compressing objects: 24% (204/846) 2024-08-20T21:48:11.8144048Z remote: Compressing objects: 25% (212/846) 2024-08-20T21:48:11.8151008Z remote: Compressing objects: 26% (220/846) 2024-08-20T21:48:11.8160253Z remote: Compressing objects: 27% (229/846) 2024-08-20T21:48:11.8174701Z remote: Compressing objects: 28% (237/846) 2024-08-20T21:48:11.8193307Z remote: Compressing objects: 29% (246/846) 2024-08-20T21:48:11.8196049Z remote: Compressing objects: 30% (254/846) 2024-08-20T21:48:11.8215615Z remote: Compressing objects: 31% (263/846) 2024-08-20T21:48:11.8219748Z remote: Compressing objects: 32% (271/846) 2024-08-20T21:48:11.8228918Z remote: Compressing objects: 33% (280/846) 2024-08-20T21:48:11.8238954Z remote: Compressing objects: 34% (288/846) 2024-08-20T21:48:11.8244546Z remote: Compressing objects: 35% (297/846) 2024-08-20T21:48:11.8251487Z remote: Compressing objects: 36% (305/846) 2024-08-20T21:48:11.8260472Z remote: Compressing objects: 37% (314/846) 2024-08-20T21:48:11.8267781Z remote: Compressing objects: 38% (322/846) 2024-08-20T21:48:11.8277233Z remote: Compressing objects: 39% (330/846) 2024-08-20T21:48:11.8281331Z remote: Compressing objects: 40% (339/846) 2024-08-20T21:48:11.8288718Z remote: Compressing objects: 41% (347/846) 2024-08-20T21:48:11.8294478Z remote: Compressing objects: 42% (356/846) 2024-08-20T21:48:11.8301151Z remote: Compressing objects: 43% (364/846) 2024-08-20T21:48:11.8306233Z remote: Compressing objects: 44% (373/846) 2024-08-20T21:48:11.8316864Z remote: Compressing objects: 45% (381/846) 2024-08-20T21:48:11.8319564Z remote: Compressing objects: 46% (390/846) 2024-08-20T21:48:11.8325549Z remote: Compressing objects: 47% (398/846) 2024-08-20T21:48:11.8330159Z remote: Compressing objects: 48% (407/846) 2024-08-20T21:48:11.8332886Z remote: Compressing objects: 49% (415/846) 2024-08-20T21:48:11.8337113Z remote: Compressing objects: 50% (423/846) 2024-08-20T21:48:11.8339796Z remote: Compressing objects: 51% (432/846) 2024-08-20T21:48:11.8342734Z remote: Compressing objects: 52% (440/846) 2024-08-20T21:48:11.8345920Z remote: Compressing objects: 53% (449/846) 2024-08-20T21:48:11.8348803Z remote: Compressing objects: 54% (457/846) 2024-08-20T21:48:11.8354654Z remote: Compressing objects: 55% (466/846) 2024-08-20T21:48:11.8355536Z remote: Compressing objects: 56% (474/846) 2024-08-20T21:48:11.8356204Z remote: Compressing objects: 57% (483/846) 2024-08-20T21:48:11.8356861Z remote: Compressing objects: 58% (491/846) 2024-08-20T21:48:11.8357723Z remote: Compressing objects: 59% (500/846) 2024-08-20T21:48:11.8358393Z remote: Compressing objects: 60% (508/846) 2024-08-20T21:48:11.8359054Z remote: Compressing objects: 61% (517/846) 2024-08-20T21:48:11.8359701Z remote: Compressing objects: 62% (525/846) 2024-08-20T21:48:11.8360358Z remote: Compressing objects: 63% (533/846) 2024-08-20T21:48:11.8361017Z remote: Compressing objects: 64% (542/846) 2024-08-20T21:48:11.8361670Z remote: Compressing objects: 65% (550/846) 2024-08-20T21:48:11.8367697Z remote: Compressing objects: 66% (559/846) 2024-08-20T21:48:11.8377074Z remote: Compressing objects: 67% (567/846) 2024-08-20T21:48:11.8378524Z remote: Compressing objects: 68% (576/846) 2024-08-20T21:48:11.8384354Z remote: Compressing objects: 69% (584/846) 2024-08-20T21:48:11.8387915Z remote: Compressing objects: 70% (593/846) 2024-08-20T21:48:11.8392422Z remote: Compressing objects: 71% (601/846) 2024-08-20T21:48:11.8395526Z remote: Compressing objects: 72% (610/846) 2024-08-20T21:48:11.8399570Z remote: Compressing objects: 73% (618/846) 2024-08-20T21:48:11.8403947Z remote: Compressing objects: 74% (627/846) 2024-08-20T21:48:11.8406655Z remote: Compressing objects: 75% (635/846) 2024-08-20T21:48:11.8410497Z remote: Compressing objects: 76% (643/846) 2024-08-20T21:48:11.8413022Z remote: Compressing objects: 77% (652/846) 2024-08-20T21:48:11.8417135Z remote: Compressing objects: 78% (660/846) 2024-08-20T21:48:11.8419960Z remote: Compressing objects: 79% (669/846) 2024-08-20T21:48:11.8421746Z remote: Compressing objects: 80% (677/846) 2024-08-20T21:48:11.8424394Z remote: Compressing objects: 81% (686/846) 2024-08-20T21:48:11.8426369Z remote: Compressing objects: 82% (694/846) 2024-08-20T21:48:11.8429071Z remote: Compressing objects: 83% (703/846) 2024-08-20T21:48:11.8431304Z remote: Compressing objects: 84% (711/846) 2024-08-20T21:48:11.8433215Z remote: Compressing objects: 85% (720/846) 2024-08-20T21:48:11.8435412Z remote: Compressing objects: 86% (728/846) 2024-08-20T21:48:11.8437390Z remote: Compressing objects: 87% (737/846) 2024-08-20T21:48:11.8439314Z remote: Compressing objects: 88% (745/846) 2024-08-20T21:48:11.8441944Z remote: Compressing objects: 89% (753/846) 2024-08-20T21:48:11.8444216Z remote: Compressing objects: 90% (762/846) 2024-08-20T21:48:11.8445980Z remote: Compressing objects: 91% (770/846) 2024-08-20T21:48:11.8447065Z remote: Compressing objects: 92% (779/846) 2024-08-20T21:48:11.8448087Z remote: Compressing objects: 93% (787/846) 2024-08-20T21:48:11.8449054Z remote: Compressing objects: 94% (796/846) 2024-08-20T21:48:11.8450428Z remote: Compressing objects: 95% (804/846) 2024-08-20T21:48:11.8452656Z remote: Compressing objects: 96% (813/846) 2024-08-20T21:48:11.8453350Z remote: Compressing objects: 97% (821/846) 2024-08-20T21:48:11.8455167Z remote: Compressing objects: 98% (830/846) 2024-08-20T21:48:11.8455937Z remote: Compressing objects: 99% (838/846) 2024-08-20T21:48:11.8457390Z remote: Compressing objects: 100% (846/846) 2024-08-20T21:48:11.8458104Z remote: Compressing objects: 100% (846/846), done. 2024-08-20T21:48:39.4975776Z remote: Total 1010161 (delta 1077), reused 1266 (delta 802), pack-reused 1008510 (from 1) 2024-08-20T21:49:01.3356813Z [command]/usr/bin/git rev-parse --verify --quiet 40ec5f6ddd9787aca0449b24128343ff4c4a88b3^{object} 2024-08-20T21:49:01.3388636Z 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:49:01.3394882Z ##[endgroup] 2024-08-20T21:49:01.3395700Z ##[group]Determining the checkout info 2024-08-20T21:49:01.3397684Z ##[endgroup] 2024-08-20T21:49:01.3398361Z ##[group]Checking out the ref 2024-08-20T21:49:01.3402786Z [command]/usr/bin/git checkout --quiet --force 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:49:03.2319714Z ##[endgroup] 2024-08-20T21:49:03.2320451Z ##[group]Setting up auth for fetching submodules 2024-08-20T21:49:03.2327528Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T21:49:03.2405899Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-20T21:49:03.2425489Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-20T21:49:03.2463080Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-20T21:49:03.2499857Z ##[endgroup] 2024-08-20T21:49:03.2500477Z ##[group]Fetching submodules 2024-08-20T21:49:03.2506881Z [command]/usr/bin/git submodule sync --recursive 2024-08-20T21:49:03.2856673Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-20T21:49:03.3195478Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-20T21:49:03.3198847Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-20T21:49:03.3202526Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-20T21:49:03.3206136Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-20T21:49:03.3210763Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-20T21:49:03.3213632Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-20T21:49:03.3218012Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-20T21:49:03.3222324Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-20T21:49:03.3226746Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-20T21:49:03.3232011Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-20T21:49:03.3236483Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-20T21:49:03.3241090Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-20T21:49:03.3247479Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-20T21:49:03.3252520Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-20T21:49:03.3257454Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-20T21:49:03.3262897Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:49:03.3271023Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-20T21:49:03.3277049Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-20T21:49:03.3282376Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-20T21:49:03.3288179Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-20T21:49:03.3293918Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-20T21:49:03.3299953Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-20T21:49:03.3305738Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-20T21:49:03.3311940Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-20T21:49:03.3318291Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-20T21:49:03.3325093Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-20T21:49:03.3331399Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-20T21:49:03.3338021Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-20T21:49:03.3344767Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-20T21:49:03.3351819Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-20T21:49:03.3358614Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-20T21:49:03.3365824Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-20T21:49:03.3373766Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-20T21:49:03.3383515Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-20T21:49:03.3414969Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-20T21:49:03.7007661Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-20T21:49:03.9015808Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-20T21:49:04.1065181Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-20T21:49:04.3873818Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-20T21:49:06.3758173Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-20T21:49:18.2543241Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-20T21:49:18.7010587Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-20T21:49:19.1269795Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-20T21:49:19.8211251Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-20T21:49:21.2612185Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-20T21:49:23.7047139Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-20T21:49:30.5729386Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-20T21:49:32.1385713Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-20T21:49:34.0162322Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-20T21:49:35.3833764Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-20T21:49:35.8688537Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-20T21:49:36.2480650Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-20T21:49:37.6192368Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-20T21:49:38.0447304Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-20T21:49:38.3118755Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-20T21:49:40.2217177Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-20T21:49:41.0397610Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-20T21:49:41.7262595Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-20T21:49:48.5184998Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-20T21:49:50.6197908Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-20T21:49:54.7953593Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-20T21:49:55.0287410Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-20T21:50:05.4246938Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-20T21:50:05.6233900Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-20T21:50:05.8325638Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-20T21:50:06.8314153Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-20T21:50:07.1580079Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-20T21:50:07.8638850Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-20T21:50:08.3275114Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-20T21:50:08.3420007Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-20T21:50:08.3529632Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-20T21:50:08.3860278Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-20T21:50:08.4343243Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-20T21:50:09.7423425Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-20T21:50:09.7717801Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-20T21:50:09.8275622Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-20T21:50:09.9524228Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-20T21:50:09.9968187Z Submodule path 'third_party/cudnn_frontend': checked out '23511ba176243f27b3b275da1fb3814ea805a171' 2024-08-20T21:50:10.6876758Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-20T21:50:11.0117009Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-20T21:50:11.1209582Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-20T21:50:11.1230352Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:50:11.1234443Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:50:11.1238175Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:50:11.1242067Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:50:11.1246185Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:50:11.1277031Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-20T21:50:12.3371894Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-20T21:50:13.0138842Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-20T21:50:15.4232352Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-20T21:50:16.5593852Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-20T21:50:16.9188215Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-20T21:50:17.0420871Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-20T21:50:17.5890351Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-20T21:50:17.6678224Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-20T21:50:17.6824557Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-20T21:50:17.8422233Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-20T21:50:17.8914358Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-20T21:50:17.9407501Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-20T21:50:17.9741948Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-20T21:50:18.0309522Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T21:50:18.0465765Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-20T21:50:18.0486459Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-20T21:50:18.0517491Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-20T21:50:32.6729035Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-20T21:50:32.6948242Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-20T21:50:32.8068536Z Submodule path 'third_party/kineto': checked out 'd9753139d181b9ff42872465aac0e5d3018be415' 2024-08-20T21:50:32.8090672Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:50:32.8094070Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:50:32.8097969Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:50:32.8129230Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-20T21:50:33.4497984Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-20T21:50:34.7563869Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-20T21:50:36.0244748Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-20T21:50:36.0264888Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:50:36.0268661Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:50:36.0273041Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:50:36.0277080Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:50:36.0281094Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:50:36.0285446Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:50:36.0289693Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:50:36.0294298Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:50:36.0326332Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-20T21:50:37.0087410Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-20T21:50:37.4203536Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-20T21:50:38.6965222Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-20T21:50:39.0027721Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-20T21:50:39.5728046Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-20T21:50:40.7290287Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-20T21:50:47.8099762Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-20T21:50:48.2876219Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-20T21:50:48.3119599Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-20T21:50:48.3587687Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-20T21:50:48.3749926Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-20T21:50:48.3770362Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:50:48.3801454Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-20T21:50:48.7214936Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-20T21:50:48.7452873Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-20T21:50:48.7983124Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-20T21:50:48.9361122Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-20T21:50:48.9565382Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-20T21:50:49.0060993Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-20T21:50:49.0803543Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-20T21:50:49.1263710Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-20T21:50:49.1579328Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-20T21:50:49.2974392Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-20T21:50:49.8086051Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-20T21:50:49.8127361Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-20T21:50:49.8129980Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-20T21:50:49.8161830Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-20T21:50:50.3435882Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-20T21:50:51.4043083Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-20T21:50:51.4481731Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-20T21:50:51.5428271Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-20T21:50:51.5451078Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:50:51.5454797Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:50:51.5458632Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:50:51.5462454Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:50:51.5466602Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:50:51.5470919Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:50:51.5475241Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:50:51.5479221Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:50:51.5509280Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-20T21:50:52.0067271Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-20T21:50:53.1174810Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-20T21:50:53.4417246Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-20T21:51:00.3103135Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-20T21:51:00.5844865Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-20T21:51:00.8066936Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-20T21:51:01.1781456Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-20T21:51:08.6387894Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-20T21:51:08.6909181Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-20T21:51:08.7105023Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-20T21:51:08.8464362Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-20T21:51:08.8624650Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-20T21:51:08.8810163Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-20T21:51:08.9014801Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-20T21:51:08.9034522Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:51:08.9038422Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:51:08.9067368Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-20T21:51:10.8923883Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-20T21:51:12.3750686Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-20T21:51:12.4335352Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T21:51:13.1269686Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-20T21:51:13.1409327Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-20T21:51:13.4858708Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-20T21:51:13.4885249Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:51:13.4889653Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-20T21:51:13.4920207Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-20T21:51:14.0356980Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-20T21:51:15.2260315Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-20T21:51:15.3159839Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-20T21:51:15.3266988Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-20T21:51:15.3413662Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-20T21:51:15.3875972Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-20T21:51:15.4235603Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-20T21:51:15.4777173Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-20T21:51:15.5131256Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-20T21:51:15.5151577Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:51:15.5155083Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:51:15.5158977Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:51:15.5162681Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:51:15.5192346Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-20T21:51:16.6699111Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-20T21:51:16.8950308Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-20T21:51:18.2470139Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-20T21:51:19.3024475Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-20T21:51:19.3218197Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-20T21:51:19.4084358Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-20T21:51:19.4445434Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-20T21:51:19.4464700Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:51:19.4495084Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-20T21:51:19.6510653Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-20T21:51:19.6550618Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-20T21:51:19.6894003Z Entering 'android/libs/fbjni' 2024-08-20T21:51:19.6941137Z Entering 'third_party/FP16' 2024-08-20T21:51:19.6989527Z Entering 'third_party/FXdiv' 2024-08-20T21:51:19.7038822Z Entering 'third_party/NNPACK' 2024-08-20T21:51:19.7087744Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T21:51:19.7135276Z Entering 'third_party/XNNPACK' 2024-08-20T21:51:19.7202323Z Entering 'third_party/benchmark' 2024-08-20T21:51:19.7249536Z Entering 'third_party/cpp-httplib' 2024-08-20T21:51:19.7297704Z Entering 'third_party/cpuinfo' 2024-08-20T21:51:19.7346118Z Entering 'third_party/cudnn_frontend' 2024-08-20T21:51:19.7395844Z Entering 'third_party/cutlass' 2024-08-20T21:51:19.7452685Z Entering 'third_party/eigen' 2024-08-20T21:51:19.7505590Z Entering 'third_party/fbgemm' 2024-08-20T21:51:19.7553359Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:51:19.7601306Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:51:19.7647890Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:51:19.7703939Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:51:19.7750416Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:51:19.7798803Z Entering 'third_party/flatbuffers' 2024-08-20T21:51:19.7849777Z Entering 'third_party/fmt' 2024-08-20T21:51:19.7898910Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:51:19.7946502Z Entering 'third_party/gloo' 2024-08-20T21:51:19.7995531Z Entering 'third_party/googletest' 2024-08-20T21:51:19.8042581Z Entering 'third_party/ideep' 2024-08-20T21:51:19.8089397Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T21:51:19.8143443Z Entering 'third_party/ittapi' 2024-08-20T21:51:19.8191223Z Entering 'third_party/kineto' 2024-08-20T21:51:19.8240386Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:51:19.8288013Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:51:19.8336668Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:51:19.8385240Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:51:19.8431995Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:51:19.8480142Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:51:19.8529497Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:51:19.8579451Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:51:19.8626585Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:51:19.8675760Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:51:19.8725298Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:51:19.8773068Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:51:19.8821228Z Entering 'third_party/mimalloc' 2024-08-20T21:51:19.8868859Z Entering 'third_party/nccl/nccl' 2024-08-20T21:51:19.8917718Z Entering 'third_party/nlohmann' 2024-08-20T21:51:19.8967036Z Entering 'third_party/onnx' 2024-08-20T21:51:19.9033239Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T21:51:19.9081123Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T21:51:19.9130872Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T21:51:19.9180652Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:51:19.9227839Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:51:19.9276367Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:51:19.9322461Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:51:19.9370801Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:51:19.9418300Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:51:19.9465384Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:51:19.9512468Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:51:19.9562058Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:51:19.9612326Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:51:19.9682281Z Entering 'third_party/pocketfft' 2024-08-20T21:51:19.9729700Z Entering 'third_party/protobuf' 2024-08-20T21:51:19.9782747Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:51:19.9829166Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T21:51:19.9880293Z Entering 'third_party/psimd' 2024-08-20T21:51:19.9927086Z Entering 'third_party/pthreadpool' 2024-08-20T21:51:19.9974663Z Entering 'third_party/pybind11' 2024-08-20T21:51:20.0021771Z Entering 'third_party/python-peachpy' 2024-08-20T21:51:20.0069308Z Entering 'third_party/sleef' 2024-08-20T21:51:20.0116820Z Entering 'third_party/tensorpipe' 2024-08-20T21:51:20.0164448Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:51:20.0212295Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:51:20.0258854Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:51:20.0306962Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:51:20.0353656Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:51:20.0417768Z ##[endgroup] 2024-08-20T21:51:20.0422568Z ##[group]Persisting credentials for submodules 2024-08-20T21:51:20.0425039Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-20T21:51:20.0766865Z Entering 'android/libs/fbjni' 2024-08-20T21:51:20.0829558Z Entering 'third_party/FP16' 2024-08-20T21:51:20.0891845Z Entering 'third_party/FXdiv' 2024-08-20T21:51:20.0953647Z Entering 'third_party/NNPACK' 2024-08-20T21:51:20.1017000Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T21:51:20.1079441Z Entering 'third_party/XNNPACK' 2024-08-20T21:51:20.1158997Z Entering 'third_party/benchmark' 2024-08-20T21:51:20.1221850Z Entering 'third_party/cpp-httplib' 2024-08-20T21:51:20.1284068Z Entering 'third_party/cpuinfo' 2024-08-20T21:51:20.1346818Z Entering 'third_party/cudnn_frontend' 2024-08-20T21:51:20.1410573Z Entering 'third_party/cutlass' 2024-08-20T21:51:20.1481749Z Entering 'third_party/eigen' 2024-08-20T21:51:20.1545376Z Entering 'third_party/fbgemm' 2024-08-20T21:51:20.1609290Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:51:20.1670267Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:51:20.1731951Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:51:20.1801724Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:51:20.1862907Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:51:20.1927939Z Entering 'third_party/flatbuffers' 2024-08-20T21:51:20.1992754Z Entering 'third_party/fmt' 2024-08-20T21:51:20.2054464Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:51:20.2117563Z Entering 'third_party/gloo' 2024-08-20T21:51:20.2180811Z Entering 'third_party/googletest' 2024-08-20T21:51:20.2242719Z Entering 'third_party/ideep' 2024-08-20T21:51:20.2303904Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T21:51:20.2372963Z Entering 'third_party/ittapi' 2024-08-20T21:51:20.2434244Z Entering 'third_party/kineto' 2024-08-20T21:51:20.2496839Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:51:20.2558703Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:51:20.2622899Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:51:20.2685906Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:51:20.2747591Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:51:20.2810223Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:51:20.2875465Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:51:20.2937343Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:51:20.3001569Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:51:20.3065757Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:51:20.3130200Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:51:20.3192283Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:51:20.3255002Z Entering 'third_party/mimalloc' 2024-08-20T21:51:20.3318051Z Entering 'third_party/nccl/nccl' 2024-08-20T21:51:20.3381783Z Entering 'third_party/nlohmann' 2024-08-20T21:51:20.3444608Z Entering 'third_party/onnx' 2024-08-20T21:51:20.3524595Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T21:51:20.3586468Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T21:51:20.3649934Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T21:51:20.3713893Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:51:20.3776191Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:51:20.3836904Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:51:20.3898192Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:51:20.3960915Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:51:20.4022482Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:51:20.4083851Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:51:20.4144433Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:51:20.4209779Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:51:20.4273650Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:51:20.4356452Z Entering 'third_party/pocketfft' 2024-08-20T21:51:20.4420004Z Entering 'third_party/protobuf' 2024-08-20T21:51:20.4488254Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:51:20.4549267Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T21:51:20.4612931Z Entering 'third_party/psimd' 2024-08-20T21:51:20.4675109Z Entering 'third_party/pthreadpool' 2024-08-20T21:51:20.4736511Z Entering 'third_party/pybind11' 2024-08-20T21:51:20.4798980Z Entering 'third_party/python-peachpy' 2024-08-20T21:51:20.4860444Z Entering 'third_party/sleef' 2024-08-20T21:51:20.4923610Z Entering 'third_party/tensorpipe' 2024-08-20T21:51:20.4985206Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:51:20.5046740Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:51:20.5107858Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:51:20.5168938Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:51:20.5229677Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:51:20.5311639Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-20T21:51:20.5653416Z Entering 'android/libs/fbjni' 2024-08-20T21:51:20.5711836Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-20T21:51:20.5730389Z Entering 'third_party/FP16' 2024-08-20T21:51:20.5789069Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-20T21:51:20.5806900Z Entering 'third_party/FXdiv' 2024-08-20T21:51:20.5865525Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-20T21:51:20.5886141Z Entering 'third_party/NNPACK' 2024-08-20T21:51:20.5943974Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-20T21:51:20.5962560Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T21:51:20.6021288Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-20T21:51:20.6039731Z Entering 'third_party/XNNPACK' 2024-08-20T21:51:20.6098639Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-20T21:51:20.6135330Z Entering 'third_party/benchmark' 2024-08-20T21:51:20.6193953Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-20T21:51:20.6212219Z Entering 'third_party/cpp-httplib' 2024-08-20T21:51:20.6273515Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-20T21:51:20.6291482Z Entering 'third_party/cpuinfo' 2024-08-20T21:51:20.6349083Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T21:51:20.6367916Z Entering 'third_party/cudnn_frontend' 2024-08-20T21:51:20.6427072Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-20T21:51:20.6445509Z Entering 'third_party/cutlass' 2024-08-20T21:51:20.6505091Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-20T21:51:20.6531294Z Entering 'third_party/eigen' 2024-08-20T21:51:20.6590130Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-20T21:51:20.6611144Z Entering 'third_party/fbgemm' 2024-08-20T21:51:20.6671590Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-20T21:51:20.6689635Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:51:20.6747395Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-20T21:51:20.6765642Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:51:20.6824829Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T21:51:20.6842846Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:51:20.6901637Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-20T21:51:20.6926691Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:51:20.6986134Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:51:20.7003863Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:51:20.7061402Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-20T21:51:20.7082356Z Entering 'third_party/flatbuffers' 2024-08-20T21:51:20.7140670Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-20T21:51:20.7162217Z Entering 'third_party/fmt' 2024-08-20T21:51:20.7221749Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-20T21:51:20.7240097Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:51:20.7298412Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-20T21:51:20.7316620Z Entering 'third_party/gloo' 2024-08-20T21:51:20.7374974Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-20T21:51:20.7393595Z Entering 'third_party/googletest' 2024-08-20T21:51:20.7451610Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:51:20.7470645Z Entering 'third_party/ideep' 2024-08-20T21:51:20.7528672Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-20T21:51:20.7545815Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T21:51:20.7604380Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-20T21:51:20.7630324Z Entering 'third_party/ittapi' 2024-08-20T21:51:20.7689370Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-20T21:51:20.7707390Z Entering 'third_party/kineto' 2024-08-20T21:51:20.7765460Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-20T21:51:20.7783864Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:51:20.7843045Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-20T21:51:20.7860628Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:51:20.7920388Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-20T21:51:20.7939667Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:51:20.7999664Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-20T21:51:20.8018021Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:51:20.8079499Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-20T21:51:20.8097552Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:51:20.8157028Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-20T21:51:20.8174639Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:51:20.8234968Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-20T21:51:20.8254968Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:51:20.8315438Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-20T21:51:20.8334573Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:51:20.8394214Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:51:20.8412575Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:51:20.8471458Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-20T21:51:20.8491659Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:51:20.8550179Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-20T21:51:20.8570901Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:51:20.8628986Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-20T21:51:20.8647343Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:51:20.8706047Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-20T21:51:20.8725946Z Entering 'third_party/mimalloc' 2024-08-20T21:51:20.8784776Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-20T21:51:20.8802940Z Entering 'third_party/nccl/nccl' 2024-08-20T21:51:20.8861981Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-20T21:51:20.8881657Z Entering 'third_party/nlohmann' 2024-08-20T21:51:20.8939216Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-20T21:51:20.8958722Z Entering 'third_party/onnx' 2024-08-20T21:51:20.9018435Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-20T21:51:20.9053872Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T21:51:20.9112948Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-20T21:51:20.9130950Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T21:51:20.9191546Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-20T21:51:20.9211890Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T21:51:20.9271076Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-20T21:51:20.9291062Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:51:20.9348960Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-20T21:51:20.9366850Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:51:20.9425470Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:51:20.9443569Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:51:20.9502660Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-20T21:51:20.9520418Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:51:20.9579742Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-20T21:51:20.9599149Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:51:20.9656810Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-20T21:51:20.9675504Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:51:20.9732943Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-20T21:51:20.9750753Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:51:20.9812003Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-20T21:51:20.9828925Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:51:20.9887782Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-20T21:51:20.9907413Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:51:20.9965943Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-20T21:51:20.9986481Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:51:21.0044625Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-20T21:51:21.0086350Z Entering 'third_party/pocketfft' 2024-08-20T21:51:21.0144494Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-20T21:51:21.0162345Z Entering 'third_party/protobuf' 2024-08-20T21:51:21.0221539Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-20T21:51:21.0243264Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:51:21.0303817Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-20T21:51:21.0321594Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T21:51:21.0380553Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:51:21.0401197Z Entering 'third_party/psimd' 2024-08-20T21:51:21.0459564Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-20T21:51:21.0478825Z Entering 'third_party/pthreadpool' 2024-08-20T21:51:21.0536956Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-20T21:51:21.0554965Z Entering 'third_party/pybind11' 2024-08-20T21:51:21.0614886Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-20T21:51:21.0633389Z Entering 'third_party/python-peachpy' 2024-08-20T21:51:21.0691991Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-20T21:51:21.0709978Z Entering 'third_party/sleef' 2024-08-20T21:51:21.0768264Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-20T21:51:21.0787027Z Entering 'third_party/tensorpipe' 2024-08-20T21:51:21.0847343Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-20T21:51:21.0865110Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:51:21.0924182Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:51:21.0942089Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:51:21.1000413Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-20T21:51:21.1018013Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:51:21.1076508Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-20T21:51:21.1094576Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:51:21.1152719Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-20T21:51:21.1169391Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:51:21.1229403Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-20T21:51:21.2272802Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-20T21:51:21.2612519Z Entering 'android/libs/fbjni' 2024-08-20T21:51:21.2659323Z Entering 'third_party/FP16' 2024-08-20T21:51:21.2708055Z Entering 'third_party/FXdiv' 2024-08-20T21:51:21.2756733Z Entering 'third_party/NNPACK' 2024-08-20T21:51:21.2805808Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T21:51:21.2852988Z Entering 'third_party/XNNPACK' 2024-08-20T21:51:21.2919135Z Entering 'third_party/benchmark' 2024-08-20T21:51:21.2966574Z Entering 'third_party/cpp-httplib' 2024-08-20T21:51:21.3015556Z Entering 'third_party/cpuinfo' 2024-08-20T21:51:21.3063395Z Entering 'third_party/cudnn_frontend' 2024-08-20T21:51:21.3114072Z Entering 'third_party/cutlass' 2024-08-20T21:51:21.3171511Z Entering 'third_party/eigen' 2024-08-20T21:51:21.3221191Z Entering 'third_party/fbgemm' 2024-08-20T21:51:21.3270472Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:51:21.3317185Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:51:21.3364645Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:51:21.3421481Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:51:21.3467923Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:51:21.3516355Z Entering 'third_party/flatbuffers' 2024-08-20T21:51:21.3566981Z Entering 'third_party/fmt' 2024-08-20T21:51:21.3616096Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:51:21.3664376Z Entering 'third_party/gloo' 2024-08-20T21:51:21.3713351Z Entering 'third_party/googletest' 2024-08-20T21:51:21.3761246Z Entering 'third_party/ideep' 2024-08-20T21:51:21.3808219Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T21:51:21.3862162Z Entering 'third_party/ittapi' 2024-08-20T21:51:21.3910434Z Entering 'third_party/kineto' 2024-08-20T21:51:21.3958644Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:51:21.4008944Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:51:21.4057592Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:51:21.4106280Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:51:21.4154383Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:51:21.4203119Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:51:21.4252003Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:51:21.4300448Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:51:21.4347855Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:51:21.4397403Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:51:21.4446666Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:51:21.4494237Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:51:21.4542559Z Entering 'third_party/mimalloc' 2024-08-20T21:51:21.4591165Z Entering 'third_party/nccl/nccl' 2024-08-20T21:51:21.4638734Z Entering 'third_party/nlohmann' 2024-08-20T21:51:21.4688645Z Entering 'third_party/onnx' 2024-08-20T21:51:21.4753344Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T21:51:21.4801868Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T21:51:21.4851544Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T21:51:21.4903325Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:51:21.4950008Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:51:21.4996735Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:51:21.5043951Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:51:21.5093170Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:51:21.5139891Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:51:21.5186475Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:51:21.5232272Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:51:21.5282547Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:51:21.5330729Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:51:21.5402903Z Entering 'third_party/pocketfft' 2024-08-20T21:51:21.5449832Z Entering 'third_party/protobuf' 2024-08-20T21:51:21.5501530Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:51:21.5548364Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T21:51:21.5608585Z Entering 'third_party/psimd' 2024-08-20T21:51:21.5656077Z Entering 'third_party/pthreadpool' 2024-08-20T21:51:21.5705202Z Entering 'third_party/pybind11' 2024-08-20T21:51:21.5752752Z Entering 'third_party/python-peachpy' 2024-08-20T21:51:21.5801253Z Entering 'third_party/sleef' 2024-08-20T21:51:21.5848610Z Entering 'third_party/tensorpipe' 2024-08-20T21:51:21.5897393Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:51:21.5943842Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:51:21.5991981Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:51:21.6038717Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:51:21.6086028Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:51:21.6152685Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-20T21:51:21.6496573Z Entering 'android/libs/fbjni' 2024-08-20T21:51:21.6544786Z Entering 'third_party/FP16' 2024-08-20T21:51:21.6592922Z Entering 'third_party/FXdiv' 2024-08-20T21:51:21.6640374Z Entering 'third_party/NNPACK' 2024-08-20T21:51:21.6688930Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T21:51:21.6736985Z Entering 'third_party/XNNPACK' 2024-08-20T21:51:21.6803691Z Entering 'third_party/benchmark' 2024-08-20T21:51:21.6851781Z Entering 'third_party/cpp-httplib' 2024-08-20T21:51:21.6900173Z Entering 'third_party/cpuinfo' 2024-08-20T21:51:21.6948249Z Entering 'third_party/cudnn_frontend' 2024-08-20T21:51:21.6998169Z Entering 'third_party/cutlass' 2024-08-20T21:51:21.7054594Z Entering 'third_party/eigen' 2024-08-20T21:51:21.7106840Z Entering 'third_party/fbgemm' 2024-08-20T21:51:21.7154848Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:51:21.7202681Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:51:21.7249932Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:51:21.7305042Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:51:21.7351638Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:51:21.7400911Z Entering 'third_party/flatbuffers' 2024-08-20T21:51:21.7451751Z Entering 'third_party/fmt' 2024-08-20T21:51:21.7501159Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:51:21.7549189Z Entering 'third_party/gloo' 2024-08-20T21:51:21.7598587Z Entering 'third_party/googletest' 2024-08-20T21:51:21.7645747Z Entering 'third_party/ideep' 2024-08-20T21:51:21.7694483Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T21:51:21.7750226Z Entering 'third_party/ittapi' 2024-08-20T21:51:21.7798045Z Entering 'third_party/kineto' 2024-08-20T21:51:21.7845848Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:51:21.7895179Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:51:21.7944790Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:51:21.7994539Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:51:21.8043144Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:51:21.8090481Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:51:21.8139746Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:51:21.8190502Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:51:21.8237933Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:51:21.8288296Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:51:21.8337695Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:51:21.8385142Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:51:21.8433712Z Entering 'third_party/mimalloc' 2024-08-20T21:51:21.8482759Z Entering 'third_party/nccl/nccl' 2024-08-20T21:51:21.8530737Z Entering 'third_party/nlohmann' 2024-08-20T21:51:21.8581917Z Entering 'third_party/onnx' 2024-08-20T21:51:21.8648148Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T21:51:21.8696203Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T21:51:21.8746140Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T21:51:21.8797576Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:51:21.8844596Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:51:21.8892689Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:51:21.8939501Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:51:21.8989828Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:51:21.9036610Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:51:21.9083382Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:51:21.9129785Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:51:21.9181016Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:51:21.9230040Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:51:21.9300764Z Entering 'third_party/pocketfft' 2024-08-20T21:51:21.9348178Z Entering 'third_party/protobuf' 2024-08-20T21:51:21.9400543Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:51:21.9448116Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T21:51:21.9497952Z Entering 'third_party/psimd' 2024-08-20T21:51:21.9545169Z Entering 'third_party/pthreadpool' 2024-08-20T21:51:21.9594284Z Entering 'third_party/pybind11' 2024-08-20T21:51:21.9641803Z Entering 'third_party/python-peachpy' 2024-08-20T21:51:21.9691566Z Entering 'third_party/sleef' 2024-08-20T21:51:21.9738963Z Entering 'third_party/tensorpipe' 2024-08-20T21:51:21.9787085Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:51:21.9834456Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:51:21.9882147Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:51:21.9929749Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:51:21.9976350Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:51:22.0039689Z ##[endgroup] 2024-08-20T21:51:22.0087851Z [command]/usr/bin/git log -1 --format='%H' 2024-08-20T21:51:22.0119530Z '40ec5f6ddd9787aca0449b24128343ff4c4a88b3' 2024-08-20T21:51:22.0354719Z Prepare all required actions 2024-08-20T21:51:22.0355322Z Getting action download info 2024-08-20T21:51:22.1784688Z ##[group]Run ./.github/actions/setup-linux 2024-08-20T21:51:22.1785222Z env: 2024-08-20T21:51:22.1785551Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:22.1785954Z ##[endgroup] 2024-08-20T21:51:22.1894281Z ##[group]Run set -euo pipefail 2024-08-20T21:51:22.1894807Z set -euo pipefail 2024-08-20T21:51:22.1895282Z function get_ec2_metadata() { 2024-08-20T21:51:22.1895930Z  # Pulled from instance metadata endpoint for EC2 2024-08-20T21:51:22.1897053Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-20T21:51:22.1898229Z  category=$1 2024-08-20T21:51:22.1898862Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-20T21:51:22.1899651Z  runner_name_str=i-0cc1c134a48939331 2024-08-20T21:51:22.1900344Z  if [[ -f /.inarc ]]; then 2024-08-20T21:51:22.1900948Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-20T21:51:22.1901650Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-20T21:51:22.1902509Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-20T21:51:22.1903250Z  else 2024-08-20T21:51:22.1903856Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-20T21:51:22.1904571Z  fi 2024-08-20T21:51:22.1904897Z } 2024-08-20T21:51:22.1905333Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-20T21:51:22.1906060Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-20T21:51:22.1906878Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-20T21:51:22.1907563Z echo "system info $(uname -a)" 2024-08-20T21:51:22.1916597Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:22.1917221Z env: 2024-08-20T21:51:22.1917546Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:22.1917976Z ##[endgroup] 2024-08-20T21:51:22.2027733Z ami-id: ami-06c68f701d8090592 2024-08-20T21:51:22.2092725Z instance-id: i-0cc1c134a48939331 2024-08-20T21:51:22.2152320Z instance-type: g3.8xlarge 2024-08-20T21:51:22.2165372Z system info Linux ip-10-0-2-62.ec2.internal 6.1.94-99.176.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-20T21:51:22.2195973Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T21:51:22.2197143Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T21:51:22.2204770Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:22.2205402Z env: 2024-08-20T21:51:22.2205713Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:22.2206134Z ##[endgroup] 2024-08-20T21:51:22.2289941Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-20T21:51:22.2290689Z if systemctl is-active --quiet docker; then 2024-08-20T21:51:22.2291355Z  echo "Docker daemon is running..."; 2024-08-20T21:51:22.2291908Z else 2024-08-20T21:51:22.2292498Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-20T21:51:22.2293225Z fi 2024-08-20T21:51:22.2299932Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:22.2300548Z env: 2024-08-20T21:51:22.2300875Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:22.2301278Z ##[endgroup] 2024-08-20T21:51:22.2382258Z Docker daemon is running... 2024-08-20T21:51:22.2441531Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T21:51:22.2442239Z with: 2024-08-20T21:51:22.2442552Z shell: bash 2024-08-20T21:51:22.2442888Z timeout_minutes: 5 2024-08-20T21:51:22.2443277Z max_attempts: 3 2024-08-20T21:51:22.2443653Z retry_wait_seconds: 30 2024-08-20T21:51:22.2447664Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-08-20T21:51:22.2451583Z polling_interval_seconds: 1 2024-08-20T21:51:22.2452044Z warning_on_retry: true 2024-08-20T21:51:22.2452451Z continue_on_error: false 2024-08-20T21:51:22.2453004Z env: 2024-08-20T21:51:22.2453328Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:22.2453740Z AWS_RETRY_MODE: standard 2024-08-20T21:51:22.2454160Z AWS_MAX_ATTEMPTS: 5 2024-08-20T21:51:22.2454570Z AWS_DEFAULT_REGION: us-east-1 2024-08-20T21:51:22.2455016Z ##[endgroup] 2024-08-20T21:51:23.6956924Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T21:51:23.6957983Z Configure a credential helper to remove this warning. See 2024-08-20T21:51:23.6959074Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T21:51:23.6959730Z 2024-08-20T21:51:23.6959874Z Login Succeeded 2024-08-20T21:51:24.3182895Z Command completed after 1 attempt(s). 2024-08-20T21:51:24.3242664Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T21:51:24.3243559Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T21:51:24.3244384Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T21:51:24.3252810Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:24.3253416Z env: 2024-08-20T21:51:24.3253750Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:24.3254179Z ##[endgroup] 2024-08-20T21:51:24.3350479Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T21:51:24.3351439Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T21:51:24.3352170Z # shellcheck disable=SC2046 2024-08-20T21:51:24.3352738Z docker stop $(docker ps -q) || true 2024-08-20T21:51:24.3353312Z # Prune all of the docker images 2024-08-20T21:51:24.3353861Z docker system prune -af 2024-08-20T21:51:24.3360502Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:24.3361099Z env: 2024-08-20T21:51:24.3361429Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:24.3361854Z ##[endgroup] 2024-08-20T21:51:24.3708116Z "docker stop" requires at least 1 argument. 2024-08-20T21:51:24.3708934Z See 'docker stop --help'. 2024-08-20T21:51:24.3709246Z 2024-08-20T21:51:24.3709518Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-20T21:51:24.3710003Z 2024-08-20T21:51:24.3710181Z Stop one or more running containers 2024-08-20T21:51:24.3906522Z Total reclaimed space: 0B 2024-08-20T21:51:24.3980132Z ##[group]Run set +e 2024-08-20T21:51:24.3980591Z set +e 2024-08-20T21:51:24.3980936Z set -x 2024-08-20T21:51:24.3981285Z  2024-08-20T21:51:24.3981670Z PT_DOMAIN=download.pytorch.org 2024-08-20T21:51:24.3982634Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-20T21:51:24.3983985Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-20T21:51:24.3984921Z # one is returned at random 2024-08-20T21:51:24.3985571Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-20T21:51:24.3986213Z  2024-08-20T21:51:24.3986588Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T21:51:24.3987336Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-20T21:51:24.3988471Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-20T21:51:24.3989179Z  2024-08-20T21:51:24.3989564Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T21:51:24.3990221Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-20T21:51:24.3990844Z  exit 1 2024-08-20T21:51:24.3991221Z  fi 2024-08-20T21:51:24.3991546Z fi 2024-08-20T21:51:24.3991875Z  2024-08-20T21:51:24.3992303Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-20T21:51:24.3992926Z  # Clean up any old records first 2024-08-20T21:51:24.3993550Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-20T21:51:24.3994101Z fi 2024-08-20T21:51:24.3994411Z  2024-08-20T21:51:24.3995090Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-20T21:51:24.3995765Z cat /etc/hosts 2024-08-20T21:51:24.4003276Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:24.4003895Z env: 2024-08-20T21:51:24.4004231Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:24.4004640Z ##[endgroup] 2024-08-20T21:51:24.4033310Z + PT_DOMAIN=download.pytorch.org 2024-08-20T21:51:24.4039408Z ++ dig -4 +short download.pytorch.org 2024-08-20T21:51:24.4040190Z ++ tail -n1 2024-08-20T21:51:24.4280084Z + RESOLVED_IP=18.160.10.28 2024-08-20T21:51:24.4280626Z + '[' -z 18.160.10.28 ']' 2024-08-20T21:51:24.4281148Z + grep -r download.pytorch.org /etc/hosts 2024-08-20T21:51:24.4293836Z 18.165.98.58 download.pytorch.org 2024-08-20T21:51:24.4295575Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-08-20T21:51:24.5771280Z + echo '18.160.10.28 download.pytorch.org' 2024-08-20T21:51:24.5771955Z + sudo tee -a /etc/hosts 2024-08-20T21:51:24.6310627Z 18.160.10.28 download.pytorch.org 2024-08-20T21:51:24.6329718Z + cat /etc/hosts 2024-08-20T21:51:24.6339848Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-20T21:51:24.6354813Z ::1 localhost6 localhost6.localdomain6 2024-08-20T21:51:24.6355529Z 18.160.10.28 download.pytorch.org 2024-08-20T21:51:24.6558843Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-20T21:51:24.6559636Z with: 2024-08-20T21:51:24.6560845Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:24.6562232Z docker-build-dir: .ci/docker 2024-08-20T21:51:24.6562696Z working-directory: . 2024-08-20T21:51:24.6563264Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:24.6563923Z force-push: false 2024-08-20T21:51:24.6564285Z env: 2024-08-20T21:51:24.6564596Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:24.6565036Z ##[endgroup] 2024-08-20T21:51:24.6592690Z ##[group]Run set -ex 2024-08-20T21:51:24.6593132Z set -ex 2024-08-20T21:51:24.6593490Z  2024-08-20T21:51:24.6594169Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-20T21:51:24.6595464Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-20T21:51:24.6596474Z # job could then download the pre-built image as usual 2024-08-20T21:51:24.6597400Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-20T21:51:24.6598244Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:24.6599030Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:24.6599707Z  2024-08-20T21:51:24.6600334Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-20T21:51:24.6601110Z  exit 0 2024-08-20T21:51:24.6601468Z else 2024-08-20T21:51:24.6601916Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:24.6602470Z fi 2024-08-20T21:51:24.6602786Z  2024-08-20T21:51:24.6603358Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-20T21:51:24.6604430Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-20T21:51:24.6605384Z  # use it as it is, but first let's extract the tag 2024-08-20T21:51:24.6606262Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-20T21:51:24.6607169Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:24.6608041Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:24.6608724Z else 2024-08-20T21:51:24.6609264Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-20T21:51:24.6610313Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:24.6611417Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:24.6612358Z fi 2024-08-20T21:51:24.6619856Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:24.6620478Z env: 2024-08-20T21:51:24.6620802Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:24.6621218Z REPO_NAME: pytorch 2024-08-20T21:51:24.6622476Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:24.6623839Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T21:51:24.6624448Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:24.6625074Z ##[endgroup] 2024-08-20T21:51:24.6653532Z + [[ ! -d .ci/docker ]] 2024-08-20T21:51:24.6654068Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-20T21:51:24.6654528Z + echo skip=false 2024-08-20T21:51:24.6656694Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-20T21:51:24.6662057Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:24.6663466Z ++ awk -F '[:,]' '{print $2}' 2024-08-20T21:51:24.6686089Z + DOCKER_TAG=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:24.6686972Z + echo docker-tag=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:24.6688936Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:24.6726445Z ##[group]Run set +e 2024-08-20T21:51:24.6726882Z set +e 2024-08-20T21:51:24.6727243Z set -x 2024-08-20T21:51:24.6727582Z  2024-08-20T21:51:24.6727911Z login() { 2024-08-20T21:51:24.6728725Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T21:51:24.6729613Z } 2024-08-20T21:51:24.6729923Z  2024-08-20T21:51:24.6730260Z retry () { 2024-08-20T21:51:24.6730720Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T21:51:24.6731268Z } 2024-08-20T21:51:24.6731593Z  2024-08-20T21:51:24.6731950Z retry login "${DOCKER_REGISTRY}" 2024-08-20T21:51:24.6732453Z  2024-08-20T21:51:24.6733023Z # Check if image already exists, if it does then skip building it 2024-08-20T21:51:24.6733892Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-20T21:51:24.6734495Z  exit 0 2024-08-20T21:51:24.6734853Z fi 2024-08-20T21:51:24.6735178Z  2024-08-20T21:51:24.6735773Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-20T21:51:24.6736811Z # be empty. The default action would be to continue rebuild the image 2024-08-20T21:51:24.6737725Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-20T21:51:24.6738548Z  # if we're on the base branch then use the parent commit 2024-08-20T21:51:24.6739277Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-20T21:51:24.6739802Z else 2024-08-20T21:51:24.6740423Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-20T21:51:24.6741282Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-20T21:51:24.6741898Z fi 2024-08-20T21:51:24.6742213Z  2024-08-20T21:51:24.6742595Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-20T21:51:24.6743218Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:24.6743775Z  2024-08-20T21:51:24.6744570Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-20T21:51:24.6745714Z  exit 0 2024-08-20T21:51:24.6746076Z fi 2024-08-20T21:51:24.6746389Z  2024-08-20T21:51:24.6746920Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-20T21:51:24.6748143Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-20T21:51:24.6749136Z  exit 1 2024-08-20T21:51:24.6749497Z fi 2024-08-20T21:51:24.6749831Z  2024-08-20T21:51:24.6750410Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-20T21:51:24.6751590Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-20T21:51:24.6752668Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-20T21:51:24.6753874Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-20T21:51:24.6755243Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-20T21:51:24.6756045Z fi 2024-08-20T21:51:24.6756377Z  2024-08-20T21:51:24.6757020Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:24.6763587Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:24.6764203Z env: 2024-08-20T21:51:24.6764513Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:24.6764954Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T21:51:24.6765525Z BASE_REVISION: 91f3d614142df02f619e44a68e3d9e0dfeba49ec 2024-08-20T21:51:24.6766943Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:24.6768365Z DOCKER_TAG: f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:24.6769770Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:24.6770451Z ##[endgroup] 2024-08-20T21:51:24.6798069Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:24.6798888Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:24.6801029Z + aws ecr get-login-password --region us-east-1 2024-08-20T21:51:24.6802339Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:25.3481105Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T21:51:25.3482156Z Configure a credential helper to remove this warning. See 2024-08-20T21:51:25.3483248Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T21:51:25.3483910Z 2024-08-20T21:51:25.3528140Z Login Succeeded 2024-08-20T21:51:25.3529786Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:25.5778653Z { 2024-08-20T21:51:25.5779266Z "schemaVersion": 2, 2024-08-20T21:51:25.5780508Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-20T21:51:25.5781246Z "config": { 2024-08-20T21:51:25.5781797Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-20T21:51:25.5782473Z "size": 47467, 2024-08-20T21:51:25.5783396Z "digest": "sha256:35f4945dfeab4592c8657de7107f96a9fd9a51999f582a0064937246789c0825" 2024-08-20T21:51:25.5784158Z }, 2024-08-20T21:51:25.5784472Z "layers": [ 2024-08-20T21:51:25.5784814Z { 2024-08-20T21:51:25.5785332Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5786034Z "size": 28580681, 2024-08-20T21:51:25.5787974Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-08-20T21:51:25.5789366Z }, 2024-08-20T21:51:25.5789775Z { 2024-08-20T21:51:25.5790342Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5791275Z "size": 7943353, 2024-08-20T21:51:25.5791966Z "digest": "sha256:09d415c238d76b32a7ea4a6e6add9542db9a5641f7f183af70aae185d0709e58" 2024-08-20T21:51:25.5792747Z }, 2024-08-20T21:51:25.5793027Z { 2024-08-20T21:51:25.5793564Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5794270Z "size": 56226389, 2024-08-20T21:51:25.5794945Z "digest": "sha256:9fe6e2e61518cba6844870c03b285737daec35e62baf25ae7744629ed3a7b470" 2024-08-20T21:51:25.5795724Z }, 2024-08-20T21:51:25.5796018Z { 2024-08-20T21:51:25.5796602Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5797343Z "size": 187, 2024-08-20T21:51:25.5798026Z "digest": "sha256:41f16248e682693ff20b3032c1d5e5541cc87c5af898ae2ff9b24d2940e59100" 2024-08-20T21:51:25.5798839Z }, 2024-08-20T21:51:25.5799344Z + exit 0 2024-08-20T21:51:25.5799664Z { 2024-08-20T21:51:25.5800198Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5800879Z "size": 6885, 2024-08-20T21:51:25.5801554Z "digest": "sha256:95d7b781703928cf3c4eece39d800cccb76728c375fedf51ecd83833fb25e458" 2024-08-20T21:51:25.5802326Z }, 2024-08-20T21:51:25.5802601Z { 2024-08-20T21:51:25.5803293Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5804016Z "size": 1383662475, 2024-08-20T21:51:25.5804671Z "digest": "sha256:8f6c9048534734f4c873935293b7296225846ceb31c1a158400a67ea170dde7f" 2024-08-20T21:51:25.5805412Z }, 2024-08-20T21:51:25.5805710Z { 2024-08-20T21:51:25.5806221Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5806917Z "size": 62282, 2024-08-20T21:51:25.5807577Z "digest": "sha256:ab17245097e491b9368790714f9d90ed447bf0973bd677cfe6f2456d62b72a13" 2024-08-20T21:51:25.5808310Z }, 2024-08-20T21:51:25.5808605Z { 2024-08-20T21:51:25.5809133Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5809813Z "size": 1684, 2024-08-20T21:51:25.5810496Z "digest": "sha256:dfecd7e9912b76ed460b8edd5a85f1943666e38a973ab5458177cf2c7c3110e3" 2024-08-20T21:51:25.5811273Z }, 2024-08-20T21:51:25.5811550Z { 2024-08-20T21:51:25.5812087Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5812781Z "size": 1522, 2024-08-20T21:51:25.5813435Z "digest": "sha256:464a8f74544589bf7b57f9a4cadcb6681e5ed00758f6c35025e691df4e88e890" 2024-08-20T21:51:25.5814200Z }, 2024-08-20T21:51:25.5814497Z { 2024-08-20T21:51:25.5815010Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5815706Z "size": 2445884909, 2024-08-20T21:51:25.5816385Z "digest": "sha256:d67c111fa58886d8904335b5122039816c8e51af5844cb6fe4e14154d9a7c016" 2024-08-20T21:51:25.5817118Z }, 2024-08-20T21:51:25.5817410Z { 2024-08-20T21:51:25.5817940Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5818611Z "size": 85882, 2024-08-20T21:51:25.5819287Z "digest": "sha256:0723abde19c09431db80b5d0f0e003541f6232c2da5c4f35c635c861875937ae" 2024-08-20T21:51:25.5820048Z }, 2024-08-20T21:51:25.5820321Z { 2024-08-20T21:51:25.5820851Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5821539Z "size": 1822, 2024-08-20T21:51:25.5822204Z "digest": "sha256:72d2b53f7a0105206afbdce791a85d403e2413ac60a2ea3fec0bb057b69925ad" 2024-08-20T21:51:25.5822984Z }, 2024-08-20T21:51:25.5823277Z { 2024-08-20T21:51:25.5823788Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5824478Z "size": 654861296, 2024-08-20T21:51:25.5825191Z "digest": "sha256:dbc6fccd9c70404c8728c8d319bbe9cd5ef59e56b27d0a3ff9385dc2007420e2" 2024-08-20T21:51:25.5825957Z }, 2024-08-20T21:51:25.5826252Z { 2024-08-20T21:51:25.5826781Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5827451Z "size": 544, 2024-08-20T21:51:25.5828118Z "digest": "sha256:ff6a6ddc73105bc6954a81b0d5ed486362bfd4399595a378a7c6594a332069a3" 2024-08-20T21:51:25.5828993Z }, 2024-08-20T21:51:25.5829272Z { 2024-08-20T21:51:25.5829807Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5830496Z "size": 1258, 2024-08-20T21:51:25.5831172Z "digest": "sha256:fac2b1b246ed3a00ba10fdfe9dc50559f8978915b204b903bc5fb47908fb5bd5" 2024-08-20T21:51:25.5831954Z }, 2024-08-20T21:51:25.5832253Z { 2024-08-20T21:51:25.5832764Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5833450Z "size": 484, 2024-08-20T21:51:25.5834125Z "digest": "sha256:bc574f79516b484ef9cff3a621d4e8f6dd0ced47a0c7de6959a2213b313cd871" 2024-08-20T21:51:25.5834885Z }, 2024-08-20T21:51:25.5835181Z { 2024-08-20T21:51:25.5835708Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5836379Z "size": 91712209, 2024-08-20T21:51:25.5837085Z "digest": "sha256:c23d42adcadecf7e1d93290ebf26fe3936841777be574d8e2a4c71719d43f05c" 2024-08-20T21:51:25.5837875Z }, 2024-08-20T21:51:25.5838148Z { 2024-08-20T21:51:25.5838677Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5839359Z "size": 3390, 2024-08-20T21:51:25.5840166Z "digest": "sha256:59b0a331db9f6d35dab3eef54ae05d703663a7cfe6ea9ec139fdc8c2221834da" 2024-08-20T21:51:25.5840973Z }, 2024-08-20T21:51:25.5841269Z { 2024-08-20T21:51:25.5841780Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5842464Z "size": 1909, 2024-08-20T21:51:25.5843142Z "digest": "sha256:478a9a2bdefc98de190384a54d89524fb0d691b7155ffba4df69a35a50e4afae" 2024-08-20T21:51:25.5843916Z }, 2024-08-20T21:51:25.5844197Z { 2024-08-20T21:51:25.5844730Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5845405Z "size": 699, 2024-08-20T21:51:25.5846071Z "digest": "sha256:28e54511f1b116a9d632caa0c4be2cd1b6ca7542f815560ec598b81d6d956d31" 2024-08-20T21:51:25.5846851Z }, 2024-08-20T21:51:25.5847129Z { 2024-08-20T21:51:25.5847657Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5848348Z "size": 2891829542, 2024-08-20T21:51:25.5849037Z "digest": "sha256:5247f18de8815b263c1380f613ce6f8b836db6766c3b137ef30b18fac324a4a3" 2024-08-20T21:51:25.5849784Z }, 2024-08-20T21:51:25.5850081Z { 2024-08-20T21:51:25.5850613Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5851279Z "size": 380, 2024-08-20T21:51:25.5851914Z "digest": "sha256:38760708e2d79494fe04b599b375145b8968797e5a063609cb007de5037ff0cf" 2024-08-20T21:51:25.5852655Z }, 2024-08-20T21:51:25.5852929Z { 2024-08-20T21:51:25.5853464Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5854154Z "size": 12876, 2024-08-20T21:51:25.5854829Z "digest": "sha256:863dbde6ed0c5edba3f86f2907198ab33eed3dd4eed51375189fd722a66b051a" 2024-08-20T21:51:25.5855622Z }, 2024-08-20T21:51:25.5855932Z { 2024-08-20T21:51:25.5856460Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5857130Z "size": 805, 2024-08-20T21:51:25.5857802Z "digest": "sha256:f283ab568a00b37f29b9d76f77c2f99a05b336ee3b1ffff8577b6bd7df2e95f3" 2024-08-20T21:51:25.5858579Z }, 2024-08-20T21:51:25.5858856Z { 2024-08-20T21:51:25.5859393Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5860083Z "size": 106, 2024-08-20T21:51:25.5860726Z "digest": "sha256:3e39f30b1d5c27e19c987aa8f8fbb36778a68cd534872e718053f71997457dab" 2024-08-20T21:51:25.5861489Z }, 2024-08-20T21:51:25.5861782Z { 2024-08-20T21:51:25.5862296Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5862990Z "size": 503, 2024-08-20T21:51:25.5863640Z "digest": "sha256:521c3d72534c80920fa11f85e7276b2130851936dba93947a4dc647eaf8855ba" 2024-08-20T21:51:25.5864374Z }, 2024-08-20T21:51:25.5864670Z { 2024-08-20T21:51:25.5865201Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5865976Z "size": 121477349, 2024-08-20T21:51:25.5866671Z "digest": "sha256:a975a4480a6d2abb7c44d3b8fe0e198c4d2666336f05f12936968ac5ac02fbe3" 2024-08-20T21:51:25.5867436Z }, 2024-08-20T21:51:25.5867714Z { 2024-08-20T21:51:25.5868256Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5869310Z "size": 109, 2024-08-20T21:51:25.5869980Z "digest": "sha256:1f2c278212ad728862a858a586ea5b666cea7384cb3b51900fc12afdd1fb0508" 2024-08-20T21:51:25.5870743Z }, 2024-08-20T21:51:25.5871037Z { 2024-08-20T21:51:25.5871548Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5872237Z "size": 489, 2024-08-20T21:51:25.5872903Z "digest": "sha256:b721dd867e80edbea20e5691ce6254378a75d20d38436ffc551dbb91453aefc5" 2024-08-20T21:51:25.5873648Z }, 2024-08-20T21:51:25.5873938Z { 2024-08-20T21:51:25.5874467Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5875144Z "size": 297, 2024-08-20T21:51:25.5875821Z "digest": "sha256:7a74b7ad649d3f0cc23bef1d20005cc436d2f7d478b2e81519d40ad7aa8ea251" 2024-08-20T21:51:25.5876601Z }, 2024-08-20T21:51:25.5876876Z { 2024-08-20T21:51:25.5877542Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5878255Z "size": 103, 2024-08-20T21:51:25.5878908Z "digest": "sha256:b53e1ad1deec07d1a5f093f57f5048627e112a68a5f512f9d9d229d0a91b9b2b" 2024-08-20T21:51:25.5879681Z }, 2024-08-20T21:51:25.5879975Z { 2024-08-20T21:51:25.5880486Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5881172Z "size": 1472, 2024-08-20T21:51:25.5881849Z "digest": "sha256:016dc3faa7efaa82e2cb84df6d34535024761fcfe67540e7e1f82686190841ee" 2024-08-20T21:51:25.5882603Z }, 2024-08-20T21:51:25.5882899Z { 2024-08-20T21:51:25.5883428Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5884107Z "size": 594729536, 2024-08-20T21:51:25.5884795Z "digest": "sha256:384dce3a4ba2bb506140986d6a45133d58345f055cbf72d018cff59716419012" 2024-08-20T21:51:25.5885551Z }, 2024-08-20T21:51:25.5885823Z { 2024-08-20T21:51:25.5886355Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5887044Z "size": 164, 2024-08-20T21:51:25.5887679Z "digest": "sha256:75461ad9f7bb4e53c4a6bd26539108b012c321304219c48a8852353fc2260613" 2024-08-20T21:51:25.5888434Z }, 2024-08-20T21:51:25.5888724Z { 2024-08-20T21:51:25.5889233Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5889924Z "size": 562, 2024-08-20T21:51:25.5890593Z "digest": "sha256:eb2062102d1137b2c485af726affb99a0f62f1a159a6c6ff0fbc936dac18596d" 2024-08-20T21:51:25.5891366Z }, 2024-08-20T21:51:25.5891643Z { 2024-08-20T21:51:25.5892170Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5892840Z "size": 35806203, 2024-08-20T21:51:25.5893528Z "digest": "sha256:3cc17f105239b758d0e1ff69aa738939b0a89561da055484a0a9e2602e5a543f" 2024-08-20T21:51:25.5894285Z }, 2024-08-20T21:51:25.5894556Z { 2024-08-20T21:51:25.5895085Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5895771Z "size": 104, 2024-08-20T21:51:25.5896428Z "digest": "sha256:585501be07997851ac265aba0a5b71e2c2e9afb4cc9c30da46cd9f2f5ffe063d" 2024-08-20T21:51:25.5897196Z }, 2024-08-20T21:51:25.5897490Z { 2024-08-20T21:51:25.5897998Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5898687Z "size": 424, 2024-08-20T21:51:25.5899376Z "digest": "sha256:6c49da68c9b4f370ed5ecfbb742ff5cad0ccc3d855e71fc10fdf603cabe9d87b" 2024-08-20T21:51:25.5900164Z }, 2024-08-20T21:51:25.5900438Z { 2024-08-20T21:51:25.5900963Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5901651Z "size": 20262169, 2024-08-20T21:51:25.5902300Z "digest": "sha256:497539689260668c0d109b8e22341ac13b1c482fd9c0b2e8371b8c88bd9fc766" 2024-08-20T21:51:25.5903204Z }, 2024-08-20T21:51:25.5903500Z { 2024-08-20T21:51:25.5904014Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5904708Z "size": 438, 2024-08-20T21:51:25.5905379Z "digest": "sha256:5843bb8b81bb68867db16a80d2ebb31396b14b3fa08b419ce9ab07b0791eb662" 2024-08-20T21:51:25.5906135Z }, 2024-08-20T21:51:25.5906430Z { 2024-08-20T21:51:25.5906959Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5907626Z "size": 699, 2024-08-20T21:51:25.5908293Z "digest": "sha256:28e54511f1b116a9d632caa0c4be2cd1b6ca7542f815560ec598b81d6d956d31" 2024-08-20T21:51:25.5909057Z }, 2024-08-20T21:51:25.5909335Z { 2024-08-20T21:51:25.5909864Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5910547Z "size": 141, 2024-08-20T21:51:25.5911171Z "digest": "sha256:222174e4e2b25c47241345129f9ec8c9c219ef78391d65f6089d89339be6ddb7" 2024-08-20T21:51:25.5911926Z }, 2024-08-20T21:51:25.5912218Z { 2024-08-20T21:51:25.5912729Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5913415Z "size": 135, 2024-08-20T21:51:25.5914186Z "digest": "sha256:8d55064a9f207c06db9da40e295a3430ac0b288cfdf6968903dddb487789b380" 2024-08-20T21:51:25.5914954Z }, 2024-08-20T21:51:25.5915253Z { 2024-08-20T21:51:25.5915782Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5916450Z "size": 32, 2024-08-20T21:51:25.5917119Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:51:25.5917893Z }, 2024-08-20T21:51:25.5918168Z { 2024-08-20T21:51:25.5918695Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5919379Z "size": 194, 2024-08-20T21:51:25.5920022Z "digest": "sha256:25114d5bae5b7775201e9a1f79c06cc0ab7ca238a8357e83018a63d782aef5d5" 2024-08-20T21:51:25.5920783Z }, 2024-08-20T21:51:25.5921085Z { 2024-08-20T21:51:25.5921597Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5922287Z "size": 565, 2024-08-20T21:51:25.5922947Z "digest": "sha256:7dd58e631031354e323ac12b1e8223aa4b198de49bd1ac0d10d11ee97933d4a8" 2024-08-20T21:51:25.5923689Z }, 2024-08-20T21:51:25.5923995Z { 2024-08-20T21:51:25.5924531Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5925205Z "size": 43162887, 2024-08-20T21:51:25.5925880Z "digest": "sha256:b125137ca92e3bd051d165ca69937cd4939aa82b5095a4379e3b940187fdb4fe" 2024-08-20T21:51:25.5926638Z }, 2024-08-20T21:51:25.5926912Z { 2024-08-20T21:51:25.5927444Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5928129Z "size": 104, 2024-08-20T21:51:25.5928781Z "digest": "sha256:12dfa73284bb2bfe40228d03e187cbf9f4d59c5300419db1d5d562ad9ca4a5cd" 2024-08-20T21:51:25.5929549Z }, 2024-08-20T21:51:25.5929842Z { 2024-08-20T21:51:25.5930359Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5931052Z "size": 1379, 2024-08-20T21:51:25.5931713Z "digest": "sha256:045427066d9a1cb897779196ada05b7b790c1dce05bead327f658f20933e9a76" 2024-08-20T21:51:25.5932456Z }, 2024-08-20T21:51:25.5932753Z { 2024-08-20T21:51:25.5933284Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5933954Z "size": 699, 2024-08-20T21:51:25.5934618Z "digest": "sha256:28e54511f1b116a9d632caa0c4be2cd1b6ca7542f815560ec598b81d6d956d31" 2024-08-20T21:51:25.5935388Z }, 2024-08-20T21:51:25.5935661Z { 2024-08-20T21:51:25.5936190Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5936878Z "size": 139, 2024-08-20T21:51:25.5937540Z "digest": "sha256:b6428379fcaafd1025a47060dccafde4f6ac9d22475fff6ce5e51fc531bd9d98" 2024-08-20T21:51:25.5938321Z }, 2024-08-20T21:51:25.5938614Z { 2024-08-20T21:51:25.5939124Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5939915Z "size": 119, 2024-08-20T21:51:25.5940640Z "digest": "sha256:7f8afc2d089702d4d576f8a6a8841b21e66c0acbf3b735f76a4fe1c4cb0f1fa2" 2024-08-20T21:51:25.5941399Z }, 2024-08-20T21:51:25.5941702Z { 2024-08-20T21:51:25.5942242Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5942919Z "size": 4327084570, 2024-08-20T21:51:25.5943620Z "digest": "sha256:ae2172ebb76524e0aabfc93971b239448922f1f340cb20dbe61460dbd250f202" 2024-08-20T21:51:25.5944387Z }, 2024-08-20T21:51:25.5944662Z { 2024-08-20T21:51:25.5945191Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5945874Z "size": 173, 2024-08-20T21:51:25.5946537Z "digest": "sha256:c6cba7526eea3b7a7febd01cb06e32f421ab15e17a3692967040d95eb0675cca" 2024-08-20T21:51:25.5947319Z }, 2024-08-20T21:51:25.5947614Z { 2024-08-20T21:51:25.5948127Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5948825Z "size": 907, 2024-08-20T21:51:25.5949493Z "digest": "sha256:5be7fac2e2a0876700857b9032bc92b1f3b8470f6bcff815e18101d7d71c4054" 2024-08-20T21:51:25.5950240Z }, 2024-08-20T21:51:25.5950536Z { 2024-08-20T21:51:25.5951069Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5951832Z "size": 699, 2024-08-20T21:51:25.5952519Z "digest": "sha256:28e54511f1b116a9d632caa0c4be2cd1b6ca7542f815560ec598b81d6d956d31" 2024-08-20T21:51:25.5953288Z }, 2024-08-20T21:51:25.5953564Z { 2024-08-20T21:51:25.5954096Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5954782Z "size": 134, 2024-08-20T21:51:25.5955433Z "digest": "sha256:eea60c1ffc991e92d764514ee25756132c369fa2fc0de806c6f1f8eb64718a5b" 2024-08-20T21:51:25.5956201Z }, 2024-08-20T21:51:25.5956494Z { 2024-08-20T21:51:25.5957004Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5957690Z "size": 32, 2024-08-20T21:51:25.5958367Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:51:25.5959123Z }, 2024-08-20T21:51:25.5959418Z { 2024-08-20T21:51:25.5959949Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5960621Z "size": 158, 2024-08-20T21:51:25.5961306Z "digest": "sha256:e4b1fa4f961f28a8004b20020e9f3e14ddda223a8cf6273a7ec5dfc7b58a9b04" 2024-08-20T21:51:25.5962080Z }, 2024-08-20T21:51:25.5962358Z { 2024-08-20T21:51:25.5962886Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5963573Z "size": 1841, 2024-08-20T21:51:25.5964243Z "digest": "sha256:ff20aefbe592a1fd39676d339cf9ebce003fde8a0b5702b8301d756bdee912c7" 2024-08-20T21:51:25.5965029Z }, 2024-08-20T21:51:25.5965324Z { 2024-08-20T21:51:25.5965834Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5966524Z "size": 7529768, 2024-08-20T21:51:25.5967200Z "digest": "sha256:39f017df093b119d169de8c649be9cf342a3075430b27bf2388dc9ccdefafac8" 2024-08-20T21:51:25.5967955Z }, 2024-08-20T21:51:25.5968250Z { 2024-08-20T21:51:25.5968776Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5970257Z "size": 163, 2024-08-20T21:51:25.5970944Z "digest": "sha256:dcb4b1c5a6f604842cbe094756013849c141d2c73b9764babb9b3cf35684a797" 2024-08-20T21:51:25.5971719Z }, 2024-08-20T21:51:25.5971994Z { 2024-08-20T21:51:25.5972526Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5973211Z "size": 7943, 2024-08-20T21:51:25.5973852Z "digest": "sha256:862b4f5c5cd13e6c7641578efd673953421540d1b64af3e8a362b709a3ef443a" 2024-08-20T21:51:25.5974614Z }, 2024-08-20T21:51:25.5974909Z { 2024-08-20T21:51:25.5975423Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5976112Z "size": 8070, 2024-08-20T21:51:25.5976770Z "digest": "sha256:46546d0a53bb910aa9a28f5073146d18a700ca0d1ca7407044085170059104cf" 2024-08-20T21:51:25.5977664Z }, 2024-08-20T21:51:25.5977964Z { 2024-08-20T21:51:25.5978496Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5979161Z "size": 301, 2024-08-20T21:51:25.5979846Z "digest": "sha256:82a31e7860af47e63f76dcfa09bf02b9ed5ba7f0c2c17804cbdd4e4f9dc7ab1d" 2024-08-20T21:51:25.5980628Z }, 2024-08-20T21:51:25.5980905Z { 2024-08-20T21:51:25.5981437Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5982128Z "size": 7629503, 2024-08-20T21:51:25.5982813Z "digest": "sha256:ac2f6809f82b0e504eb4e1bfada74e84542eaff1a26c06047ffecdcabc8f2f04" 2024-08-20T21:51:25.5983604Z }, 2024-08-20T21:51:25.5983898Z { 2024-08-20T21:51:25.5984410Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5985096Z "size": 108, 2024-08-20T21:51:25.5985764Z "digest": "sha256:2b45e44613094d77c76cfbe3d12bd0a314f3cfea24cf97303d9c6ee51715a021" 2024-08-20T21:51:25.5986524Z }, 2024-08-20T21:51:25.5986816Z { 2024-08-20T21:51:25.5987342Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5988007Z "size": 54145661, 2024-08-20T21:51:25.5988684Z "digest": "sha256:403462e9ee27186eada6b3b99891b31b4394b51e12d017f8fc1c302e33444a8a" 2024-08-20T21:51:25.5989558Z }, 2024-08-20T21:51:25.5989849Z { 2024-08-20T21:51:25.5990384Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5991078Z "size": 472, 2024-08-20T21:51:25.5991708Z "digest": "sha256:889f478638723a13f19e34f1f415d0c7dd7070c02484cb105d03601eb16e45f4" 2024-08-20T21:51:25.5992453Z }, 2024-08-20T21:51:25.5992734Z { 2024-08-20T21:51:25.5993250Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5993942Z "size": 1373229330, 2024-08-20T21:51:25.5994634Z "digest": "sha256:23898c5065b1e76bbf101c2151c1f06c50633abb686f1a87ee578de1acce2eaf" 2024-08-20T21:51:25.5995372Z }, 2024-08-20T21:51:25.5995673Z { 2024-08-20T21:51:25.5996195Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5996855Z "size": 106, 2024-08-20T21:51:25.5997519Z "digest": "sha256:f1caf060fc92271565fb07a66ff67d61a3bb761687913767cebd4240754275db" 2024-08-20T21:51:25.5998288Z }, 2024-08-20T21:51:25.5998570Z { 2024-08-20T21:51:25.5999105Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.5999794Z "size": 559, 2024-08-20T21:51:25.6000427Z "digest": "sha256:621b6cedbe16b797e97c910d222978f9f3077752e0386617189079966c30ee02" 2024-08-20T21:51:25.6001184Z }, 2024-08-20T21:51:25.6001476Z { 2024-08-20T21:51:25.6001985Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.6002674Z "size": 36532053, 2024-08-20T21:51:25.6003355Z "digest": "sha256:b9c038a9eb4ac830af69e080b96c2d591156429f179e4e922d025624341693d4" 2024-08-20T21:51:25.6004094Z }, 2024-08-20T21:51:25.6004384Z { 2024-08-20T21:51:25.6004915Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.6005582Z "size": 111, 2024-08-20T21:51:25.6006251Z "digest": "sha256:74046cd1f067b5edf32140ab31499a79acf8919e57afd4bc4debeafeb7f993ed" 2024-08-20T21:51:25.6007025Z }, 2024-08-20T21:51:25.6007333Z { 2024-08-20T21:51:25.6007873Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.6008561Z "size": 32, 2024-08-20T21:51:25.6009216Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:51:25.6009991Z }, 2024-08-20T21:51:25.6010286Z { 2024-08-20T21:51:25.6010800Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.6011488Z "size": 32, 2024-08-20T21:51:25.6012158Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:51:25.6012911Z }, 2024-08-20T21:51:25.6013203Z { 2024-08-20T21:51:25.6013730Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.6014501Z "size": 32, 2024-08-20T21:51:25.6015172Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:51:25.6015943Z }, 2024-08-20T21:51:25.6016220Z { 2024-08-20T21:51:25.6016758Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:25.6017449Z "size": 32, 2024-08-20T21:51:25.6018098Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:51:25.6018878Z } 2024-08-20T21:51:25.6019174Z ] 2024-08-20T21:51:25.6019448Z } 2024-08-20T21:51:25.6149751Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T21:51:25.6150363Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T21:51:25.6151034Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-20T21:51:25.6158565Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:25.6159164Z env: 2024-08-20T21:51:25.6159496Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:25.6160792Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:25.6162103Z ##[endgroup] 2024-08-20T21:51:25.6193387Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9-f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:25.6260372Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-20T21:51:25.6261114Z with: 2024-08-20T21:51:25.6262296Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:25.6263798Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:25.6264421Z env: 2024-08-20T21:51:25.6264753Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:25.6265175Z ##[endgroup] 2024-08-20T21:51:25.6289560Z ##[group]Run set -x 2024-08-20T21:51:25.6289969Z set -x 2024-08-20T21:51:25.6290325Z set +e 2024-08-20T21:51:25.6290677Z  2024-08-20T21:51:25.6291012Z login() { 2024-08-20T21:51:25.6291812Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T21:51:25.6292709Z } 2024-08-20T21:51:25.6293033Z  2024-08-20T21:51:25.6293401Z retry () { 2024-08-20T21:51:25.6293860Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T21:51:25.6294403Z } 2024-08-20T21:51:25.6294730Z  2024-08-20T21:51:25.6295088Z retry login "${DOCKER_REGISTRY}" 2024-08-20T21:51:25.6295593Z  2024-08-20T21:51:25.6295914Z set -e 2024-08-20T21:51:25.6296472Z # ignore output since only exit code is used for conditional 2024-08-20T21:51:25.6297342Z # only pull docker image if it's not available locally 2024-08-20T21:51:25.6298293Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-20T21:51:25.6299168Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-20T21:51:25.6299689Z fi 2024-08-20T21:51:25.6306364Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:25.6306979Z env: 2024-08-20T21:51:25.6307295Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:25.6308577Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:25.6310070Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:25.6310698Z ##[endgroup] 2024-08-20T21:51:25.6337430Z + set +e 2024-08-20T21:51:25.6338078Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:25.6338889Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:25.6341606Z + aws ecr get-login-password --region us-east-1 2024-08-20T21:51:25.6342934Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:26.2990521Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T21:51:26.2991558Z Configure a credential helper to remove this warning. See 2024-08-20T21:51:26.2992646Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T21:51:26.2993310Z 2024-08-20T21:51:26.2993497Z Login Succeeded 2024-08-20T21:51:26.3004283Z + set -e 2024-08-20T21:51:26.3007016Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:26.3177375Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:26.3179820Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:26.4632961Z f6d216893d65c7b8ae43df4daaf247db808378e9: Pulling from pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9 2024-08-20T21:51:26.4633997Z 7a2c55901189: Pulling fs layer 2024-08-20T21:51:26.4634473Z 09d415c238d7: Pulling fs layer 2024-08-20T21:51:26.4635040Z 9fe6e2e61518: Pulling fs layer 2024-08-20T21:51:26.4635926Z 41f16248e682: Pulling fs layer 2024-08-20T21:51:26.4636851Z 95d7b7817039: Pulling fs layer 2024-08-20T21:51:26.4637352Z 8f6c90485347: Pulling fs layer 2024-08-20T21:51:26.4637804Z ab17245097e4: Pulling fs layer 2024-08-20T21:51:26.4638524Z dfecd7e9912b: Pulling fs layer 2024-08-20T21:51:26.4638993Z 464a8f745445: Pulling fs layer 2024-08-20T21:51:26.4639437Z d67c111fa588: Pulling fs layer 2024-08-20T21:51:26.4639891Z 0723abde19c0: Pulling fs layer 2024-08-20T21:51:26.4640416Z 72d2b53f7a01: Pulling fs layer 2024-08-20T21:51:26.4640857Z dbc6fccd9c70: Pulling fs layer 2024-08-20T21:51:26.4641323Z ff6a6ddc7310: Pulling fs layer 2024-08-20T21:51:26.4641786Z fac2b1b246ed: Pulling fs layer 2024-08-20T21:51:26.4642245Z bc574f79516b: Pulling fs layer 2024-08-20T21:51:26.4643084Z c23d42adcade: Pulling fs layer 2024-08-20T21:51:26.4643569Z 59b0a331db9f: Pulling fs layer 2024-08-20T21:51:26.4644010Z 478a9a2bdefc: Pulling fs layer 2024-08-20T21:51:26.4644471Z 28e54511f1b1: Pulling fs layer 2024-08-20T21:51:26.4644923Z 5247f18de881: Pulling fs layer 2024-08-20T21:51:26.4645362Z 38760708e2d7: Pulling fs layer 2024-08-20T21:51:26.4645818Z 863dbde6ed0c: Pulling fs layer 2024-08-20T21:51:26.4646266Z dfecd7e9912b: Waiting 2024-08-20T21:51:26.4646658Z f283ab568a00: Pulling fs layer 2024-08-20T21:51:26.4647113Z 3e39f30b1d5c: Pulling fs layer 2024-08-20T21:51:26.4647547Z 464a8f745445: Waiting 2024-08-20T21:51:26.4647927Z 521c3d72534c: Pulling fs layer 2024-08-20T21:51:26.4648360Z 41f16248e682: Waiting 2024-08-20T21:51:26.4648764Z a975a4480a6d: Pulling fs layer 2024-08-20T21:51:26.4649183Z d67c111fa588: Waiting 2024-08-20T21:51:26.4649562Z 95d7b7817039: Waiting 2024-08-20T21:51:26.4649961Z 1f2c278212ad: Pulling fs layer 2024-08-20T21:51:26.4650386Z 8f6c90485347: Waiting 2024-08-20T21:51:26.4650766Z 0723abde19c0: Waiting 2024-08-20T21:51:26.4651165Z b721dd867e80: Pulling fs layer 2024-08-20T21:51:26.4651606Z 7a74b7ad649d: Pulling fs layer 2024-08-20T21:51:26.4652043Z ff6a6ddc7310: Waiting 2024-08-20T21:51:26.4652431Z dbc6fccd9c70: Waiting 2024-08-20T21:51:26.4652831Z b53e1ad1deec: Pulling fs layer 2024-08-20T21:51:26.4653271Z ab17245097e4: Waiting 2024-08-20T21:51:26.4653664Z fac2b1b246ed: Waiting 2024-08-20T21:51:26.4654028Z 72d2b53f7a01: Waiting 2024-08-20T21:51:26.4654403Z b721dd867e80: Waiting 2024-08-20T21:51:26.4654804Z 016dc3faa7ef: Pulling fs layer 2024-08-20T21:51:26.4655247Z 384dce3a4ba2: Pulling fs layer 2024-08-20T21:51:26.4655706Z 75461ad9f7bb: Pulling fs layer 2024-08-20T21:51:26.4656165Z eb2062102d11: Pulling fs layer 2024-08-20T21:51:26.4656585Z bc574f79516b: Waiting 2024-08-20T21:51:26.4656963Z 478a9a2bdefc: Waiting 2024-08-20T21:51:26.4657365Z 3cc17f105239: Pulling fs layer 2024-08-20T21:51:26.4657992Z 28e54511f1b1: Waiting 2024-08-20T21:51:26.4658380Z c23d42adcade: Waiting 2024-08-20T21:51:26.4658785Z 585501be0799: Pulling fs layer 2024-08-20T21:51:26.4659224Z 6c49da68c9b4: Pulling fs layer 2024-08-20T21:51:26.4659663Z 59b0a331db9f: Waiting 2024-08-20T21:51:26.4660038Z 497539689260: Pulling fs layer 2024-08-20T21:51:26.4660488Z 5843bb8b81bb: Pulling fs layer 2024-08-20T21:51:26.4660927Z 5247f18de881: Waiting 2024-08-20T21:51:26.4661306Z 222174e4e2b2: Pulling fs layer 2024-08-20T21:51:26.4661758Z 8d55064a9f20: Pulling fs layer 2024-08-20T21:51:26.4662194Z 3e39f30b1d5c: Waiting 2024-08-20T21:51:26.4662575Z 4f4fb700ef54: Pulling fs layer 2024-08-20T21:51:26.4663030Z 25114d5bae5b: Pulling fs layer 2024-08-20T21:51:26.4663481Z 7dd58e631031: Pulling fs layer 2024-08-20T21:51:26.4663900Z a975a4480a6d: Waiting 2024-08-20T21:51:26.4664336Z b125137ca92e: Pulling fs layer 2024-08-20T21:51:26.4664776Z 7a74b7ad649d: Waiting 2024-08-20T21:51:26.4665157Z 1f2c278212ad: Waiting 2024-08-20T21:51:26.4665541Z 12dfa73284bb: Pulling fs layer 2024-08-20T21:51:26.4666001Z 045427066d9a: Pulling fs layer 2024-08-20T21:51:26.4666434Z 3cc17f105239: Waiting 2024-08-20T21:51:26.4666802Z 016dc3faa7ef: Waiting 2024-08-20T21:51:26.4667192Z b53e1ad1deec: Waiting 2024-08-20T21:51:26.4667594Z b6428379fcaa: Pulling fs layer 2024-08-20T21:51:26.4668036Z 7f8afc2d0897: Pulling fs layer 2024-08-20T21:51:26.4668598Z 6c49da68c9b4: Waiting 2024-08-20T21:51:26.4669512Z ae2172ebb765: Pulling fs layer 2024-08-20T21:51:26.4670005Z c6cba7526eea: Pulling fs layer 2024-08-20T21:51:26.4670472Z 5be7fac2e2a0: Pulling fs layer 2024-08-20T21:51:26.4670918Z 8d55064a9f20: Waiting 2024-08-20T21:51:26.4671282Z f283ab568a00: Waiting 2024-08-20T21:51:26.4671686Z eea60c1ffc99: Pulling fs layer 2024-08-20T21:51:26.4672134Z 7dd58e631031: Waiting 2024-08-20T21:51:26.4672523Z e4b1fa4f961f: Pulling fs layer 2024-08-20T21:51:26.4672966Z 5843bb8b81bb: Waiting 2024-08-20T21:51:26.4673349Z b6428379fcaa: Waiting 2024-08-20T21:51:26.4673738Z ff20aefbe592: Pulling fs layer 2024-08-20T21:51:26.4674210Z 39f017df093b: Pulling fs layer 2024-08-20T21:51:26.4674651Z 7f8afc2d0897: Waiting 2024-08-20T21:51:26.4675040Z dcb4b1c5a6f6: Pulling fs layer 2024-08-20T21:51:26.4675484Z ae2172ebb765: Waiting 2024-08-20T21:51:26.4675869Z 862b4f5c5cd1: Pulling fs layer 2024-08-20T21:51:26.4676330Z 46546d0a53bb: Pulling fs layer 2024-08-20T21:51:26.4676772Z 25114d5bae5b: Waiting 2024-08-20T21:51:26.4677198Z 82a31e7860af: Pulling fs layer 2024-08-20T21:51:26.4678013Z b125137ca92e: Waiting 2024-08-20T21:51:26.4678661Z ff20aefbe592: Waiting 2024-08-20T21:51:26.4679308Z dcb4b1c5a6f6: Waiting 2024-08-20T21:51:26.4679958Z 39f017df093b: Waiting 2024-08-20T21:51:26.4680617Z ac2f6809f82b: Pulling fs layer 2024-08-20T21:51:26.4681329Z 862b4f5c5cd1: Waiting 2024-08-20T21:51:26.4681971Z 2b45e4461309: Pulling fs layer 2024-08-20T21:51:26.4682708Z 403462e9ee27: Pulling fs layer 2024-08-20T21:51:26.4683399Z 889f47863872: Pulling fs layer 2024-08-20T21:51:26.4684103Z 23898c5065b1: Pulling fs layer 2024-08-20T21:51:26.4684780Z 46546d0a53bb: Waiting 2024-08-20T21:51:26.4685347Z 82a31e7860af: Waiting 2024-08-20T21:51:26.4686073Z f1caf060fc92: Pulling fs layer 2024-08-20T21:51:26.4686935Z 23898c5065b1: Waiting 2024-08-20T21:51:26.4687682Z 521c3d72534c: Waiting 2024-08-20T21:51:26.4688443Z 621b6cedbe16: Pulling fs layer 2024-08-20T21:51:26.4689068Z eb2062102d11: Waiting 2024-08-20T21:51:26.4689467Z b9c038a9eb4a: Pulling fs layer 2024-08-20T21:51:26.4689914Z 621b6cedbe16: Waiting 2024-08-20T21:51:26.4690315Z 74046cd1f067: Pulling fs layer 2024-08-20T21:51:26.4690740Z 5be7fac2e2a0: Waiting 2024-08-20T21:51:26.4691124Z 045427066d9a: Waiting 2024-08-20T21:51:26.4691500Z 74046cd1f067: Waiting 2024-08-20T21:51:26.4691864Z ac2f6809f82b: Waiting 2024-08-20T21:51:26.4692240Z 75461ad9f7bb: Waiting 2024-08-20T21:51:26.4692621Z 4f4fb700ef54: Waiting 2024-08-20T21:51:26.4692976Z 889f47863872: Waiting 2024-08-20T21:51:26.4693347Z f1caf060fc92: Waiting 2024-08-20T21:51:26.4693725Z b9c038a9eb4a: Waiting 2024-08-20T21:51:26.4694086Z 12dfa73284bb: Waiting 2024-08-20T21:51:26.5965106Z 09d415c238d7: Verifying Checksum 2024-08-20T21:51:26.5965688Z 09d415c238d7: Download complete 2024-08-20T21:51:26.6865919Z 41f16248e682: Verifying Checksum 2024-08-20T21:51:26.6866439Z 41f16248e682: Download complete 2024-08-20T21:51:26.7648310Z 95d7b7817039: Download complete 2024-08-20T21:51:26.8146038Z 7a2c55901189: Verifying Checksum 2024-08-20T21:51:26.8147145Z 7a2c55901189: Download complete 2024-08-20T21:51:26.8935573Z ab17245097e4: Download complete 2024-08-20T21:51:26.9770858Z dfecd7e9912b: Verifying Checksum 2024-08-20T21:51:26.9771581Z dfecd7e9912b: Download complete 2024-08-20T21:51:27.0480195Z 464a8f745445: Download complete 2024-08-20T21:51:27.0794986Z 9fe6e2e61518: Verifying Checksum 2024-08-20T21:51:27.0796075Z 9fe6e2e61518: Download complete 2024-08-20T21:51:27.1610610Z 0723abde19c0: Verifying Checksum 2024-08-20T21:51:27.1611358Z 0723abde19c0: Download complete 2024-08-20T21:51:27.2399079Z 72d2b53f7a01: Download complete 2024-08-20T21:51:28.1079973Z 7a2c55901189: Pull complete 2024-08-20T21:51:28.4461839Z 09d415c238d7: Pull complete 2024-08-20T21:51:29.5078472Z 9fe6e2e61518: Pull complete 2024-08-20T21:51:29.5274221Z 41f16248e682: Pull complete 2024-08-20T21:51:29.5481764Z 95d7b7817039: Pull complete 2024-08-20T21:51:33.8433164Z dbc6fccd9c70: Verifying Checksum 2024-08-20T21:51:33.8434077Z dbc6fccd9c70: Download complete 2024-08-20T21:51:33.9137483Z ff6a6ddc7310: Download complete 2024-08-20T21:51:33.9888918Z fac2b1b246ed: Download complete 2024-08-20T21:51:34.0607305Z bc574f79516b: Verifying Checksum 2024-08-20T21:51:34.0608395Z bc574f79516b: Download complete 2024-08-20T21:51:35.0355404Z c23d42adcade: Verifying Checksum 2024-08-20T21:51:35.0356265Z c23d42adcade: Download complete 2024-08-20T21:51:35.1293585Z 59b0a331db9f: Download complete 2024-08-20T21:51:35.2044250Z 478a9a2bdefc: Verifying Checksum 2024-08-20T21:51:35.2045073Z 478a9a2bdefc: Download complete 2024-08-20T21:51:35.2824284Z 28e54511f1b1: Download complete 2024-08-20T21:51:40.6602270Z 8f6c90485347: Verifying Checksum 2024-08-20T21:51:40.6602897Z 8f6c90485347: Download complete 2024-08-20T21:51:40.7348584Z 38760708e2d7: Download complete 2024-08-20T21:51:40.8177879Z 863dbde6ed0c: Verifying Checksum 2024-08-20T21:51:40.8178722Z 863dbde6ed0c: Download complete 2024-08-20T21:51:40.8861748Z f283ab568a00: Download complete 2024-08-20T21:51:40.9559394Z 3e39f30b1d5c: Verifying Checksum 2024-08-20T21:51:40.9560265Z 3e39f30b1d5c: Download complete 2024-08-20T21:51:41.0721598Z 521c3d72534c: Verifying Checksum 2024-08-20T21:51:41.0722332Z 521c3d72534c: Download complete 2024-08-20T21:51:42.3424950Z a975a4480a6d: Verifying Checksum 2024-08-20T21:51:42.3425900Z a975a4480a6d: Download complete 2024-08-20T21:51:42.4307739Z 1f2c278212ad: Verifying Checksum 2024-08-20T21:51:42.4308350Z 1f2c278212ad: Download complete 2024-08-20T21:51:42.5077214Z b721dd867e80: Download complete 2024-08-20T21:51:42.5845412Z 7a74b7ad649d: Verifying Checksum 2024-08-20T21:51:42.5846502Z 7a74b7ad649d: Download complete 2024-08-20T21:51:42.6666838Z b53e1ad1deec: Download complete 2024-08-20T21:51:42.7426651Z 016dc3faa7ef: Verifying Checksum 2024-08-20T21:51:42.7427461Z 016dc3faa7ef: Download complete 2024-08-20T21:51:48.7518270Z 384dce3a4ba2: Verifying Checksum 2024-08-20T21:51:48.7518871Z 384dce3a4ba2: Download complete 2024-08-20T21:51:48.8319560Z 75461ad9f7bb: Verifying Checksum 2024-08-20T21:51:48.8320546Z 75461ad9f7bb: Download complete 2024-08-20T21:51:48.9156677Z eb2062102d11: Verifying Checksum 2024-08-20T21:51:48.9157275Z eb2062102d11: Download complete 2024-08-20T21:51:49.3690792Z 3cc17f105239: Verifying Checksum 2024-08-20T21:51:49.3691412Z 3cc17f105239: Download complete 2024-08-20T21:51:49.4811414Z 585501be0799: Download complete 2024-08-20T21:51:49.5677006Z 6c49da68c9b4: Verifying Checksum 2024-08-20T21:51:49.5677581Z 6c49da68c9b4: Download complete 2024-08-20T21:51:49.8252632Z 497539689260: Verifying Checksum 2024-08-20T21:51:49.8253217Z 497539689260: Download complete 2024-08-20T21:51:49.9235044Z 5843bb8b81bb: Download complete 2024-08-20T21:51:50.0049775Z 222174e4e2b2: Verifying Checksum 2024-08-20T21:51:50.0050357Z 222174e4e2b2: Download complete 2024-08-20T21:51:50.0830778Z 8d55064a9f20: Verifying Checksum 2024-08-20T21:51:50.0831338Z 8d55064a9f20: Download complete 2024-08-20T21:51:50.0950851Z 4f4fb700ef54: Verifying Checksum 2024-08-20T21:51:50.0951374Z 4f4fb700ef54: Download complete 2024-08-20T21:51:50.1826501Z 25114d5bae5b: Verifying Checksum 2024-08-20T21:51:50.1827117Z 25114d5bae5b: Download complete 2024-08-20T21:51:50.8185105Z b125137ca92e: Verifying Checksum 2024-08-20T21:51:50.8185700Z b125137ca92e: Download complete 2024-08-20T21:51:50.9067410Z 12dfa73284bb: Verifying Checksum 2024-08-20T21:51:50.9067997Z 12dfa73284bb: Download complete 2024-08-20T21:51:50.9780643Z 045427066d9a: Verifying Checksum 2024-08-20T21:51:50.9781213Z 045427066d9a: Download complete 2024-08-20T21:51:51.0629580Z b6428379fcaa: Verifying Checksum 2024-08-20T21:51:51.0630171Z b6428379fcaa: Download complete 2024-08-20T21:51:51.1422727Z 7f8afc2d0897: Verifying Checksum 2024-08-20T21:51:51.1423318Z 7f8afc2d0897: Download complete 2024-08-20T21:51:51.5533043Z d67c111fa588: Verifying Checksum 2024-08-20T21:51:51.5533647Z d67c111fa588: Download complete 2024-08-20T21:51:51.6330386Z c6cba7526eea: Download complete 2024-08-20T21:51:51.7155797Z 5be7fac2e2a0: Verifying Checksum 2024-08-20T21:51:51.7156660Z 5be7fac2e2a0: Download complete 2024-08-20T21:51:51.7920205Z eea60c1ffc99: Verifying Checksum 2024-08-20T21:51:51.7920775Z eea60c1ffc99: Download complete 2024-08-20T21:51:51.8542044Z e4b1fa4f961f: Verifying Checksum 2024-08-20T21:51:51.8543120Z e4b1fa4f961f: Download complete 2024-08-20T21:51:51.9460075Z ff20aefbe592: Download complete 2024-08-20T21:51:52.0824292Z 39f017df093b: Verifying Checksum 2024-08-20T21:51:52.0824879Z 39f017df093b: Download complete 2024-08-20T21:51:52.1751206Z dcb4b1c5a6f6: Verifying Checksum 2024-08-20T21:51:52.1751811Z dcb4b1c5a6f6: Download complete 2024-08-20T21:51:52.2432757Z 862b4f5c5cd1: Download complete 2024-08-20T21:51:52.3477656Z 46546d0a53bb: Verifying Checksum 2024-08-20T21:51:52.3478234Z 46546d0a53bb: Download complete 2024-08-20T21:51:52.4200643Z 82a31e7860af: Download complete 2024-08-20T21:51:52.5496536Z ac2f6809f82b: Verifying Checksum 2024-08-20T21:51:52.5497127Z ac2f6809f82b: Download complete 2024-08-20T21:51:52.6275640Z 2b45e4461309: Verifying Checksum 2024-08-20T21:51:52.6276217Z 2b45e4461309: Download complete 2024-08-20T21:51:53.2249732Z 403462e9ee27: Verifying Checksum 2024-08-20T21:51:53.2250343Z 403462e9ee27: Download complete 2024-08-20T21:51:53.2923540Z 889f47863872: Verifying Checksum 2024-08-20T21:51:53.2924582Z 889f47863872: Download complete 2024-08-20T21:51:57.9347615Z 8f6c90485347: Pull complete 2024-08-20T21:51:58.3796765Z ab17245097e4: Pull complete 2024-08-20T21:51:58.8204315Z dfecd7e9912b: Pull complete 2024-08-20T21:51:59.2681253Z 464a8f745445: Pull complete 2024-08-20T21:52:04.2546772Z 5247f18de881: Verifying Checksum 2024-08-20T21:52:04.2547388Z 5247f18de881: Download complete 2024-08-20T21:52:04.3464727Z f1caf060fc92: Download complete 2024-08-20T21:52:04.4215687Z 621b6cedbe16: Verifying Checksum 2024-08-20T21:52:04.4216282Z 621b6cedbe16: Download complete 2024-08-20T21:52:04.8463752Z b9c038a9eb4a: Verifying Checksum 2024-08-20T21:52:04.8464347Z b9c038a9eb4a: Download complete 2024-08-20T21:52:04.9275524Z 74046cd1f067: Verifying Checksum 2024-08-20T21:52:04.9276136Z 74046cd1f067: Download complete 2024-08-20T21:52:07.0882948Z 23898c5065b1: Verifying Checksum 2024-08-20T21:52:07.0900616Z 23898c5065b1: Download complete 2024-08-20T21:52:33.5233813Z d67c111fa588: Pull complete 2024-08-20T21:52:33.9701506Z 0723abde19c0: Pull complete 2024-08-20T21:52:34.4163737Z 72d2b53f7a01: Pull complete 2024-08-20T21:52:34.4661922Z ae2172ebb765: Verifying Checksum 2024-08-20T21:52:34.4662518Z ae2172ebb765: Download complete 2024-08-20T21:52:48.1632812Z dbc6fccd9c70: Pull complete 2024-08-20T21:52:48.6077422Z ff6a6ddc7310: Pull complete 2024-08-20T21:52:49.0573777Z fac2b1b246ed: Pull complete 2024-08-20T21:52:49.4517657Z bc574f79516b: Pull complete 2024-08-20T21:52:52.5908192Z c23d42adcade: Pull complete 2024-08-20T21:52:52.9713074Z 59b0a331db9f: Pull complete 2024-08-20T21:52:53.3491400Z 478a9a2bdefc: Pull complete 2024-08-20T21:52:53.6825750Z 28e54511f1b1: Pull complete 2024-08-20T21:53:58.4313878Z 5247f18de881: Pull complete 2024-08-20T21:53:58.8828535Z 38760708e2d7: Pull complete 2024-08-20T21:53:59.2044854Z 863dbde6ed0c: Pull complete 2024-08-20T21:53:59.5041774Z f283ab568a00: Pull complete 2024-08-20T21:53:59.7950946Z 3e39f30b1d5c: Pull complete 2024-08-20T21:54:00.0866450Z 521c3d72534c: Pull complete 2024-08-20T21:54:04.5167422Z a975a4480a6d: Pull complete 2024-08-20T21:54:04.9727803Z 1f2c278212ad: Pull complete 2024-08-20T21:54:05.3409382Z b721dd867e80: Pull complete 2024-08-20T21:54:05.6530786Z 7a74b7ad649d: Pull complete 2024-08-20T21:54:05.8504347Z b53e1ad1deec: Pull complete 2024-08-20T21:54:06.0736336Z 016dc3faa7ef: Pull complete 2024-08-20T21:54:18.0522284Z 384dce3a4ba2: Pull complete 2024-08-20T21:54:18.2358451Z 75461ad9f7bb: Pull complete 2024-08-20T21:54:18.4525960Z eb2062102d11: Pull complete 2024-08-20T21:54:19.6786448Z 3cc17f105239: Pull complete 2024-08-20T21:54:20.1265851Z 585501be0799: Pull complete 2024-08-20T21:54:20.5725724Z 6c49da68c9b4: Pull complete 2024-08-20T21:54:21.2220836Z 497539689260: Pull complete 2024-08-20T21:54:21.5442195Z 5843bb8b81bb: Pull complete 2024-08-20T21:54:22.4500229Z 222174e4e2b2: Pull complete 2024-08-20T21:54:22.8529982Z 8d55064a9f20: Pull complete 2024-08-20T21:54:23.2887407Z 4f4fb700ef54: Pull complete 2024-08-20T21:54:23.7233336Z 25114d5bae5b: Pull complete 2024-08-20T21:54:24.1731173Z 7dd58e631031: Pull complete 2024-08-20T21:54:27.1988144Z b125137ca92e: Pull complete 2024-08-20T21:54:27.5556097Z 12dfa73284bb: Pull complete 2024-08-20T21:54:27.9604594Z 045427066d9a: Pull complete 2024-08-20T21:54:28.6982725Z b6428379fcaa: Pull complete 2024-08-20T21:54:29.0838604Z 7f8afc2d0897: Pull complete 2024-08-20T21:56:04.7658742Z ae2172ebb765: Pull complete 2024-08-20T21:56:05.2253721Z c6cba7526eea: Pull complete 2024-08-20T21:56:05.6792288Z 5be7fac2e2a0: Pull complete 2024-08-20T21:56:06.5697933Z eea60c1ffc99: Pull complete 2024-08-20T21:56:07.4849352Z e4b1fa4f961f: Pull complete 2024-08-20T21:56:07.9345238Z ff20aefbe592: Pull complete 2024-08-20T21:56:08.5650893Z 39f017df093b: Pull complete 2024-08-20T21:56:09.0218912Z dcb4b1c5a6f6: Pull complete 2024-08-20T21:56:09.4755425Z 862b4f5c5cd1: Pull complete 2024-08-20T21:56:09.9168152Z 46546d0a53bb: Pull complete 2024-08-20T21:56:10.3380038Z 82a31e7860af: Pull complete 2024-08-20T21:56:12.1197364Z ac2f6809f82b: Pull complete 2024-08-20T21:56:12.5526248Z 2b45e4461309: Pull complete 2024-08-20T21:56:15.2501188Z 403462e9ee27: Pull complete 2024-08-20T21:56:15.7009757Z 889f47863872: Pull complete 2024-08-20T21:56:31.1390096Z 23898c5065b1: Pull complete 2024-08-20T21:56:31.4041511Z f1caf060fc92: Pull complete 2024-08-20T21:56:31.6722799Z 621b6cedbe16: Pull complete 2024-08-20T21:56:32.6254539Z b9c038a9eb4a: Pull complete 2024-08-20T21:56:33.0731274Z 74046cd1f067: Pull complete 2024-08-20T21:56:35.0992091Z Digest: sha256:0b3aa0836d4f18656c98dcf355b46cbae2123243424ebbed0929206c3d798126 2024-08-20T21:56:35.1932990Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:56:35.2480214Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:56:35.2553180Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T21:56:35.2554383Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T21:56:35.2564485Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:56:35.2565125Z env: 2024-08-20T21:56:35.2565446Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:56:35.2565868Z ##[endgroup] 2024-08-20T21:56:35.2793490Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-08-20T21:56:35.2794202Z with: 2024-08-20T21:56:35.2794528Z driver-version: 550.54.15 2024-08-20T21:56:35.2794947Z env: 2024-08-20T21:56:35.2795259Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:56:35.2795681Z ##[endgroup] 2024-08-20T21:56:35.3074621Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T21:56:35.3075322Z with: 2024-08-20T21:56:35.3075638Z timeout_minutes: 10 2024-08-20T21:56:35.3076036Z max_attempts: 3 2024-08-20T21:56:35.3118984Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-08-20T21:56:35.3162024Z retry_wait_seconds: 10 2024-08-20T21:56:35.3162486Z polling_interval_seconds: 1 2024-08-20T21:56:35.3162925Z warning_on_retry: true 2024-08-20T21:56:35.3163352Z continue_on_error: false 2024-08-20T21:56:35.3163757Z env: 2024-08-20T21:56:35.3164066Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:56:35.3164500Z DRIVER_VERSION: 550.54.15 2024-08-20T21:56:35.3164920Z ##[endgroup] 2024-08-20T21:56:35.4142514Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-08-20T21:56:35.4144015Z + pre_install_nvidia_driver_amzn2 2024-08-20T21:56:35.4145322Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-08-20T21:56:35.8338614Z No match for argument: nvidia-driver-latest-dkms 2024-08-20T21:56:35.8339282Z No packages marked for removal. 2024-08-20T21:56:35.8424035Z Dependencies resolved. 2024-08-20T21:56:35.8437862Z Nothing to do. 2024-08-20T21:56:35.8438577Z Complete! 2024-08-20T21:56:35.9661499Z + install_nvidia_driver_common 2024-08-20T21:56:35.9665006Z + echo 'Before installing NVIDIA driver' 2024-08-20T21:56:35.9665578Z Before installing NVIDIA driver 2024-08-20T21:56:35.9667366Z + lspci 2024-08-20T21:56:35.9862403Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-08-20T21:56:35.9863426Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T21:56:35.9864403Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-08-20T21:56:35.9865349Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-08-20T21:56:35.9866156Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-08-20T21:56:35.9867018Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T21:56:35.9868050Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-20T21:56:35.9869507Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-20T21:56:35.9870825Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-08-20T21:56:35.9871785Z + lsmod 2024-08-20T21:56:35.9907111Z Module Size Used by 2024-08-20T21:56:35.9907610Z ib_core 454656 0 2024-08-20T21:56:35.9908078Z veth 36864 0 2024-08-20T21:56:35.9908527Z nvidia_modeset 1351680 0 2024-08-20T21:56:35.9909038Z video 65536 1 nvidia_modeset 2024-08-20T21:56:35.9909575Z wmi 36864 1 video 2024-08-20T21:56:35.9910063Z nvidia_uvm 4706304 0 2024-08-20T21:56:35.9910608Z nvidia 54071296 11 nvidia_uvm,nvidia_modeset 2024-08-20T21:56:35.9911220Z drm 602112 1 nvidia 2024-08-20T21:56:35.9911778Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T21:56:35.9912444Z backlight 24576 3 video,drm,nvidia_modeset 2024-08-20T21:56:35.9913064Z i2c_core 106496 2 nvidia,drm 2024-08-20T21:56:35.9913595Z xt_conntrack 16384 1 2024-08-20T21:56:35.9914067Z nft_chain_nat 16384 3 2024-08-20T21:56:35.9914536Z xt_MASQUERADE 20480 1 2024-08-20T21:56:35.9915104Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T21:56:35.9915718Z nf_conntrack_netlink 57344 0 2024-08-20T21:56:35.9916426Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T21:56:35.9917229Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T21:56:35.9917807Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T21:56:35.9918322Z xfrm_user 57344 1 2024-08-20T21:56:35.9918830Z xfrm_algo 16384 1 xfrm_user 2024-08-20T21:56:35.9919349Z xt_addrtype 16384 2 2024-08-20T21:56:35.9919797Z nft_compat 20480 4 2024-08-20T21:56:35.9920366Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T21:56:35.9921132Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T21:56:35.9921808Z br_netfilter 36864 0 2024-08-20T21:56:35.9922299Z bridge 307200 1 br_netfilter 2024-08-20T21:56:35.9922856Z stp 16384 1 bridge 2024-08-20T21:56:35.9923353Z llc 16384 2 bridge,stp 2024-08-20T21:56:35.9923864Z overlay 167936 0 2024-08-20T21:56:35.9924321Z tls 114688 0 2024-08-20T21:56:35.9924759Z ata_piix 45056 0 2024-08-20T21:56:35.9925232Z libata 401408 1 ata_piix 2024-08-20T21:56:35.9925746Z scsi_mod 290816 1 libata 2024-08-20T21:56:35.9926226Z nls_ascii 16384 1 2024-08-20T21:56:35.9926690Z ghash_clmulni_intel 16384 0 2024-08-20T21:56:35.9927155Z ena 167936 0 2024-08-20T21:56:35.9927584Z nls_cp437 20480 1 2024-08-20T21:56:35.9928094Z scsi_common 16384 2 scsi_mod,libata 2024-08-20T21:56:35.9928643Z ptp 36864 1 ena 2024-08-20T21:56:35.9929094Z i8042 45056 0 2024-08-20T21:56:35.9929550Z pps_core 24576 1 ptp 2024-08-20T21:56:35.9930030Z vfat 24576 1 2024-08-20T21:56:35.9930470Z aesni_intel 393216 0 2024-08-20T21:56:35.9930938Z serio 28672 3 i8042 2024-08-20T21:56:35.9931420Z fat 86016 1 vfat 2024-08-20T21:56:35.9931920Z crypto_simd 16384 1 aesni_intel 2024-08-20T21:56:35.9932566Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T21:56:35.9933179Z button 24576 0 2024-08-20T21:56:35.9933612Z sunrpc 692224 1 2024-08-20T21:56:35.9934067Z sch_fq_codel 20480 9 2024-08-20T21:56:35.9934517Z dm_mod 188416 0 2024-08-20T21:56:35.9934962Z fuse 163840 1 2024-08-20T21:56:35.9935400Z dax 45056 1 dm_mod 2024-08-20T21:56:35.9935882Z loop 36864 0 2024-08-20T21:56:35.9936330Z configfs 57344 1 2024-08-20T21:56:35.9936760Z dmi_sysfs 20480 0 2024-08-20T21:56:35.9937207Z crc32_pclmul 16384 0 2024-08-20T21:56:35.9937889Z crc32c_intel 24576 0 2024-08-20T21:56:35.9938332Z + modinfo nvidia 2024-08-20T21:56:35.9939211Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T21:56:35.9940101Z alias: char-major-195-* 2024-08-20T21:56:35.9940578Z version: 550.54.15 2024-08-20T21:56:35.9940986Z supported: external 2024-08-20T21:56:35.9941404Z license: NVIDIA 2024-08-20T21:56:35.9941871Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T21:56:35.9942465Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T21:56:35.9943035Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T21:56:35.9943591Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T21:56:35.9944204Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T21:56:35.9944810Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T21:56:35.9945404Z depends: i2c-core,drm 2024-08-20T21:56:35.9945847Z retpoline: Y 2024-08-20T21:56:35.9946227Z name: nvidia 2024-08-20T21:56:35.9946985Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T21:56:35.9947863Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T21:56:35.9948674Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T21:56:35.9949400Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T21:56:35.9949946Z parm: NVreg_RmLogonRC:int 2024-08-20T21:56:35.9950483Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T21:56:35.9951028Z parm: NVreg_DeviceFileUID:int 2024-08-20T21:56:35.9951559Z parm: NVreg_DeviceFileGID:int 2024-08-20T21:56:35.9952097Z parm: NVreg_DeviceFileMode:int 2024-08-20T21:56:35.9952735Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T21:56:35.9953434Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T21:56:35.9954025Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T21:56:35.9954548Z parm: NVreg_EnableMSI:int 2024-08-20T21:56:35.9955069Z parm: NVreg_TCEBypassMode:int 2024-08-20T21:56:35.9955635Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T21:56:35.9956282Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T21:56:35.9957007Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T21:56:35.9957708Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T21:56:35.9958452Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T21:56:35.9959201Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T21:56:35.9959972Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T21:56:35.9960706Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T21:56:35.9961317Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T21:56:35.9961964Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T21:56:35.9962649Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T21:56:35.9963273Z parm: NVreg_MemoryPoolSize:int 2024-08-20T21:56:35.9963841Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T21:56:35.9964444Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T21:56:35.9965025Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T21:56:35.9965560Z parm: NVreg_NvLinkDisable:int 2024-08-20T21:56:35.9966175Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T21:56:35.9966832Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T21:56:35.9967423Z parm: NVreg_EnableResizableBar:int 2024-08-20T21:56:35.9968027Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T21:56:35.9968632Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T21:56:35.9969667Z parm: NVreg_RegistryDwords:charp 2024-08-20T21:56:35.9970291Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T21:56:35.9970863Z parm: NVreg_RmMsg:charp 2024-08-20T21:56:35.9971403Z parm: NVreg_GpuBlacklist:charp 2024-08-20T21:56:35.9972136Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T21:56:35.9972855Z parm: NVreg_ExcludedGpus:charp 2024-08-20T21:56:35.9973416Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T21:56:35.9973988Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T21:56:35.9974576Z parm: NVreg_ImexChannelCount:int 2024-08-20T21:56:35.9975137Z parm: rm_firmware_active:charp 2024-08-20T21:56:35.9975628Z + HAS_NVIDIA_DRIVER=0 2024-08-20T21:56:35.9976113Z ++ command -v nvidia-smi 2024-08-20T21:56:35.9976609Z + '[' -x /usr/bin/nvidia-smi ']' 2024-08-20T21:56:35.9977043Z + set +e 2024-08-20T21:56:35.9977684Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-08-20T21:56:36.0201830Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-08-20T21:56:36.0202344Z + NVIDIA_SMI_STATUS=0 2024-08-20T21:56:36.0202851Z + '[' 0 -ne 0 ']' 2024-08-20T21:56:36.0203297Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-08-20T21:56:36.0203747Z + HAS_NVIDIA_DRIVER=1 2024-08-20T21:56:36.0204689Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-08-20T21:56:36.0205598Z + set -e 2024-08-20T21:56:36.0205949Z + '[' 1 -eq 0 ']' 2024-08-20T21:56:36.0206660Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-08-20T21:56:36.0207519Z + post_install_nvidia_driver_common 2024-08-20T21:56:36.0208386Z + sudo modprobe nvidia 2024-08-20T21:56:36.1774497Z + echo 'After installing NVIDIA driver' 2024-08-20T21:56:36.1775043Z + lspci 2024-08-20T21:56:36.1775403Z After installing NVIDIA driver 2024-08-20T21:56:36.1964185Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-08-20T21:56:36.1965290Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T21:56:36.1966264Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-08-20T21:56:36.1967208Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-08-20T21:56:36.1968037Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-08-20T21:56:36.1968903Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T21:56:36.1970273Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-20T21:56:36.1971370Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-20T21:56:36.1972418Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-08-20T21:56:36.1973141Z + lsmod 2024-08-20T21:56:36.1995339Z Module Size Used by 2024-08-20T21:56:36.1996031Z ib_core 454656 0 2024-08-20T21:56:36.1996979Z veth 36864 0 2024-08-20T21:56:36.1997922Z nvidia_modeset 1351680 0 2024-08-20T21:56:36.1998862Z video 65536 1 nvidia_modeset 2024-08-20T21:56:36.1999394Z wmi 36864 1 video 2024-08-20T21:56:36.1999880Z nvidia_uvm 4706304 0 2024-08-20T21:56:36.2000455Z nvidia 54071296 11 nvidia_uvm,nvidia_modeset 2024-08-20T21:56:36.2001048Z drm 602112 1 nvidia 2024-08-20T21:56:36.2001596Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T21:56:36.2002277Z backlight 24576 3 video,drm,nvidia_modeset 2024-08-20T21:56:36.2002911Z i2c_core 106496 2 nvidia,drm 2024-08-20T21:56:36.2003418Z xt_conntrack 16384 1 2024-08-20T21:56:36.2003884Z nft_chain_nat 16384 3 2024-08-20T21:56:36.2004348Z xt_MASQUERADE 20480 1 2024-08-20T21:56:36.2004888Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T21:56:36.2005492Z nf_conntrack_netlink 57344 0 2024-08-20T21:56:36.2006214Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T21:56:36.2006997Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T21:56:36.2007564Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T21:56:36.2008096Z xfrm_user 57344 1 2024-08-20T21:56:36.2008944Z xfrm_algo 16384 1 xfrm_user 2024-08-20T21:56:36.2009479Z xt_addrtype 16384 2 2024-08-20T21:56:36.2009942Z nft_compat 20480 4 2024-08-20T21:56:36.2010485Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T21:56:36.2011246Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T21:56:36.2012091Z br_netfilter 36864 0 2024-08-20T21:56:36.2012806Z bridge 307200 1 br_netfilter 2024-08-20T21:56:36.2013514Z stp 16384 1 bridge 2024-08-20T21:56:36.2014129Z llc 16384 2 bridge,stp 2024-08-20T21:56:36.2014626Z overlay 167936 0 2024-08-20T21:56:36.2015081Z tls 114688 0 2024-08-20T21:56:36.2015534Z ata_piix 45056 0 2024-08-20T21:56:36.2015991Z libata 401408 1 ata_piix 2024-08-20T21:56:36.2016509Z scsi_mod 290816 1 libata 2024-08-20T21:56:36.2017010Z nls_ascii 16384 1 2024-08-20T21:56:36.2017467Z ghash_clmulni_intel 16384 0 2024-08-20T21:56:36.2017926Z ena 167936 0 2024-08-20T21:56:36.2018370Z nls_cp437 20480 1 2024-08-20T21:56:36.2018863Z scsi_common 16384 2 scsi_mod,libata 2024-08-20T21:56:36.2019410Z ptp 36864 1 ena 2024-08-20T21:56:36.2019877Z i8042 45056 0 2024-08-20T21:56:36.2020329Z pps_core 24576 1 ptp 2024-08-20T21:56:36.2020789Z vfat 24576 1 2024-08-20T21:56:36.2021244Z aesni_intel 393216 0 2024-08-20T21:56:36.2021704Z serio 28672 3 i8042 2024-08-20T21:56:36.2022183Z fat 86016 1 vfat 2024-08-20T21:56:36.2022687Z crypto_simd 16384 1 aesni_intel 2024-08-20T21:56:36.2023348Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T21:56:36.2023944Z button 24576 0 2024-08-20T21:56:36.2024402Z sunrpc 692224 1 2024-08-20T21:56:36.2024865Z sch_fq_codel 20480 9 2024-08-20T21:56:36.2025307Z dm_mod 188416 0 2024-08-20T21:56:36.2025755Z fuse 163840 1 2024-08-20T21:56:36.2026216Z dax 45056 1 dm_mod 2024-08-20T21:56:36.2026691Z loop 36864 0 2024-08-20T21:56:36.2027144Z configfs 57344 1 2024-08-20T21:56:36.2027582Z dmi_sysfs 20480 0 2024-08-20T21:56:36.2028039Z crc32_pclmul 16384 0 2024-08-20T21:56:36.2028500Z crc32c_intel 24576 0 2024-08-20T21:56:36.2028927Z + modinfo nvidia 2024-08-20T21:56:36.2029816Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T21:56:36.2030697Z alias: char-major-195-* 2024-08-20T21:56:36.2031166Z version: 550.54.15 2024-08-20T21:56:36.2031580Z supported: external 2024-08-20T21:56:36.2031996Z license: NVIDIA 2024-08-20T21:56:36.2032446Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T21:56:36.2033046Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T21:56:36.2033616Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T21:56:36.2034186Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T21:56:36.2034798Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T21:56:36.2035397Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T21:56:36.2036010Z depends: i2c-core,drm 2024-08-20T21:56:36.2036445Z retpoline: Y 2024-08-20T21:56:36.2036814Z name: nvidia 2024-08-20T21:56:36.2037584Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T21:56:36.2038446Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T21:56:36.2039252Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T21:56:36.2039999Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T21:56:36.2040528Z parm: NVreg_RmLogonRC:int 2024-08-20T21:56:36.2041187Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T21:56:36.2041869Z parm: NVreg_DeviceFileUID:int 2024-08-20T21:56:36.2042392Z parm: NVreg_DeviceFileGID:int 2024-08-20T21:56:36.2042938Z parm: NVreg_DeviceFileMode:int 2024-08-20T21:56:36.2043597Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T21:56:36.2044287Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T21:56:36.2044883Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T21:56:36.2045415Z parm: NVreg_EnableMSI:int 2024-08-20T21:56:36.2045914Z parm: NVreg_TCEBypassMode:int 2024-08-20T21:56:36.2046485Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T21:56:36.2047148Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T21:56:36.2047861Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T21:56:36.2048562Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T21:56:36.2049325Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T21:56:36.2050067Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T21:56:36.2050845Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T21:56:36.2051588Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T21:56:36.2052184Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T21:56:36.2052850Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T21:56:36.2053530Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T21:56:36.2054129Z parm: NVreg_MemoryPoolSize:int 2024-08-20T21:56:36.2054704Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T21:56:36.2055304Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T21:56:36.2055891Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T21:56:36.2056434Z parm: NVreg_NvLinkDisable:int 2024-08-20T21:56:36.2057061Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T21:56:36.2057703Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T21:56:36.2058307Z parm: NVreg_EnableResizableBar:int 2024-08-20T21:56:36.2058917Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T21:56:36.2059520Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T21:56:36.2060120Z parm: NVreg_RegistryDwords:charp 2024-08-20T21:56:36.2060736Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T21:56:36.2061326Z parm: NVreg_RmMsg:charp 2024-08-20T21:56:36.2061834Z parm: NVreg_GpuBlacklist:charp 2024-08-20T21:56:36.2062411Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T21:56:36.2062992Z parm: NVreg_ExcludedGpus:charp 2024-08-20T21:56:36.2063558Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T21:56:36.2064151Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T21:56:36.2064726Z parm: NVreg_ImexChannelCount:int 2024-08-20T21:56:36.2065287Z parm: rm_firmware_active:charp 2024-08-20T21:56:36.2065779Z + set +e 2024-08-20T21:56:36.2066142Z + nvidia-smi 2024-08-20T21:56:36.2280477Z Tue Aug 20 21:56:36 2024 2024-08-20T21:56:36.2281276Z +-----------------------------------------------------------------------------------------+ 2024-08-20T21:56:36.2282316Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T21:56:36.2283309Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T21:56:36.2284327Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T21:56:36.2285470Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T21:56:36.2286341Z | | | MIG M. | 2024-08-20T21:56:36.2287012Z |=========================================+========================+======================| 2024-08-20T21:56:36.2367766Z | 0 Tesla M60 On | 00000000:00:1D.0 Off | 0 | 2024-08-20T21:56:36.2368872Z | N/A 30C P8 14W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-20T21:56:36.2370152Z | | | N/A | 2024-08-20T21:56:36.2371041Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T21:56:36.2435381Z | 1 Tesla M60 On | 00000000:00:1E.0 Off | 0 | 2024-08-20T21:56:36.2436286Z | N/A 32C P8 14W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-20T21:56:36.2437074Z | | | N/A | 2024-08-20T21:56:36.2437946Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T21:56:36.2438689Z 2024-08-20T21:56:36.2439547Z +-----------------------------------------------------------------------------------------+ 2024-08-20T21:56:36.2440335Z | Processes: | 2024-08-20T21:56:36.2441224Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T21:56:36.2442100Z | ID ID Usage | 2024-08-20T21:56:36.2442788Z |=========================================================================================| 2024-08-20T21:56:36.2443614Z | No running processes found | 2024-08-20T21:56:36.2444587Z +-----------------------------------------------------------------------------------------+ 2024-08-20T21:56:36.3586653Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-08-20T21:56:36.3773898Z Tesla M60 2024-08-20T21:56:36.3842486Z + NVIDIA_SMI_STATUS=0 2024-08-20T21:56:36.3843013Z + '[' 0 -eq 0 ']' 2024-08-20T21:56:36.3843514Z + echo 'INFO: Ignoring allowed status 0' 2024-08-20T21:56:36.3844057Z + set -e 2024-08-20T21:56:36.3844408Z INFO: Ignoring allowed status 0 2024-08-20T21:56:36.3850354Z == Installing nvidia container toolkit for amzn2023 == 2024-08-20T21:56:36.3855414Z + sudo yum install -y yum-utils 2024-08-20T21:56:36.9296491Z Last metadata expiration check: 0:26:28 ago on Tue Aug 20 21:30:08 2024. 2024-08-20T21:56:36.9598247Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-08-20T21:56:37.0017107Z Dependencies resolved. 2024-08-20T21:56:37.0204286Z Nothing to do. 2024-08-20T21:56:37.0204993Z Complete! 2024-08-20T21:56:37.1556365Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-08-20T21:56:37.1557658Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T21:56:37.1559334Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T21:56:37.5151121Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T21:56:37.6040533Z + sudo yum install -y nvidia-docker2 2024-08-20T21:56:38.2224652Z nvidia-container-toolkit 8.5 kB/s | 833 B 00:00 2024-08-20T21:56:38.2528287Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-08-20T21:56:38.2948175Z Dependencies resolved. 2024-08-20T21:56:38.3136569Z Nothing to do. 2024-08-20T21:56:38.3139098Z Complete! 2024-08-20T21:56:38.4658484Z + sudo systemctl restart docker 2024-08-20T21:57:18.5453495Z nvidia-persistenced failed to initialize. Check syslog for more details. 2024-08-20T21:57:18.5731069Z Tue Aug 20 21:57:18 2024 2024-08-20T21:57:18.5732500Z +-----------------------------------------------------------------------------------------+ 2024-08-20T21:57:18.5734117Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T21:57:18.5735471Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T21:57:18.5737018Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T21:57:18.5739003Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T21:57:18.5740312Z | | | MIG M. | 2024-08-20T21:57:18.5740974Z |=========================================+========================+======================| 2024-08-20T21:57:18.5820893Z | 0 Tesla M60 On | 00000000:00:1D.0 Off | 0 | 2024-08-20T21:57:18.5821841Z | N/A 30C P8 14W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-20T21:57:18.5822649Z | | | N/A | 2024-08-20T21:57:18.5823529Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T21:57:18.5885441Z | 1 Tesla M60 On | 00000000:00:1E.0 Off | 0 | 2024-08-20T21:57:18.5887006Z | N/A 32C P8 14W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-20T21:57:18.5887825Z | | | N/A | 2024-08-20T21:57:18.5888684Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T21:57:18.5889435Z 2024-08-20T21:57:18.5890276Z +-----------------------------------------------------------------------------------------+ 2024-08-20T21:57:18.5891072Z | Processes: | 2024-08-20T21:57:18.5891961Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T21:57:18.5892844Z | ID ID Usage | 2024-08-20T21:57:18.5893546Z |=========================================================================================| 2024-08-20T21:57:18.5894371Z | No running processes found | 2024-08-20T21:57:18.5895354Z +-----------------------------------------------------------------------------------------+ 2024-08-20T21:57:19.4431298Z Command completed after 1 attempt(s). 2024-08-20T21:57:19.4534052Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T21:57:19.4535057Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T21:57:19.4535923Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-20T21:57:19.4536738Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:57:19.4547954Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:57:19.4548576Z env: 2024-08-20T21:57:19.4548913Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:57:19.4549470Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:57:19.4550070Z ##[endgroup] 2024-08-20T21:57:19.7884483Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T21:57:19.8092442Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.9/site-packages (5.9.1) 2024-08-20T21:57:19.8100096Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.9/site-packages (11.525.84) 2024-08-20T21:57:19.9693699Z Prepare all required actions 2024-08-20T21:57:19.9694670Z Getting action download info 2024-08-20T21:57:20.1328419Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-20T21:57:20.3721966Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-20T21:57:20.5361978Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-20T21:57:20.5362577Z with: 2024-08-20T21:57:20.5363163Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-08-20T21:57:20.5363703Z s3-bucket: gha-artifacts 2024-08-20T21:57:20.5364118Z env: 2024-08-20T21:57:20.5364430Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:57:20.5364989Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:57:20.5365581Z ##[endgroup] 2024-08-20T21:57:20.6800082Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T21:57:20.6800641Z with: 2024-08-20T21:57:20.6801077Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-08-20T21:57:20.6801606Z s3-bucket: gha-artifacts 2024-08-20T21:57:20.6802035Z region: us-east-1 2024-08-20T21:57:20.6802398Z env: 2024-08-20T21:57:20.6802708Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:57:20.6803260Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:57:20.6803854Z ##[endgroup] 2024-08-20T21:57:21.2959357Z (node:116460) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T21:57:21.2960261Z 2024-08-20T21:57:21.2960609Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T21:57:21.2961522Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T21:57:21.2962651Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T21:57:21.3743656Z Found 1 objects with prefix pytorch/pytorch/10479309237/linux-focal-cuda11.8-py3.10-gcc9/ 2024-08-20T21:57:21.3745055Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T21:57:46.8106898Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T21:57:46.8117137Z Artifact download has finished successfully 2024-08-20T21:57:46.8356823Z ##[group]Run unzip -o artifacts.zip 2024-08-20T21:57:46.8357368Z unzip -o artifacts.zip 2024-08-20T21:57:46.8365281Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:57:46.8365903Z env: 2024-08-20T21:57:46.8366236Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:57:46.8366797Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:57:46.8367396Z ##[endgroup] 2024-08-20T21:57:46.8416028Z Archive: artifacts.zip 2024-08-20T21:57:46.8417805Z creating: dist/ 2024-08-20T21:57:49.2832980Z inflating: dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T21:57:49.2833819Z creating: build/custom_test_artifacts/ 2024-08-20T21:57:49.2834570Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-20T21:57:49.2835515Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-20T21:57:49.2836569Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-20T21:57:49.2843798Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T21:57:49.2845025Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-20T21:57:49.2846253Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T21:57:49.2847540Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T21:57:49.2848806Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T21:57:49.2850240Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T21:57:49.2851986Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T21:57:49.2853331Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T21:57:49.2854622Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T21:57:49.2856139Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T21:57:49.2857639Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T21:57:49.2859336Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T21:57:49.2860821Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T21:57:49.2862336Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T21:57:49.2863849Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T21:57:49.2865201Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T21:57:49.2866515Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T21:57:49.2920324Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T21:57:49.2922174Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T21:57:49.2924072Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T21:57:49.2926011Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T21:57:49.2927915Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T21:57:49.2929713Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T21:57:49.2931534Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T21:57:49.2933371Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T21:57:49.2935192Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T21:57:49.2975988Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T21:57:49.3023767Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T21:57:49.3025757Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T21:57:49.3027736Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T21:57:49.3029339Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T21:57:49.3030925Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T21:57:49.3032546Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T21:57:49.3034101Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T21:57:49.3035666Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T21:57:49.3106352Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T21:57:49.3179942Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T21:57:49.3181498Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T21:57:49.3182767Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-20T21:57:49.3183891Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-20T21:57:49.3185254Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-20T21:57:49.3186433Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-20T21:57:49.3187768Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-20T21:57:49.3189267Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-20T21:57:49.3190714Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-20T21:57:49.3192050Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-20T21:57:49.3193444Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-20T21:57:49.3194845Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-20T21:57:49.3196237Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-20T21:57:49.3197635Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-20T21:57:49.3199011Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-20T21:57:49.3215849Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-20T21:57:49.3387398Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-20T21:57:49.3388700Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-20T21:57:49.3390115Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-20T21:57:49.3391692Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-20T21:57:49.3393190Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-20T21:57:49.3394605Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-20T21:57:49.3396084Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-20T21:57:49.3397563Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-20T21:57:49.3399027Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-20T21:57:49.3400497Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-20T21:57:49.3401951Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-20T21:57:49.3420658Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-20T21:57:49.3520428Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-20T21:57:49.3521983Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T21:57:49.3523381Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-20T21:57:49.3524794Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-20T21:57:49.3525999Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-20T21:57:49.3527169Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-20T21:57:49.3528330Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-08-20T21:57:49.3529876Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-20T21:57:49.3530958Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-20T21:57:49.3532279Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-20T21:57:49.3676224Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-20T21:57:49.3751726Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-20T21:57:49.3752686Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-20T21:57:49.3753575Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-20T21:57:49.3754632Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-20T21:57:49.3761838Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T21:57:49.3763029Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-20T21:57:49.3764227Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T21:57:49.3765504Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T21:57:49.3766755Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T21:57:49.3768154Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T21:57:49.3769933Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T21:57:49.3771220Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T21:57:49.3772481Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T21:57:49.3773970Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T21:57:49.3775465Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T21:57:49.3777284Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T21:57:49.3778884Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T21:57:49.3780411Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T21:57:49.3781900Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T21:57:49.3783253Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T21:57:49.3784546Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T21:57:49.3838250Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T21:57:49.3840111Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T21:57:49.3841995Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T21:57:49.3843914Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T21:57:49.3845790Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T21:57:49.3847805Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T21:57:49.3849661Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T21:57:49.3851482Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T21:57:49.3853258Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T21:57:49.3894054Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T21:57:49.3941611Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T21:57:49.3943430Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T21:57:49.3945103Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T21:57:49.3946686Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T21:57:49.3948267Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T21:57:49.3949861Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T21:57:49.3951389Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T21:57:49.3952968Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T21:57:49.4023709Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T21:57:49.4097671Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T21:57:49.4099201Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T21:57:49.4100484Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-20T21:57:49.4101587Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-20T21:57:49.4102751Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-20T21:57:49.4103959Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-20T21:57:49.4105397Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-20T21:57:49.4106940Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-20T21:57:49.4108409Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-20T21:57:49.4109808Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-20T21:57:49.4111238Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-20T21:57:49.4112668Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-20T21:57:49.4114090Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-20T21:57:49.4115527Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-20T21:57:49.4116944Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-20T21:57:49.4133564Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-20T21:57:49.4211095Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-20T21:57:49.4212665Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T21:57:49.4214052Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-20T21:57:49.4215302Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-20T21:57:49.4216443Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-20T21:57:49.4217741Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-20T21:57:49.4218892Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-08-20T21:57:49.4219958Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-20T21:57:49.4221144Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-20T21:57:49.4222187Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-20T21:57:49.4282358Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-20T21:57:49.4283323Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-20T21:57:49.4284303Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-20T21:57:49.4285441Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-20T21:57:49.4292441Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T21:57:49.4293723Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-20T21:57:49.4295022Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T21:57:49.4296386Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T21:57:49.4297714Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T21:57:49.4299217Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T21:57:49.4300740Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T21:57:49.4302105Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T21:57:49.4303447Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T21:57:49.4305031Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T21:57:49.4306614Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T21:57:49.4308175Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T21:57:49.4309719Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T21:57:49.4311304Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T21:57:49.4312905Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T21:57:49.4314323Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T21:57:49.4315693Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T21:57:49.4368773Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T21:57:49.4371176Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T21:57:49.4373324Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T21:57:49.4375365Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T21:57:49.4377351Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T21:57:49.4379286Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T21:57:49.4381328Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T21:57:49.4383226Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T21:57:49.4385125Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T21:57:49.4424401Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T21:57:49.4472470Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T21:57:49.4474350Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T21:57:49.4476122Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T21:57:49.4477816Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T21:57:49.4479532Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T21:57:49.4481203Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T21:57:49.4482834Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T21:57:49.4484503Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T21:57:49.4554823Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T21:57:49.4628846Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T21:57:49.4630477Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T21:57:49.4631837Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-20T21:57:49.4633036Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-20T21:57:49.4634285Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-20T21:57:49.4635597Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-20T21:57:49.4637074Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-20T21:57:49.4638697Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-20T21:57:49.4640291Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-20T21:57:49.4641790Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-20T21:57:49.4643328Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-20T21:57:49.4645035Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-20T21:57:49.4646619Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-20T21:57:49.4648164Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-20T21:57:49.4649693Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-20T21:57:49.4651312Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-20T21:57:49.4796422Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-20T21:57:49.4798009Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-20T21:57:49.4799565Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-20T21:57:49.4801320Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-20T21:57:49.4803003Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-20T21:57:49.4804581Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-20T21:57:49.4806182Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-20T21:57:49.4807811Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-20T21:57:49.4809439Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-20T21:57:49.4811047Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-20T21:57:49.4812657Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-20T21:57:49.4829729Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-20T21:57:49.4896518Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-20T21:57:49.4898208Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T21:57:49.4899711Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-20T21:57:49.4901067Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-20T21:57:49.4902319Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-20T21:57:49.4903540Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-20T21:57:49.4904792Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-08-20T21:57:49.4905968Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-20T21:57:49.4907050Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-20T21:57:49.4908145Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-20T21:57:49.5032549Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-20T21:57:49.5084519Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-20T21:57:49.5085325Z creating: build/lib/ 2024-08-20T21:57:49.5096408Z inflating: build/lib/libpthreadpool.a 2024-08-20T21:57:49.5097347Z inflating: build/lib/libclog.a 2024-08-20T21:57:49.5100387Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-20T21:57:49.5110655Z inflating: build/lib/libcpuinfo.a 2024-08-20T21:57:49.5120618Z inflating: build/lib/libcpuinfo_internals.a 2024-08-20T21:57:49.5227024Z inflating: build/lib/libprotobuf-lite.a 2024-08-20T21:57:49.5773892Z inflating: build/lib/libprotobuf.a 2024-08-20T21:57:49.5852222Z inflating: build/lib/libgtest.a 2024-08-20T21:57:49.5942663Z inflating: build/lib/libbenchmark.a 2024-08-20T21:57:49.5952271Z inflating: build/lib/libittnotify.a 2024-08-20T21:57:49.6029637Z inflating: build/lib/libasmjit.a 2024-08-20T21:57:49.6063272Z inflating: build/lib/libtensorpipe_uv.a 2024-08-20T21:57:49.6216508Z inflating: build/lib/libgloo.a 2024-08-20T21:57:49.6241525Z inflating: build/lib/libfmt.a 2024-08-20T21:57:49.6243701Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-20T21:57:49.6358589Z inflating: build/lib/libc10.so 2024-08-20T21:57:49.6360375Z inflating: build/lib/libtorch_global_deps.so 2024-08-20T21:57:49.6965856Z inflating: build/lib/libprotoc.a 2024-08-20T21:57:49.6989808Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-20T21:57:49.7011417Z inflating: build/lib/libgmock.a 2024-08-20T21:57:49.7012333Z inflating: build/lib/libgtest_main.a 2024-08-20T21:57:49.7013400Z inflating: build/lib/libbenchmark_main.a 2024-08-20T21:57:50.9130020Z inflating: build/lib/libdnnl.a 2024-08-20T21:57:50.9151876Z inflating: build/lib/libnnpack.a 2024-08-20T21:57:50.9838169Z inflating: build/lib/libtensorpipe.a 2024-08-20T21:57:50.9908373Z inflating: build/lib/libc10_cuda.so 2024-08-20T21:57:51.1406500Z inflating: build/lib/libfbgemm.a 2024-08-20T21:57:51.1407247Z inflating: build/lib/libgmock_main.a 2024-08-20T21:57:51.2018201Z inflating: build/lib/libkineto.a 2024-08-20T21:57:51.2324539Z inflating: build/lib/libtensorpipe_cuda.a 2024-08-20T21:57:51.2547408Z inflating: build/lib/libXNNPACK.a 2024-08-20T21:57:51.3007314Z inflating: build/lib/libgloo_cuda.a 2024-08-20T21:57:51.3058193Z inflating: build/lib/libonnx_proto.a 2024-08-20T21:57:51.3889535Z inflating: build/lib/libonnx.a 2024-08-20T21:57:54.3299496Z inflating: build/lib/libtorch_cpu.so 2024-08-20T21:57:54.3304902Z inflating: build/lib/libshm.so 2024-08-20T21:57:54.3310784Z inflating: build/lib/libunbox_lib.a 2024-08-20T21:57:56.7543954Z inflating: build/lib/libtorch_cuda.so 2024-08-20T21:57:56.7545393Z inflating: build/lib/libtorch.so 2024-08-20T21:57:56.7548762Z inflating: build/lib/libc10d_cuda_test.so 2024-08-20T21:57:57.8775200Z inflating: build/lib/libtorch_cuda_linalg.so 2024-08-20T21:57:57.8800339Z inflating: build/lib/libjitbackend_test.so 2024-08-20T21:57:57.8887013Z inflating: build/lib/libtorchbind_test.so 2024-08-20T21:57:57.8918475Z inflating: build/lib/libaoti_custom_ops.so 2024-08-20T21:57:57.8949278Z inflating: build/lib/libbackend_with_compiler.so 2024-08-20T21:57:58.1359393Z inflating: build/lib/libtorch_python.so 2024-08-20T21:57:58.1401780Z inflating: build/lib/libnnapi_backend.so 2024-08-20T21:57:58.1402366Z creating: build/bin/ 2024-08-20T21:57:58.1463230Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-20T21:57:58.1526051Z inflating: build/bin/c10_DeviceGuard_test 2024-08-20T21:57:58.1588681Z inflating: build/bin/c10_Device_test 2024-08-20T21:57:58.1660631Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-20T21:57:58.1720473Z inflating: build/bin/c10_StreamGuard_test 2024-08-20T21:57:58.1785618Z inflating: build/bin/c10_Scalar_test 2024-08-20T21:57:58.1848166Z inflating: build/bin/c10_SymInt_test 2024-08-20T21:57:58.1914812Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-20T21:57:58.1982792Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-20T21:57:58.2051352Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-20T21:57:58.2137878Z inflating: build/bin/c10_cow_test 2024-08-20T21:57:58.2202760Z inflating: build/bin/c10_Bitset_test 2024-08-20T21:57:58.2262251Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-20T21:57:58.2323596Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-20T21:57:58.2385450Z inflating: build/bin/c10_Half_test 2024-08-20T21:57:58.2454024Z inflating: build/bin/c10_LeftRight_test 2024-08-20T21:57:58.2521410Z inflating: build/bin/c10_Metaprogramming_test 2024-08-20T21:57:58.2582553Z inflating: build/bin/c10_Synchronized_test 2024-08-20T21:57:58.2650017Z inflating: build/bin/c10_ThreadLocal_test 2024-08-20T21:57:58.2713397Z inflating: build/bin/c10_TypeIndex_test 2024-08-20T21:57:58.2775875Z inflating: build/bin/c10_TypeList_test 2024-08-20T21:57:58.2835378Z inflating: build/bin/c10_TypeTraits_test 2024-08-20T21:57:58.2898778Z inflating: build/bin/c10_accumulate_test 2024-08-20T21:57:58.2966016Z inflating: build/bin/c10_bfloat16_test 2024-08-20T21:57:58.3027596Z inflating: build/bin/c10_bit_cast_test 2024-08-20T21:57:58.3097316Z inflating: build/bin/c10_complex_math_test 2024-08-20T21:57:58.3161287Z inflating: build/bin/c10_exception_test 2024-08-20T21:57:58.3229604Z inflating: build/bin/c10_complex_test 2024-08-20T21:57:58.3291166Z inflating: build/bin/c10_flags_test 2024-08-20T21:57:58.3352179Z inflating: build/bin/c10_generic_math_test 2024-08-20T21:57:58.3551764Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-20T21:57:58.3613921Z inflating: build/bin/c10_irange_test 2024-08-20T21:57:58.3679678Z inflating: build/bin/c10_lazy_test 2024-08-20T21:57:58.3748920Z inflating: build/bin/c10_logging_test 2024-08-20T21:57:58.3840417Z inflating: build/bin/c10_optional_test 2024-08-20T21:57:58.3916525Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-20T21:57:58.3982470Z inflating: build/bin/c10_registry_test 2024-08-20T21:57:58.4164534Z inflating: build/bin/c10_small_vector_test 2024-08-20T21:57:58.4227570Z inflating: build/bin/c10_ssize_test 2024-08-20T21:57:58.4291258Z inflating: build/bin/c10_string_util_test 2024-08-20T21:57:58.4362599Z inflating: build/bin/c10_string_view_test 2024-08-20T21:57:58.4425373Z inflating: build/bin/c10_tempfile_test 2024-08-20T21:57:58.4494125Z inflating: build/bin/c10_typeid_test 2024-08-20T21:57:58.4553102Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-20T21:57:58.5096104Z inflating: build/bin/protoc-3.13.0.0 2024-08-20T21:57:58.5638098Z inflating: build/bin/protoc 2024-08-20T21:57:58.5703056Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-08-20T21:57:58.5767416Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-08-20T21:57:58.5831891Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-08-20T21:57:58.5895720Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-08-20T21:57:58.5955378Z inflating: build/bin/c10_cuda_CUDATest 2024-08-20T21:57:58.6020142Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-20T21:57:58.6085073Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-20T21:57:58.6149189Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-08-20T21:57:58.6582219Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-20T21:57:58.6997254Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-20T21:57:58.7394593Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-20T21:57:58.7458904Z inflating: build/bin/FileStoreTest 2024-08-20T21:57:58.7526955Z inflating: build/bin/TCPStoreTest 2024-08-20T21:57:58.7592114Z inflating: build/bin/HashStoreTest 2024-08-20T21:57:58.7656281Z inflating: build/bin/BackoffTest 2024-08-20T21:57:58.7673261Z inflating: build/bin/ProcessGroupMPITest 2024-08-20T21:57:58.7678524Z inflating: build/bin/torch_shm_manager 2024-08-20T21:57:58.7744019Z inflating: build/bin/test_edge_op_registration 2024-08-20T21:57:58.7747535Z inflating: build/bin/example_allreduce 2024-08-20T21:57:58.7811416Z inflating: build/bin/Dimname_test 2024-08-20T21:57:58.7901084Z inflating: build/bin/Dict_test 2024-08-20T21:57:58.7979303Z inflating: build/bin/MaybeOwned_test 2024-08-20T21:57:58.8049062Z inflating: build/bin/NamedTensor_test 2024-08-20T21:57:58.8121321Z inflating: build/bin/apply_utils_test 2024-08-20T21:57:58.8193431Z inflating: build/bin/atest 2024-08-20T21:57:58.8270271Z inflating: build/bin/basic 2024-08-20T21:57:58.8337276Z inflating: build/bin/broadcast_test 2024-08-20T21:57:58.8408300Z inflating: build/bin/cpu_generator_test 2024-08-20T21:57:58.8470310Z inflating: build/bin/cpu_allocator_test 2024-08-20T21:57:58.8535034Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-20T21:57:58.8596777Z inflating: build/bin/dlconvertor_test 2024-08-20T21:57:58.8657345Z inflating: build/bin/dispatch_key_set_test 2024-08-20T21:57:58.8771066Z inflating: build/bin/cpu_rng_test 2024-08-20T21:57:58.8841623Z inflating: build/bin/extension_backend_test 2024-08-20T21:57:58.8909033Z inflating: build/bin/half_test 2024-08-20T21:57:58.9025450Z inflating: build/bin/ivalue_test 2024-08-20T21:57:58.9086487Z inflating: build/bin/lazy_tensor_test 2024-08-20T21:57:58.9152632Z inflating: build/bin/math_kernel_test 2024-08-20T21:57:58.9218853Z inflating: build/bin/memory_format_test 2024-08-20T21:57:58.9283928Z inflating: build/bin/memory_overlapping_test 2024-08-20T21:57:58.9348127Z inflating: build/bin/mobile_memory_cleanup 2024-08-20T21:57:58.9416587Z inflating: build/bin/native_test 2024-08-20T21:57:58.9478446Z inflating: build/bin/operator_name_test 2024-08-20T21:57:58.9542114Z inflating: build/bin/packedtensoraccessor_test 2024-08-20T21:57:58.9605315Z inflating: build/bin/operators_test 2024-08-20T21:57:58.9687279Z inflating: build/bin/pow_test 2024-08-20T21:57:58.9749628Z inflating: build/bin/reportMemoryUsage_test 2024-08-20T21:57:58.9819560Z inflating: build/bin/quantized_test 2024-08-20T21:57:58.9881357Z inflating: build/bin/reduce_ops_test 2024-08-20T21:57:58.9950097Z inflating: build/bin/scalar_tensor_test 2024-08-20T21:57:59.0021860Z inflating: build/bin/scalar_test 2024-08-20T21:57:59.0085352Z inflating: build/bin/StorageUtils_test 2024-08-20T21:57:59.0149083Z inflating: build/bin/stride_properties_test 2024-08-20T21:57:59.0244848Z inflating: build/bin/tensor_iterator_test 2024-08-20T21:57:59.0313070Z inflating: build/bin/type_ptr_test 2024-08-20T21:57:59.0379102Z inflating: build/bin/test_parallel 2024-08-20T21:57:59.0382847Z inflating: build/bin/thread_init_test 2024-08-20T21:57:59.0446301Z inflating: build/bin/undefined_tensor_test 2024-08-20T21:57:59.0519839Z inflating: build/bin/type_test 2024-08-20T21:57:59.0521891Z inflating: build/bin/verify_api_visibility 2024-08-20T21:57:59.0584902Z inflating: build/bin/weakref_test 2024-08-20T21:57:59.0669017Z inflating: build/bin/legacy_vmap_test 2024-08-20T21:57:59.0731826Z inflating: build/bin/wrapdim_test 2024-08-20T21:57:59.0805318Z inflating: build/bin/IListRef_test 2024-08-20T21:57:59.0868079Z inflating: build/bin/xla_tensor_test 2024-08-20T21:57:59.0996980Z inflating: build/bin/List_test 2024-08-20T21:57:59.1144649Z inflating: build/bin/kernel_function_legacy_test 2024-08-20T21:57:59.1262003Z inflating: build/bin/kernel_function_test 2024-08-20T21:57:59.1343147Z inflating: build/bin/KernelFunction_test 2024-08-20T21:57:59.1498634Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-20T21:57:59.1624265Z inflating: build/bin/kernel_lambda_test 2024-08-20T21:57:59.1699271Z inflating: build/bin/kernel_stackbased_test 2024-08-20T21:57:59.1761412Z inflating: build/bin/CppSignature_test 2024-08-20T21:57:59.1879063Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-20T21:57:59.1938388Z inflating: build/bin/op_allowlist_test 2024-08-20T21:57:59.2006759Z inflating: build/bin/backend_fallback_test 2024-08-20T21:57:59.2084704Z inflating: build/bin/inline_container_test 2024-08-20T21:57:59.2447926Z inflating: build/bin/op_registration_test 2024-08-20T21:57:59.2512487Z inflating: build/bin/cuda_apply_test 2024-08-20T21:57:59.2575888Z inflating: build/bin/cuda_allocator_test 2024-08-20T21:57:59.2647863Z inflating: build/bin/cuda_atomic_ops_test 2024-08-20T21:57:59.2714858Z inflating: build/bin/cuda_caching_host_allocator_test 2024-08-20T21:57:59.2800861Z inflating: build/bin/cuda_complex_math_test 2024-08-20T21:57:59.2872967Z inflating: build/bin/cuda_complex_test 2024-08-20T21:57:59.2933325Z inflating: build/bin/cuda_device_test 2024-08-20T21:57:59.3004183Z inflating: build/bin/cuda_cub_test 2024-08-20T21:57:59.3065847Z inflating: build/bin/cuda_dlconvertor_test 2024-08-20T21:57:59.3128328Z inflating: build/bin/cuda_integer_divider_test 2024-08-20T21:57:59.3208431Z inflating: build/bin/cuda_distributions_test 2024-08-20T21:57:59.3277596Z inflating: build/bin/cuda_generator_test 2024-08-20T21:57:59.3338303Z inflating: build/bin/cuda_half_test 2024-08-20T21:57:59.3398594Z inflating: build/bin/cuda_optional_test 2024-08-20T21:57:59.3459157Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-08-20T21:57:59.3533446Z inflating: build/bin/cuda_stream_test 2024-08-20T21:57:59.3597881Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-08-20T21:57:59.3660486Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-08-20T21:57:59.3721095Z inflating: build/bin/cuda_cudnn_test 2024-08-20T21:57:59.3784877Z inflating: build/bin/cuda_vectorized_test 2024-08-20T21:57:59.3864468Z inflating: build/bin/ProcessGroupGlooTest 2024-08-20T21:57:59.3882360Z inflating: build/bin/tutorial_tensorexpr 2024-08-20T21:57:59.3952135Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-08-20T21:57:59.4030162Z inflating: build/bin/ProcessGroupNCCLTest 2024-08-20T21:57:59.4094415Z inflating: build/bin/ProcessGroupUCCTest 2024-08-20T21:57:59.4170218Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-08-20T21:57:59.4237189Z inflating: build/bin/test_dist_autograd 2024-08-20T21:57:59.4321374Z inflating: build/bin/test_cpp_rpc 2024-08-20T21:57:59.4324613Z inflating: build/bin/parallel_benchmark 2024-08-20T21:57:59.4336188Z inflating: build/bin/aot_model_compiler_test 2024-08-20T21:57:59.4418149Z inflating: build/bin/test_mobile_nnc 2024-08-20T21:57:59.4835261Z inflating: build/bin/test_lazy 2024-08-20T21:57:59.5854215Z inflating: build/bin/test_tensorexpr 2024-08-20T21:57:59.7268630Z inflating: build/bin/test_api 2024-08-20T21:57:59.7972592Z inflating: build/bin/test_jit 2024-08-20T21:57:59.7973139Z creating: .additional_ci_files/ 2024-08-20T21:57:59.8045103Z inflating: .additional_ci_files/test-times.json 2024-08-20T21:57:59.8332681Z inflating: .additional_ci_files/test-class-times.json 2024-08-20T21:57:59.8372200Z ##[group]Run rm artifacts.zip 2024-08-20T21:57:59.8372663Z rm artifacts.zip 2024-08-20T21:57:59.8379777Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:57:59.8380406Z env: 2024-08-20T21:57:59.8380724Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:57:59.8381280Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:57:59.8381877Z ##[endgroup] 2024-08-20T21:57:59.9027814Z ##[group]Run df -H 2024-08-20T21:57:59.9028181Z df -H 2024-08-20T21:57:59.9034717Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:57:59.9035341Z env: 2024-08-20T21:57:59.9035672Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:57:59.9036229Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:57:59.9036827Z ##[endgroup] 2024-08-20T21:57:59.9083046Z Filesystem Size Used Avail Use% Mounted on 2024-08-20T21:57:59.9083677Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-08-20T21:57:59.9084254Z tmpfs 129G 0 129G 0% /dev/shm 2024-08-20T21:57:59.9084820Z tmpfs 52G 635k 52G 1% /run 2024-08-20T21:57:59.9085861Z /dev/xvda1 161G 47G 115G 30% / 2024-08-20T21:57:59.9086421Z tmpfs 129G 21k 129G 1% /tmp 2024-08-20T21:57:59.9086996Z /dev/xvda128 11M 1.4M 9.2M 13% /boot/efi 2024-08-20T21:57:59.9087601Z tmpfs 26G 0 26G 0% /run/user/0 2024-08-20T21:57:59.9136249Z Prepare all required actions 2024-08-20T21:57:59.9136819Z Getting action download info 2024-08-20T21:58:00.0248502Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-20T21:58:00.0249067Z with: 2024-08-20T21:58:00.0249580Z env: 2024-08-20T21:58:00.0249912Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:00.0250454Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:00.0251055Z ##[endgroup] 2024-08-20T21:58:00.0295047Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T21:58:00.0295606Z with: 2024-08-20T21:58:00.0295927Z name: td_results 2024-08-20T21:58:00.0296299Z s3-bucket: gha-artifacts 2024-08-20T21:58:00.0296725Z region: us-east-1 2024-08-20T21:58:00.0297088Z env: 2024-08-20T21:58:00.0297396Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:00.0297948Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:00.0298543Z ##[endgroup] 2024-08-20T21:58:00.6573531Z (node:116487) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T21:58:00.6574452Z 2024-08-20T21:58:00.6574772Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T21:58:00.6575682Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T21:58:00.6576801Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T21:58:00.7481636Z Found 1 objects with prefix pytorch/pytorch/10479309237/td_results/ 2024-08-20T21:58:00.7483052Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-08-20T21:58:00.8140211Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-08-20T21:58:00.8150280Z Artifact download has finished successfully 2024-08-20T21:58:00.8356696Z ##[group]Run mkdir -p .additional_ci_files 2024-08-20T21:58:00.8357280Z mkdir -p .additional_ci_files 2024-08-20T21:58:00.8357996Z mv td_results.json .additional_ci_files/td_results.json 2024-08-20T21:58:00.8365449Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:58:00.8366058Z env: 2024-08-20T21:58:00.8366397Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:00.8366961Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:00.8367556Z ##[endgroup] 2024-08-20T21:58:00.8467627Z ##[group]Run .github/scripts/parse_ref.py 2024-08-20T21:58:00.8468239Z .github/scripts/parse_ref.py 2024-08-20T21:58:00.8475628Z shell: /usr/bin/bash -e {0} 2024-08-20T21:58:00.8476074Z env: 2024-08-20T21:58:00.8476387Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:00.8476947Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:00.8477541Z ##[endgroup] 2024-08-20T21:58:00.8834151Z Prepare all required actions 2024-08-20T21:58:00.8886024Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-20T21:58:00.8886598Z with: 2024-08-20T21:58:00.8887319Z github-token: *** 2024-08-20T21:58:00.8887682Z env: 2024-08-20T21:58:00.8888006Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:00.8888564Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:00.8889141Z ##[endgroup] 2024-08-20T21:58:00.8912208Z ##[group]Run set -eux 2024-08-20T21:58:00.8912617Z set -eux 2024-08-20T21:58:00.8913356Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-20T21:58:00.8920118Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:58:00.8920718Z env: 2024-08-20T21:58:00.8921052Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:00.8921609Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:00.8922406Z GITHUB_TOKEN: *** 2024-08-20T21:58:00.8922794Z ##[endgroup] 2024-08-20T21:58:00.8950491Z + python3 .github/scripts/get_workflow_job_id.py 10479309237 i-0cc1c134a48939331 2024-08-20T21:58:03.0001445Z setting job-id=29025850735 2024-08-20T21:58:03.0002678Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, amz2023.linux.8xlarge.nvidia.gpu) 2024-08-20T21:58:03.0278967Z Prepare all required actions 2024-08-20T21:58:03.0279548Z Getting action download info 2024-08-20T21:58:03.1543425Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-20T21:58:03.1544088Z with: 2024-08-20T21:58:03.1544918Z github-token: *** 2024-08-20T21:58:03.1546874Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}]} 2024-08-20T21:58:03.1549377Z job-name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, amz2023.linux.8xlarge.nvidia.gpu) 2024-08-20T21:58:03.1550322Z env: 2024-08-20T21:58:03.1550655Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:03.1551196Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:03.1551792Z ##[endgroup] 2024-08-20T21:58:03.1631431Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T21:58:03.1632116Z with: 2024-08-20T21:58:03.1632442Z shell: bash 2024-08-20T21:58:03.1632808Z timeout_minutes: 10 2024-08-20T21:58:03.1633188Z max_attempts: 5 2024-08-20T21:58:03.1633578Z retry_wait_seconds: 30 2024-08-20T21:58:03.1634989Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T21:58:03.1636468Z polling_interval_seconds: 1 2024-08-20T21:58:03.1636908Z warning_on_retry: true 2024-08-20T21:58:03.1637334Z continue_on_error: false 2024-08-20T21:58:03.1637741Z env: 2024-08-20T21:58:03.1638050Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:03.1638607Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:03.1639426Z GITHUB_TOKEN: *** 2024-08-20T21:58:03.1639792Z ##[endgroup] 2024-08-20T21:58:03.2552885Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T21:58:03.6054533Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T21:58:03.6252541Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.9/site-packages (2.27.1) 2024-08-20T21:58:03.6258699Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.9/site-packages (6.0.1) 2024-08-20T21:58:03.6413019Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2024.7.4) 2024-08-20T21:58:03.6427096Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-08-20T21:58:03.6433033Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-08-20T21:58:03.6447741Z Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2.0.12) 2024-08-20T21:58:04.2354336Z Command completed after 1 attempt(s). 2024-08-20T21:58:04.2414430Z ##[group]Run set -x 2024-08-20T21:58:04.2414822Z set -x 2024-08-20T21:58:04.2415192Z  2024-08-20T21:58:04.2415889Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T21:58:04.2416720Z # in runner workspace 2024-08-20T21:58:04.2417387Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-20T21:58:04.2424985Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:58:04.2425594Z env: 2024-08-20T21:58:04.2425941Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:04.2426510Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:04.2427096Z ##[endgroup] 2024-08-20T21:58:04.2457641Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-20T21:58:04.2778671Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T21:58:04.2779362Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T21:58:04.2779941Z echo "Job name: ${JOB_NAME}" 2024-08-20T21:58:04.2780470Z  2024-08-20T21:58:04.2781147Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T21:58:04.2782184Z # in runner workspace 2024-08-20T21:58:04.2782882Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-20T21:58:04.2783686Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-20T21:58:04.2784248Z  --job-name "${JOB_NAME}" \ 2024-08-20T21:58:04.2786364Z  --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}]}" \ 2024-08-20T21:58:04.2788552Z  --selected-test-configs "" \ 2024-08-20T21:58:04.2789122Z  --pr-number "${PR_NUMBER}" \ 2024-08-20T21:58:04.2789643Z  --tag "${TAG}" \ 2024-08-20T21:58:04.2790110Z  --event-name "${EVENT_NAME}" \ 2024-08-20T21:58:04.2790667Z  --schedule "${SCHEDULE}" \ 2024-08-20T21:58:04.2791195Z  --branch "${HEAD_BRANCH}" 2024-08-20T21:58:04.2797866Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:58:04.2798489Z env: 2024-08-20T21:58:04.2798825Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:04.2799364Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:04.2800207Z GITHUB_TOKEN: *** 2024-08-20T21:58:04.2801068Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, amz2023.linux.8xlarge.nvidia.gpu) 2024-08-20T21:58:04.2802025Z PR_NUMBER: 133712 2024-08-20T21:58:04.2802380Z TAG: 2024-08-20T21:58:04.2802721Z EVENT_NAME: pull_request 2024-08-20T21:58:04.2803154Z SCHEDULE: 2024-08-20T21:58:04.2803486Z HEAD_BRANCH: 2024-08-20T21:58:04.2803844Z ##[endgroup] 2024-08-20T21:58:04.2831465Z Workflow: pull 2024-08-20T21:58:04.2832543Z Job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, amz2023.linux.8xlarge.nvidia.gpu) 2024-08-20T21:58:04.5418309Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-20T21:58:04.8127613Z ##[group]Run echo "Filtered matrix:" 2024-08-20T21:58:04.8128179Z echo "Filtered matrix:" 2024-08-20T21:58:04.8130238Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}]}" 2024-08-20T21:58:04.8132286Z  2024-08-20T21:58:04.8132602Z echo 2024-08-20T21:58:04.8133080Z echo "Is the current job unstable? False" 2024-08-20T21:58:04.8133653Z  2024-08-20T21:58:04.8133963Z echo 2024-08-20T21:58:04.8134400Z echo "Is keep-going label set? False" 2024-08-20T21:58:04.8134945Z  2024-08-20T21:58:04.8135374Z echo 2024-08-20T21:58:04.8136062Z echo "Renabled issues? " 2024-08-20T21:58:04.8143310Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:58:04.8143918Z env: 2024-08-20T21:58:04.8144256Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:04.8144818Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:04.8145404Z ##[endgroup] 2024-08-20T21:58:04.8174640Z Filtered matrix: 2024-08-20T21:58:04.8176593Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: amz2023.linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 2, num_shards: 3, runner: amz2023.linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 3, num_shards: 3, runner: amz2023.linux.8xlarge.nvidia.gpu}]} 2024-08-20T21:58:04.8178656Z 2024-08-20T21:58:04.8178874Z Is the current job unstable? False 2024-08-20T21:58:04.8179260Z 2024-08-20T21:58:04.8179614Z Is keep-going label set? False 2024-08-20T21:58:04.8179937Z 2024-08-20T21:58:04.8180102Z Renabled issues? 2024-08-20T21:58:04.8235790Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T21:58:04.8236709Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T21:58:04.8243366Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:58:04.8243969Z env: 2024-08-20T21:58:04.8244308Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:04.8244869Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:04.8245454Z JOB_TIMEOUT: 360 2024-08-20T21:58:04.8245829Z ##[endgroup] 2024-08-20T21:58:04.8344523Z ##[group]Run set -x 2024-08-20T21:58:04.8345041Z set -x 2024-08-20T21:58:04.8345397Z  2024-08-20T21:58:04.8345823Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-20T21:58:04.8346479Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-20T21:58:04.8347173Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-20T21:58:04.8347802Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-20T21:58:04.8348299Z else 2024-08-20T21:58:04.8348708Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T21:58:04.8349224Z fi 2024-08-20T21:58:04.8349531Z  2024-08-20T21:58:04.8350101Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-20T21:58:04.8351046Z # TODO: Stop building test binaries as part of the build phase 2024-08-20T21:58:04.8351869Z # Used for GPU_FLAG since that doesn't play nice 2024-08-20T21:58:04.8352595Z # shellcheck disable=SC2086,SC2090 2024-08-20T21:58:04.8353153Z container_name=$(docker run \ 2024-08-20T21:58:04.8353666Z  ${GPU_FLAG:-} \ 2024-08-20T21:58:04.8354119Z  -e BUILD_ENVIRONMENT \ 2024-08-20T21:58:04.8354592Z  -e PR_NUMBER \ 2024-08-20T21:58:04.8355032Z  -e GITHUB_ACTIONS \ 2024-08-20T21:58:04.8355509Z  -e GITHUB_REPOSITORY \ 2024-08-20T21:58:04.8355986Z  -e GITHUB_WORKFLOW \ 2024-08-20T21:58:04.8356463Z  -e GITHUB_JOB \ 2024-08-20T21:58:04.8356904Z  -e GITHUB_RUN_ID \ 2024-08-20T21:58:04.8357373Z  -e GITHUB_RUN_NUMBER \ 2024-08-20T21:58:04.8357851Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-20T21:58:04.8358332Z  -e JOB_ID \ 2024-08-20T21:58:04.8358740Z  -e JOB_NAME \ 2024-08-20T21:58:04.8359143Z  -e BASE_SHA \ 2024-08-20T21:58:04.8359557Z  -e BRANCH \ 2024-08-20T21:58:04.8359959Z  -e SHA1 \ 2024-08-20T21:58:04.8360363Z  -e AWS_DEFAULT_REGION \ 2024-08-20T21:58:04.8360855Z  -e IN_WHEEL_TEST \ 2024-08-20T21:58:04.8361314Z  -e SHARD_NUMBER \ 2024-08-20T21:58:04.8361746Z  -e TEST_CONFIG \ 2024-08-20T21:58:04.8362195Z  -e NUM_TEST_SHARDS \ 2024-08-20T21:58:04.8362674Z  -e REENABLED_ISSUES \ 2024-08-20T21:58:04.8363159Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-20T21:58:04.8363680Z  -e VERBOSE_TEST_LOGS \ 2024-08-20T21:58:04.8364173Z  -e TEST_SHOWLOCALS \ 2024-08-20T21:58:04.8364631Z  -e NO_TEST_TIMEOUT \ 2024-08-20T21:58:04.8365083Z  -e NO_TD \ 2024-08-20T21:58:04.8365493Z  -e TD_DISTRIBUTED \ 2024-08-20T21:58:04.8365938Z  -e PR_LABELS \ 2024-08-20T21:58:04.8366431Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-20T21:58:04.8366986Z  -e SCCACHE_BUCKET \ 2024-08-20T21:58:04.8367451Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-20T21:58:04.8367947Z  -e XLA_CUDA \ 2024-08-20T21:58:04.8368440Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-20T21:58:04.8369443Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-20T21:58:04.8370109Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-20T21:58:04.8370734Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-20T21:58:04.8371290Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-20T21:58:04.8371842Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-08-20T21:58:04.8372583Z  -e DASHBOARD_TAG \ 2024-08-20T21:58:04.8373150Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-20T21:58:04.8373834Z  --security-opt seccomp=unconfined \ 2024-08-20T21:58:04.8374403Z  --cap-add=SYS_PTRACE \ 2024-08-20T21:58:04.8374878Z  --ipc=host \ 2024-08-20T21:58:04.8375293Z  --shm-size="${SHM_SIZE}" \ 2024-08-20T21:58:04.8375774Z  --tty \ 2024-08-20T21:58:04.8376153Z  --detach \ 2024-08-20T21:58:04.8376722Z  --name="${container_name}" \ 2024-08-20T21:58:04.8377263Z  --user jenkins \ 2024-08-20T21:58:04.8377861Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-20T21:58:04.8378534Z  -w /var/lib/jenkins/workspace \ 2024-08-20T21:58:04.8379068Z  "${DOCKER_IMAGE}" 2024-08-20T21:58:04.8379486Z ) 2024-08-20T21:58:04.8379965Z # Propagate download.pytorch.org IP to container 2024-08-20T21:58:04.8381165Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-20T21:58:04.8382386Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-20T21:58:04.8383506Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-20T21:58:04.8390627Z shell: /usr/bin/bash -e {0} 2024-08-20T21:58:04.8391060Z env: 2024-08-20T21:58:04.8391393Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:04.8391946Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:04.8392676Z BUILD_ENVIRONMENT: linux-focal-cuda11.8-py3.10-gcc9 2024-08-20T21:58:04.8393273Z PR_NUMBER: 133712 2024-08-20T21:58:04.8393676Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-20T21:58:04.8394182Z GITHUB_WORKFLOW: pull 2024-08-20T21:58:04.8394593Z GITHUB_JOB: test 2024-08-20T21:58:04.8394957Z GITHUB_RUN_ID: 10479309237 2024-08-20T21:58:04.8395398Z GITHUB_RUN_NUMBER: 238988 2024-08-20T21:58:04.8395843Z GITHUB_RUN_ATTEMPT: 1 2024-08-20T21:58:04.8396230Z JOB_ID: 29025850735 2024-08-20T21:58:04.8397087Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, amz2023.linux.8xlarge.nvidia.gpu) 2024-08-20T21:58:04.8398044Z BRANCH: pull/133712 2024-08-20T21:58:04.8398501Z SHA1: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:58:04.8399151Z BASE_SHA: 91f3d614142df02f619e44a68e3d9e0dfeba49ec 2024-08-20T21:58:04.8399732Z TEST_CONFIG: distributed 2024-08-20T21:58:04.8400155Z SHARD_NUMBER: 2 2024-08-20T21:58:04.8400516Z NUM_TEST_SHARDS: 3 2024-08-20T21:58:04.8400908Z REENABLED_ISSUES: 2024-08-20T21:58:04.8401301Z CONTINUE_THROUGH_ERROR: False 2024-08-20T21:58:04.8401775Z VERBOSE_TEST_LOGS: False 2024-08-20T21:58:04.8402208Z TEST_SHOWLOCALS: False 2024-08-20T21:58:04.8402614Z NO_TEST_TIMEOUT: False 2024-08-20T21:58:04.8403023Z NO_TD: False 2024-08-20T21:58:04.8403385Z TD_DISTRIBUTED: False 2024-08-20T21:58:04.8403879Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-20T21:58:04.8404495Z SCCACHE_S3_KEY_PREFIX: pull 2024-08-20T21:58:04.8404934Z SHM_SIZE: 2g 2024-08-20T21:58:04.8406127Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:58:04.8407413Z XLA_CUDA: 2024-08-20T21:58:04.8408007Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-20T21:58:04.8408768Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-20T21:58:04.8409294Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-20T21:58:04.8409799Z DASHBOARD_TAG: 2024-08-20T21:58:04.8410185Z HUGGING_FACE_HUB_TOKEN: 2024-08-20T21:58:04.8410609Z SCRIBE_GRAPHQL_ACCESS_TOKEN: 2024-08-20T21:58:04.8411063Z ##[endgroup] 2024-08-20T21:58:04.8438034Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-08-20T21:58:04.8438816Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *onnx* ]] 2024-08-20T21:58:04.8439427Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T21:58:04.8447676Z +++ nproc --ignore=2 2024-08-20T21:58:04.8466415Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10479309237 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:58:15.8925976Z + container_name=bf521d92eaa63669a33ef60d81f6abab162029657da975c0da2d1de2c74fc019 2024-08-20T21:58:15.8928523Z + grep download.pytorch.org /etc/hosts 2024-08-20T21:58:15.8930345Z + docker exec -i bf521d92eaa63669a33ef60d81f6abab162029657da975c0da2d1de2c74fc019 sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-20T21:58:16.0214129Z + echo DOCKER_CONTAINER_ID=bf521d92eaa63669a33ef60d81f6abab162029657da975c0da2d1de2c74fc019 2024-08-20T21:58:16.0218412Z ++ echo dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T21:58:16.0220987Z + docker exec -t bf521d92eaa63669a33ef60d81f6abab162029657da975c0da2d1de2c74fc019 sh -c 'pip install dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-20T21:58:16.5315353Z Processing ./dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+git40ec5f6) 2024-08-20T21:58:17.6612748Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.13.1) 2024-08-20T21:58:17.6617272Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (4.12.2) 2024-08-20T21:58:17.6622397Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.8.8) 2024-08-20T21:58:17.6627137Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.1.4) 2024-08-20T21:58:17.6632009Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2024.6.1) 2024-08-20T21:58:17.6639916Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.13.1) 2024-08-20T21:58:17.6663046Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.3.0) 2024-08-20T21:58:17.6683722Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.3.0) 2024-08-20T21:58:17.6708921Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.22.4) 2024-08-20T21:58:17.7238171Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.1.5) 2024-08-20T21:58:18.1532694Z Installing collected packages: torch 2024-08-20T21:58:31.1958338Z Successfully installed torch-2.5.0a0+git40ec5f6 2024-08-20T21:58:31.2700292Z + export TERM=vt100 2024-08-20T21:58:31.2700748Z + TERM=vt100 2024-08-20T21:58:31.2703979Z ++ dirname .ci/pytorch/test.sh 2024-08-20T21:58:31.2712622Z + source .ci/pytorch/common.sh 2024-08-20T21:58:31.2716355Z +++ dirname .ci/pytorch/common.sh 2024-08-20T21:58:31.2725247Z ++ source .ci/pytorch/common_utils.sh 2024-08-20T21:58:31.2727531Z +++ declare -f -t trap_add 2024-08-20T21:58:31.2734365Z ++ set -ex 2024-08-20T21:58:31.2734918Z ++ [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-08-20T21:58:31.2735515Z ++ BUILD_TEST_LIBTORCH=0 2024-08-20T21:58:31.2738706Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-08-20T21:58:31.2739688Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-20T21:58:31.2756357Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-20T21:58:31.2756921Z + trap_add cleanup_workspace EXIT 2024-08-20T21:58:31.2757427Z + trap_add_cmd=cleanup_workspace 2024-08-20T21:58:31.2757861Z + shift 2024-08-20T21:58:31.2758200Z + for trap_add_name in "$@" 2024-08-20T21:58:31.2765206Z +++ trap -p EXIT 2024-08-20T21:58:31.2768363Z ++ eval 'extract_trap_cmd ' 2024-08-20T21:58:31.2768824Z +++ extract_trap_cmd 2024-08-20T21:58:31.2769620Z +++ printf '%s\n' '' 2024-08-20T21:58:31.2770102Z ++ printf '%s\n' cleanup_workspace 2024-08-20T21:58:31.2772086Z + trap -- ' 2024-08-20T21:58:31.2772520Z cleanup_workspace' EXIT 2024-08-20T21:58:31.2773110Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-20T21:58:31.8730298Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-20T21:58:31.8752169Z + echo 'Environment variables:' 2024-08-20T21:58:31.8752690Z Environment variables: 2024-08-20T21:58:31.8753075Z + env 2024-08-20T21:58:31.8762809Z INSTALLED_DB=yes 2024-08-20T21:58:31.8763698Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-08-20T21:58:31.8764287Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T21:58:31.8764797Z NV_NVML_DEV_VERSION=11.8.86-1 2024-08-20T21:58:31.8765575Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T21:58:31.8766306Z CONTINUE_THROUGH_ERROR=False 2024-08-20T21:58:31.8766920Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-08-20T21:58:31.8767586Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-08-20T21:58:31.8768295Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-08-20T21:58:31.8768871Z HOSTNAME=bf521d92eaa6 2024-08-20T21:58:31.8770546Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_895bae81-5b51-4e20-9d1a-78a4ea1c758f 2024-08-20T21:58:31.8771577Z GITHUB_ACTION=__self 2024-08-20T21:58:31.8772004Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T21:58:31.8774701Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-08-20T21:58:31.8778122Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-08-20T21:58:31.8778768Z NV_NVTX_VERSION=11.8.86-1 2024-08-20T21:58:31.8779201Z GITHUB_RUN_NUMBER=238988 2024-08-20T21:58:31.8779613Z TEST_CONFIG=distributed 2024-08-20T21:58:31.8780053Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T21:58:31.8780663Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T21:58:31.8781237Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-08-20T21:58:31.8781832Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-08-20T21:58:31.8782359Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T21:58:31.8782863Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-08-20T21:58:31.8783440Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2024-08-20T21:58:31.8784115Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T21:58:31.8785002Z GITHUB_REF_TYPE=branch 2024-08-20T21:58:31.8785406Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T21:58:31.8785893Z NCCL_VERSION=2.16.2-1 2024-08-20T21:58:31.8786375Z BASE_SHA=91f3d614142df02f619e44a68e3d9e0dfeba49ec 2024-08-20T21:58:31.8786906Z XLA_CUDA= 2024-08-20T21:58:31.8787244Z HUGGING_FACE_HUB_TOKEN= 2024-08-20T21:58:31.8789905Z *** 2024-08-20T21:58:31.8790259Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T21:58:31.8790754Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T21:58:31.8791206Z GITHUB_ACTIONS=true 2024-08-20T21:58:31.8791834Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:31.8792547Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-08-20T21:58:31.8793223Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-08-20T21:58:31.8793794Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:58:31.8794437Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T21:58:31.8795045Z GITHUB_SHA=f2fb9405c2fa9f9502a76363091cce6fd8179736 2024-08-20T21:58:31.8795924Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/133712/merge 2024-08-20T21:58:31.8796736Z UCC_HOME=/usr 2024-08-20T21:58:31.8797159Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-08-20T21:58:31.8797642Z VERBOSE_TEST_LOGS=False 2024-08-20T21:58:31.8798096Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T21:58:31.8799044Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-08-20T21:58:31.8799620Z GITHUB_REF=refs/pull/133712/merge 2024-08-20T21:58:31.8800158Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-08-20T21:58:31.8800646Z SHARD_NUMBER=2 2024-08-20T21:58:31.8801032Z GITHUB_REF_PROTECTED=false 2024-08-20T21:58:31.8801498Z HOME=/var/lib/jenkins 2024-08-20T21:58:31.8801973Z GITHUB_API_URL=https://api.github.com 2024-08-20T21:58:31.8802536Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T21:58:31.8803162Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T21:58:31.8803785Z SCCACHE_S3_KEY_PREFIX=pull 2024-08-20T21:58:31.8804222Z CUDA_VERSION=11.8.0 2024-08-20T21:58:31.8804795Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-08-20T21:58:31.8805383Z NUM_TEST_SHARDS=3 2024-08-20T21:58:31.8805751Z UCX_HOME=/usr 2024-08-20T21:58:31.8806442Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-08-20T21:58:31.8807899Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_895bae81-5b51-4e20-9d1a-78a4ea1c758f 2024-08-20T21:58:31.8809614Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, amz2023.linux.8xlarge.nvidia.gpu) 2024-08-20T21:58:31.8811288Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_895bae81-5b51-4e20-9d1a-78a4ea1c758f 2024-08-20T21:58:31.8812806Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T21:58:31.8813691Z GITHUB_EVENT_NAME=pull_request 2024-08-20T21:58:31.8814151Z DASHBOARD_TAG= 2024-08-20T21:58:31.8814533Z GITHUB_RUN_ID=10479309237 2024-08-20T21:58:31.8815146Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-08-20T21:58:31.8815844Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-08-20T21:58:31.8817223Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_895bae81-5b51-4e20-9d1a-78a4ea1c758f 2024-08-20T21:58:31.8818392Z GITHUB_ACTOR=pytorchmergebot 2024-08-20T21:58:31.8818904Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-08-20T21:58:31.8819404Z PR_NUMBER=133712 2024-08-20T21:58:31.8819797Z GITHUB_RUN_ATTEMPT=1 2024-08-20T21:58:31.8820214Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T21:58:31.8820793Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T21:58:31.8821400Z TERM=vt100 2024-08-20T21:58:31.8821818Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-08-20T21:58:31.8822355Z INSTALLED_VISION=yes 2024-08-20T21:58:31.8822768Z BRANCH=pull/133712 2024-08-20T21:58:31.8823309Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T21:58:31.8823954Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T21:58:31.8824482Z CUDA_PATH=/usr/local/cuda 2024-08-20T21:58:31.8825456Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T21:58:31.8826547Z GITHUB_SERVER_URL=https://github.com 2024-08-20T21:58:31.8827150Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T21:58:31.8827714Z REENABLED_ISSUES= 2024-08-20T21:58:31.8828074Z SHLVL=1 2024-08-20T21:58:31.8828384Z MAX_JOBS=30 2024-08-20T21:58:31.8828760Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-08-20T21:58:31.8829191Z NVARCH=x86_64 2024-08-20T21:58:31.8829548Z GITHUB_ACTOR_ID=97764156 2024-08-20T21:58:31.8830098Z GITHUB_WORKFLOW_SHA=f2fb9405c2fa9f9502a76363091cce6fd8179736 2024-08-20T21:58:31.8830836Z GITHUB_REF_NAME=133712/merge 2024-08-20T21:58:31.8831412Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-08-20T21:58:31.8832244Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T21:58:31.8832954Z GITHUB_JOB=test 2024-08-20T21:58:31.8833452Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-08-20T21:58:31.8834123Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T21:58:31.8834757Z NO_TEST_TIMEOUT=False 2024-08-20T21:58:31.8835164Z TD_DISTRIBUTED=False 2024-08-20T21:58:31.8835622Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-08-20T21:58:31.8836164Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T21:58:31.8836680Z NV_NVPROF_VERSION=11.8.87-1 2024-08-20T21:58:31.8837104Z GITHUB_RETENTION_DAYS=90 2024-08-20T21:58:31.8837528Z OPENSSL_DIR=/opt/openssl 2024-08-20T21:58:31.8837952Z GITHUB_ACTION_REPOSITORY= 2024-08-20T21:58:31.8839260Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T21:58:31.8840654Z GITHUB_BASE_REF=gh/XuehaiPan/146/base 2024-08-20T21:58:31.8841174Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T21:58:31.8841606Z CI=true 2024-08-20T21:58:31.8842000Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-08-20T21:58:31.8842763Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T21:58:31.8843219Z JOB_ID=29025850735 2024-08-20T21:58:31.8843603Z INSTALLED_PROTOBUF=yes 2024-08-20T21:58:31.8844042Z GITHUB_HEAD_REF=gh/XuehaiPan/146/head 2024-08-20T21:58:31.8844528Z GITHUB_ACTION_REF= 2024-08-20T21:58:31.8845093Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T21:58:31.8845675Z TEST_SHOWLOCALS=False 2024-08-20T21:58:31.8846061Z GITHUB_WORKFLOW=pull 2024-08-20T21:58:31.8846477Z DEBIAN_FRONTEND=noninteractive 2024-08-20T21:58:31.8847643Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_895bae81-5b51-4e20-9d1a-78a4ea1c758f 2024-08-20T21:58:31.8848654Z NO_TD=False 2024-08-20T21:58:31.8849023Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T21:58:31.8849469Z _=/usr/bin/env 2024-08-20T21:58:31.8850057Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-20T21:58:31.9060470Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-08-20T21:58:31.9061626Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T21:58:31.9062738Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-08-20T21:58:31.9063857Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-08-20T21:58:31.9064621Z + BUILD_DIR=build 2024-08-20T21:58:31.9065015Z + BUILD_RENAMED_DIR=build_renamed 2024-08-20T21:58:31.9065477Z + BUILD_BIN_DIR=build/bin 2024-08-20T21:58:31.9065890Z + SHARD_NUMBER=2 2024-08-20T21:58:31.9066253Z + NUM_TEST_SHARDS=3 2024-08-20T21:58:31.9066636Z + export VALGRIND=ON 2024-08-20T21:58:31.9067027Z + VALGRIND=ON 2024-08-20T21:58:31.9067576Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *clang9* ]] 2024-08-20T21:58:31.9068148Z + [[ 0 == \1 ]] 2024-08-20T21:58:31.9068509Z + [[ False == \1 ]] 2024-08-20T21:58:31.9069540Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *bazel* ]] 2024-08-20T21:58:31.9070170Z ++ realpath build/custom_test_artifacts 2024-08-20T21:58:31.9078846Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-20T21:58:31.9079739Z + [[ -n '' ]] 2024-08-20T21:58:31.9080162Z + echo 'Environment variables' 2024-08-20T21:58:31.9080867Z Environment variables 2024-08-20T21:58:31.9081260Z + env 2024-08-20T21:58:31.9087669Z INSTALLED_DB=yes 2024-08-20T21:58:31.9088156Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-08-20T21:58:31.9088642Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T21:58:31.9089101Z NV_NVML_DEV_VERSION=11.8.86-1 2024-08-20T21:58:31.9089855Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T21:58:31.9090577Z CONTINUE_THROUGH_ERROR=False 2024-08-20T21:58:31.9091198Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-08-20T21:58:31.9092082Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-08-20T21:58:31.9092820Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-08-20T21:58:31.9093411Z HOSTNAME=bf521d92eaa6 2024-08-20T21:58:31.9094491Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_895bae81-5b51-4e20-9d1a-78a4ea1c758f 2024-08-20T21:58:31.9095494Z GITHUB_ACTION=__self 2024-08-20T21:58:31.9095917Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T21:58:31.9098613Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-08-20T21:58:31.9101456Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-08-20T21:58:31.9102095Z NV_NVTX_VERSION=11.8.86-1 2024-08-20T21:58:31.9102531Z GITHUB_RUN_NUMBER=238988 2024-08-20T21:58:31.9102937Z TEST_CONFIG=distributed 2024-08-20T21:58:31.9103371Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T21:58:31.9103970Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T21:58:31.9104531Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-08-20T21:58:31.9105113Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-08-20T21:58:31.9105641Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T21:58:31.9106148Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-08-20T21:58:31.9106682Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2024-08-20T21:58:31.9107330Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T21:58:31.9107948Z GITHUB_REF_TYPE=branch 2024-08-20T21:58:31.9108377Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T21:58:31.9108876Z NCCL_VERSION=2.16.2-1 2024-08-20T21:58:31.9109370Z BASE_SHA=91f3d614142df02f619e44a68e3d9e0dfeba49ec 2024-08-20T21:58:31.9109945Z XLA_CUDA= 2024-08-20T21:58:31.9110304Z HUGGING_FACE_HUB_TOKEN= 2024-08-20T21:58:31.9110814Z *** 2024-08-20T21:58:31.9111161Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T21:58:31.9111676Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T21:58:31.9112150Z GITHUB_ACTIONS=true 2024-08-20T21:58:31.9112558Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:31.9113216Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-08-20T21:58:31.9113926Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-08-20T21:58:31.9114529Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:58:31.9115202Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T21:58:31.9115846Z GITHUB_SHA=f2fb9405c2fa9f9502a76363091cce6fd8179736 2024-08-20T21:58:31.9116774Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/133712/merge 2024-08-20T21:58:31.9117622Z UCC_HOME=/usr 2024-08-20T21:58:31.9118059Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-08-20T21:58:31.9118575Z VERBOSE_TEST_LOGS=False 2024-08-20T21:58:31.9119017Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T21:58:31.9119627Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-08-20T21:58:31.9120235Z GITHUB_REF=refs/pull/133712/merge 2024-08-20T21:58:31.9120791Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-08-20T21:58:31.9121279Z SHARD_NUMBER=2 2024-08-20T21:58:31.9121653Z GITHUB_REF_PROTECTED=false 2024-08-20T21:58:31.9122108Z HOME=/var/lib/jenkins 2024-08-20T21:58:31.9122581Z GITHUB_API_URL=https://api.github.com 2024-08-20T21:58:31.9123136Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T21:58:31.9123762Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T21:58:31.9124521Z SCCACHE_S3_KEY_PREFIX=pull 2024-08-20T21:58:31.9124960Z CUDA_VERSION=11.8.0 2024-08-20T21:58:31.9125529Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-08-20T21:58:31.9126104Z NUM_TEST_SHARDS=3 2024-08-20T21:58:31.9126474Z UCX_HOME=/usr 2024-08-20T21:58:31.9127159Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-08-20T21:58:31.9128635Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_895bae81-5b51-4e20-9d1a-78a4ea1c758f 2024-08-20T21:58:31.9130422Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, amz2023.linux.8xlarge.nvidia.gpu) 2024-08-20T21:58:31.9132136Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_895bae81-5b51-4e20-9d1a-78a4ea1c758f 2024-08-20T21:58:31.9133650Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T21:58:31.9134520Z GITHUB_EVENT_NAME=pull_request 2024-08-20T21:58:31.9134995Z DASHBOARD_TAG= 2024-08-20T21:58:31.9135387Z GITHUB_RUN_ID=10479309237 2024-08-20T21:58:31.9135983Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-08-20T21:58:31.9136697Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-08-20T21:58:31.9138057Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_895bae81-5b51-4e20-9d1a-78a4ea1c758f 2024-08-20T21:58:31.9139231Z GITHUB_ACTOR=pytorchmergebot 2024-08-20T21:58:31.9139749Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-08-20T21:58:31.9140246Z PR_NUMBER=133712 2024-08-20T21:58:31.9140645Z GITHUB_RUN_ATTEMPT=1 2024-08-20T21:58:31.9141031Z VALGRIND=ON 2024-08-20T21:58:31.9141410Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T21:58:31.9141978Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T21:58:31.9142583Z TERM=vt100 2024-08-20T21:58:31.9143016Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-08-20T21:58:31.9143535Z INSTALLED_VISION=yes 2024-08-20T21:58:31.9143948Z BRANCH=pull/133712 2024-08-20T21:58:31.9144366Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T21:58:31.9144880Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T21:58:31.9145434Z CUDA_PATH=/usr/local/cuda 2024-08-20T21:58:31.9146459Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T21:58:31.9147469Z GITHUB_SERVER_URL=https://github.com 2024-08-20T21:58:31.9148096Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T21:58:31.9148705Z REENABLED_ISSUES= 2024-08-20T21:58:31.9149079Z SHLVL=1 2024-08-20T21:58:31.9149393Z MAX_JOBS=30 2024-08-20T21:58:31.9149808Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-08-20T21:58:31.9150253Z NVARCH=x86_64 2024-08-20T21:58:31.9150638Z GITHUB_ACTOR_ID=97764156 2024-08-20T21:58:31.9151240Z GITHUB_WORKFLOW_SHA=f2fb9405c2fa9f9502a76363091cce6fd8179736 2024-08-20T21:58:31.9151896Z GITHUB_REF_NAME=133712/merge 2024-08-20T21:58:31.9152482Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-08-20T21:58:31.9153377Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T21:58:31.9154122Z GITHUB_JOB=test 2024-08-20T21:58:31.9154646Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-08-20T21:58:31.9155377Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T21:58:31.9156048Z NO_TEST_TIMEOUT=False 2024-08-20T21:58:31.9156453Z TD_DISTRIBUTED=False 2024-08-20T21:58:31.9156947Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-08-20T21:58:31.9157522Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T21:58:31.9158059Z NV_NVPROF_VERSION=11.8.87-1 2024-08-20T21:58:31.9158530Z GITHUB_RETENTION_DAYS=90 2024-08-20T21:58:31.9158971Z OPENSSL_DIR=/opt/openssl 2024-08-20T21:58:31.9159427Z GITHUB_ACTION_REPOSITORY= 2024-08-20T21:58:31.9160819Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T21:58:31.9162296Z GITHUB_BASE_REF=gh/XuehaiPan/146/base 2024-08-20T21:58:31.9162830Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T21:58:31.9163306Z CI=true 2024-08-20T21:58:31.9163740Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-08-20T21:58:31.9164383Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T21:58:31.9164865Z JOB_ID=29025850735 2024-08-20T21:58:31.9165261Z INSTALLED_PROTOBUF=yes 2024-08-20T21:58:31.9165707Z GITHUB_HEAD_REF=gh/XuehaiPan/146/head 2024-08-20T21:58:31.9166230Z GITHUB_ACTION_REF= 2024-08-20T21:58:31.9166814Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T21:58:31.9192475Z TEST_SHOWLOCALS=False 2024-08-20T21:58:31.9192964Z GITHUB_WORKFLOW=pull 2024-08-20T21:58:31.9193395Z DEBIAN_FRONTEND=noninteractive 2024-08-20T21:58:31.9194844Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_895bae81-5b51-4e20-9d1a-78a4ea1c758f 2024-08-20T21:58:31.9195891Z NO_TD=False 2024-08-20T21:58:31.9196234Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T21:58:31.9196660Z _=/usr/bin/env 2024-08-20T21:58:31.9197063Z + echo 'Testing pytorch' 2024-08-20T21:58:31.9197449Z Testing pytorch 2024-08-20T21:58:31.9197811Z + export LANG=C.UTF-8 2024-08-20T21:58:31.9198214Z + LANG=C.UTF-8 2024-08-20T21:58:31.9198583Z + PR_NUMBER=133712 2024-08-20T21:58:31.9198976Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-08-20T21:58:31.9199536Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-20T21:58:31.9200255Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-08-20T21:58:31.9200834Z + [[ distributed == \s\l\o\w ]] 2024-08-20T21:58:31.9201536Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *slow-gradcheck* ]] 2024-08-20T21:58:31.9202314Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-08-20T21:58:31.9202943Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T21:58:31.9203493Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T21:58:31.9203986Z + [[ distributed == *crossref* ]] 2024-08-20T21:58:31.9204596Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-08-20T21:58:31.9205300Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *xpu* ]] 2024-08-20T21:58:31.9206050Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-08-20T21:58:31.9206697Z + pip_install --user ninja==1.10.2 2024-08-20T21:58:31.9207347Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-20T21:58:32.4301340Z Collecting ninja==1.10.2 2024-08-20T21:58:32.4532414Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-20T21:58:32.4685938Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-20T21:58:32.9056456Z Installing collected packages: ninja 2024-08-20T21:58:32.9159968Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-20T21:58:32.9161620Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-20T21:58:32.9220019Z Successfully installed ninja-1.10.2 2024-08-20T21:58:32.9914574Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T21:58:32.9917345Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T21:58:32.9919211Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-08-20T21:58:32.9919801Z + install_tlparse 2024-08-20T21:58:32.9920281Z + pip_install --user tlparse==0.3.25 2024-08-20T21:58:32.9920993Z + pip install --progress-bar off --user tlparse==0.3.25 2024-08-20T21:58:33.4972164Z Collecting tlparse==0.3.25 2024-08-20T21:58:33.5183752Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-08-20T21:58:33.5341590Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-20T21:58:33.9982261Z Installing collected packages: tlparse 2024-08-20T21:58:34.0405891Z Successfully installed tlparse-0.3.25 2024-08-20T21:58:34.1152108Z ++ python -m site --user-base 2024-08-20T21:58:34.1463907Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T21:58:34.1466347Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *asan* ]] 2024-08-20T21:58:34.1467158Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-debug* ]] 2024-08-20T21:58:34.1467946Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-08-20T21:58:34.1469653Z + echo 'We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass' 2024-08-20T21:58:34.1471120Z We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass 2024-08-20T21:58:34.1471950Z + cd test 2024-08-20T21:58:34.1472622Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-20T21:58:36.2351981Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-20T21:58:36.2352691Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-20T21:58:36.2357073Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-20T21:58:36.2357999Z + [[ distributed == *pr_time_benchmarks* ]] 2024-08-20T21:58:36.2358591Z + [[ distributed == *dynamo_eager* ]] 2024-08-20T21:58:36.2359114Z + [[ distributed == *aot_eager* ]] 2024-08-20T21:58:36.2359607Z + [[ distributed == *aot_inductor* ]] 2024-08-20T21:58:36.2360127Z + [[ distributed == *inductor* ]] 2024-08-20T21:58:36.2360620Z + [[ distributed == *dynamic* ]] 2024-08-20T21:58:36.2361082Z + [[ distributed == *cpu* ]] 2024-08-20T21:58:36.2361848Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-20T21:58:36.2401151Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-08-20T21:58:36.2401944Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-bazel-* ]] 2024-08-20T21:58:36.2404346Z + cd test 2024-08-20T21:58:36.2404947Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-20T21:58:38.1167775Z PyTorch built with: 2024-08-20T21:58:38.1168460Z - GCC 9.4 2024-08-20T21:58:38.1168895Z - C++ Version: 201703 2024-08-20T21:58:38.1170483Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T21:58:38.1171847Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T21:58:38.1172689Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T21:58:38.1173338Z - LAPACK is enabled (usually provided by MKL) 2024-08-20T21:58:38.1173927Z - NNPACK is enabled 2024-08-20T21:58:38.1174404Z - CPU capability usage: AVX2 2024-08-20T21:58:38.1174924Z - CUDA Runtime 11.8 2024-08-20T21:58:38.1175568Z - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52 2024-08-20T21:58:38.1176256Z - CuDNN 90.1 2024-08-20T21:58:38.1176651Z - Magma 2.6.1 2024-08-20T21:58:38.1185760Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-20T21:58:38.1195637Z 2024-08-20T21:58:38.3851935Z + cd test 2024-08-20T21:58:38.3852930Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-20T21:58:40.2486610Z ATen/Parallel: 2024-08-20T21:58:40.2487556Z at::get_num_threads() : 16 2024-08-20T21:58:40.2488493Z at::get_num_interop_threads() : 16 2024-08-20T21:58:40.2489003Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T21:58:40.2489500Z omp_get_max_threads() : 16 2024-08-20T21:58:40.2491169Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T21:58:40.2492222Z mkl_get_max_threads() : 16 2024-08-20T21:58:40.2493020Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T21:58:40.2493819Z std::thread::hardware_concurrency() : 32 2024-08-20T21:58:40.2494349Z Environment variables: 2024-08-20T21:58:40.2494758Z OMP_NUM_THREADS : [not set] 2024-08-20T21:58:40.2495215Z MKL_NUM_THREADS : [not set] 2024-08-20T21:58:40.2495679Z ATen parallel backend: OpenMP 2024-08-20T21:58:40.2496021Z 2024-08-20T21:58:40.5014790Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-08-20T21:58:40.5015510Z + [[ distributed == *backward* ]] 2024-08-20T21:58:40.5016031Z + [[ distributed == *xla* ]] 2024-08-20T21:58:40.5016487Z + [[ distributed == *executorch* ]] 2024-08-20T21:58:40.5017026Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-20T21:58:40.5017782Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-08-20T21:58:40.5018395Z + [[ distributed == distributed ]] 2024-08-20T21:58:40.5018899Z + test_distributed 2024-08-20T21:58:40.5019408Z + echo 'Testing distributed python tests' 2024-08-20T21:58:40.5019944Z Testing distributed python tests 2024-08-20T21:58:40.5020721Z + python test/run_test.py --distributed-tests --shard 2 3 --verbose 2024-08-20T21:58:40.6342844Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T21:58:40.6344297Z import pkg_resources 2024-08-20T21:58:45.2172948Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T21:58:45.2762299Z Ignoring disabled issues: [''] 2024-08-20T21:58:45.2905547Z Found test times from artifacts 2024-08-20T21:58:45.3454384Z Found test times from artifacts 2024-08-20T21:58:45.3472273Z Running 25% of tests based on TD 2024-08-20T21:58:45.3594694Z Running parallel tests on 2 processes 2024-08-20T21:58:45.3596946Z Name: tests to run (est. time: 10.22min) 2024-08-20T21:58:45.3598018Z Serial tests (21): 2024-08-20T21:58:45.3598936Z distributed/test_dynamo_distributed 1/1 2024-08-20T21:58:45.3599571Z distributed/pipelining/test_transformer 1/1 2024-08-20T21:58:45.3600175Z distributed/_tools/test_mod_tracker 1/1 2024-08-20T21:58:45.3600760Z distributed/checkpoint/test_nested_dict 1/1 2024-08-20T21:58:45.3601764Z distributed/test_fake_pg 1/1 2024-08-20T21:58:45.3602917Z distributed/pipelining/test_backward 1/1 2024-08-20T21:58:45.3603701Z distributed/fsdp/test_checkpoint_wrapper 1/1 2024-08-20T21:58:45.3604308Z distributed/test_symmetric_memory 1/1 2024-08-20T21:58:45.3604926Z distributed/flight_recorder/test_fr_analysis 1/1 2024-08-20T21:58:45.3605613Z distributed/_shard/sharded_tensor/test_logger 1/1 2024-08-20T21:58:45.3606219Z distributed/_tensor/test_dtensor 1/1 2024-08-20T21:58:45.3606784Z distributed/_tensor/test_random_ops 1/1 2024-08-20T21:58:45.3607405Z distributed/_tensor/test_experimental_ops 1/1 2024-08-20T21:58:45.3607983Z distributed/fsdp/test_fsdp_fx 1/1 2024-08-20T21:58:45.3608531Z distributed/_tensor/test_tensor_ops 1/1 2024-08-20T21:58:45.3609209Z distributed/_composable/fsdp/test_fully_shard_overlap 1/1 2024-08-20T21:58:45.3609954Z distributed/_tensor/experimental/test_tp_transform 1/1 2024-08-20T21:58:45.3610681Z distributed/test_compute_comm_reordering 1/1 2024-08-20T21:58:45.3611317Z distributed/checkpoint/e2e/test_fsdp_ep 1/1 2024-08-20T21:58:45.3612193Z distributed/_tools/test_mem_tracker 1/1 2024-08-20T21:58:45.3612875Z distributed/_composable/fsdp/test_fully_shard_training 1/1 2024-08-20T21:58:45.3613518Z Parallel tests (0): 2024-08-20T21:58:45.3613931Z Name: excluded (est. time: 76.52min) 2024-08-20T21:58:45.3614426Z Serial tests (54): 2024-08-20T21:58:45.3614870Z distributed/test_distributed_spawn 2/9 2024-08-20T21:58:45.3615428Z distributed/test_distributed_spawn 5/9 2024-08-20T21:58:45.3616173Z distributed/test_distributed_spawn 8/9 2024-08-20T21:58:45.3616742Z distributed/test_c10d_nccl 2/2 2024-08-20T21:58:45.3617265Z distributed/fsdp/test_fsdp_state_dict 2/2 2024-08-20T21:58:45.3617892Z distributed/rpc/cuda/test_tensorpipe_agent 2/2 2024-08-20T21:58:45.3618485Z distributed/test_c10d_ucc 1/1 2024-08-20T21:58:45.3619079Z distributed/optim/test_zero_redundancy_optimizer 1/1 2024-08-20T21:58:45.3619794Z distributed/_composable/fsdp/test_fully_shard_comm 1/1 2024-08-20T21:58:45.3620493Z distributed/fsdp/test_fsdp_use_orig_params 1/1 2024-08-20T21:58:45.3621126Z distributed/pipelining/test_schedule 1/1 2024-08-20T21:58:45.3621730Z distributed/fsdp/test_fsdp_unshard_params 1/1 2024-08-20T21:58:45.3622327Z distributed/fsdp/test_fsdp_misc 1/1 2024-08-20T21:58:45.3623001Z distributed/algorithms/quantization/test_quantization 1/1 2024-08-20T21:58:45.3623679Z distributed/_tensor/test_optimizers 1/1 2024-08-20T21:58:45.3624330Z distributed/tensor/parallel/test_tp_examples 1/1 2024-08-20T21:58:45.3624991Z distributed/fsdp/test_fsdp_hybrid_shard 1/1 2024-08-20T21:58:45.3625573Z distributed/fsdp/test_fsdp_overlap 1/1 2024-08-20T21:58:45.3626139Z distributed/algorithms/test_join 1/1 2024-08-20T21:58:45.3626728Z distributed/_tools/test_fsdp2_mem_tracker 1/1 2024-08-20T21:58:45.3627427Z distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 2024-08-20T21:58:45.3628181Z distributed/elastic/multiprocessing/api_test 1/1 2024-08-20T21:58:45.3628925Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1 2024-08-20T21:58:45.3629651Z distributed/checkpoint/test_dtensor_resharding 1/1 2024-08-20T21:58:45.3630379Z distributed/checkpoint/test_file_system_checkpoint 1/1 2024-08-20T21:58:45.3631144Z distributed/_composable/fsdp/test_fully_shard_autograd 1/1 2024-08-20T21:58:45.3632006Z distributed/_composable/fully_shard/test_fully_shard_model_checkpoint 1/1 2024-08-20T21:58:45.3632857Z distributed/_tensor/experimental/test_local_map 1/1 2024-08-20T21:58:45.3633600Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 2024-08-20T21:58:45.3634270Z distributed/fsdp/test_fsdp_apply 1/1 2024-08-20T21:58:45.3634850Z distributed/fsdp/test_fsdp_tp_integration 1/1 2024-08-20T21:58:45.3635550Z distributed/_shard/sharding_spec/test_sharding_spec 1/1 2024-08-20T21:58:45.3636359Z distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 2024-08-20T21:58:45.3637132Z distributed/checkpoint/test_fsdp_optim_state 1/1 2024-08-20T21:58:45.3637761Z distributed/fsdp/test_shard_utils 1/1 2024-08-20T21:58:45.3638381Z distributed/checkpoint/test_fsdp_model_state 1/1 2024-08-20T21:58:45.3639089Z distributed/_shard/sharded_optim/test_sharded_optim 1/1 2024-08-20T21:58:45.3639862Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 1/1 2024-08-20T21:58:45.3640635Z distributed/_composable/fsdp/test_fully_shard_init 1/1 2024-08-20T21:58:45.3641423Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 2024-08-20T21:58:45.3642132Z distributed/fsdp/test_fsdp_uneven 1/1 2024-08-20T21:58:45.3642729Z distributed/checkpoint/e2e/test_pipeline 1/1 2024-08-20T21:58:45.3643438Z distributed/_composable/fully_shard/test_fully_shard_util 1/1 2024-08-20T21:58:45.3644209Z distributed/tensor/parallel/test_tp_random_state 1/1 2024-08-20T21:58:45.3644859Z distributed/test_multi_threaded_pg 1/1 2024-08-20T21:58:45.3645383Z distributed/test_nccl 1/1 2024-08-20T21:58:45.3646013Z distributed/_tensor/debug/test_comm_mode 1/1 2024-08-20T21:58:45.3646656Z distributed/checkpoint/test_compatibility 1/1 2024-08-20T21:58:45.3647259Z distributed/elastic/events/lib_test 1/1 2024-08-20T21:58:45.3647854Z distributed/checkpoint/test_traverse 1/1 2024-08-20T21:58:45.3648457Z distributed/elastic/metrics/api_test 1/1 2024-08-20T21:58:45.3649169Z distributed/_composable/fully_shard/test_fully_shard_compile 1/1 2024-08-20T21:58:45.3650079Z distributed/_composable/fsdp/test_fully_shard_logging 1/1 2024-08-20T21:58:45.3650789Z distributed/rpc/test_tensorpipe_agent 1/1 2024-08-20T21:58:45.3651342Z Parallel tests (0): 2024-08-20T21:58:45.3674363Z Running distributed/test_dynamo_distributed 1/1 ... [2024-08-20 21:58:45.367078] 2024-08-20T21:58:45.3675174Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T21:58:45.3679735Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_dynamo_distributed.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 21:58:45.367534] 2024-08-20T22:00:03.9315030Z 2024-08-20T22:00:03.9316898Z distributed/test_dynamo_distributed 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_dynamo_distributed_1.1_1cf361ef7cc99d01_.log 2024-08-20T22:00:03.9350616Z Running 48 items in this shard: test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_call_method_forward, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_issue90375, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_symbol_splitting, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_direct, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_indirect, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_no_binding, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_torture_multi, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_asymmetric_compilation, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_asymmetric_compilation_with_fx_cache, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_scalar, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_speculation_divergence, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_tensor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_graph_break_empty_graph_still_collective, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_baseline_aot_eager_multiprocess, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_setattr, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager_static_graph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor_static_graph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_aot_autograd, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_async_subclass_no_specialize, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_custom_layer, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_aot_eager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_empty_graph_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_diff_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_same_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_orig_params_assert, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_guards, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_register_attr_or_module, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_staticmethod, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_ctx_manager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_inference, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_training, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_transpose, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_higher_order_op, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ignored_parameters, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_no_split 2024-08-20T22:00:03.9380611Z 2024-08-20T22:00:03.9381399Z Running distributed/pipelining/test_transformer 1/1 ... [2024-08-20 22:00:03.932146] 2024-08-20T22:00:03.9382235Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:00:03.9384490Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_transformer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:00:03.932642] 2024-08-20T22:00:08.6063110Z 2024-08-20T22:00:08.6065341Z distributed/pipelining/test_transformer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_transformer_1.1_ca12b2c9456d111c_.log 2024-08-20T22:00:08.6067402Z Running 1 items in this shard: test/distributed/pipelining/test_transformer.py::TransformerTests::test_ir 2024-08-20T22:00:08.6068242Z 2024-08-20T22:00:08.6068802Z Running distributed/_tools/test_mod_tracker 1/1 ... [2024-08-20 22:00:08.606403] 2024-08-20T22:00:08.6070298Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:00:08.6072760Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_mod_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:00:08.606847] 2024-08-20T22:00:12.4789282Z 2024-08-20T22:00:12.4791174Z distributed/_tools/test_mod_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_mod_tracker_1.1_fe29e931c78a3104_.log 2024-08-20T22:00:12.4794616Z Running 4 items in this shard: test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_ac, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_bw_detection, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_module_hierarchy, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_user_hooks 2024-08-20T22:00:12.4797245Z 2024-08-20T22:00:12.4797838Z Running distributed/checkpoint/test_nested_dict 1/1 ... [2024-08-20 22:00:12.479232] 2024-08-20T22:00:12.4798670Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:00:12.4802107Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_nested_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:00:12.479678] 2024-08-20T22:00:16.3516809Z 2024-08-20T22:00:16.3519563Z distributed/checkpoint/test_nested_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_nested_dict_1.1_dc19e716f04a7ca4_.log 2024-08-20T22:00:16.3524021Z Running 2 items in this shard: test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_flattening_round_trip, test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_mapping 2024-08-20T22:00:16.3525507Z 2024-08-20T22:00:16.3526014Z Running distributed/test_fake_pg 1/1 ... [2024-08-20 22:00:16.352023] 2024-08-20T22:00:16.3527002Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:00:16.3531593Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_fake_pg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:00:16.352475] 2024-08-20T22:00:20.7755062Z 2024-08-20T22:00:20.7757205Z distributed/test_fake_pg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_fake_pg_1.1_469e2e41162ce46a_.log 2024-08-20T22:00:20.7763953Z Running 13 items in this shard: test/distributed/test_fake_pg.py::TestFakePG::test_all_reduce, test/distributed/test_fake_pg.py::TestFakePG::test_allgather, test/distributed/test_fake_pg.py::TestFakePG::test_alltoall, test/distributed/test_fake_pg.py::TestFakePG::test_alltoall_base, test/distributed/test_fake_pg.py::TestFakePG::test_broadcast, test/distributed/test_fake_pg.py::TestFakePG::test_construct_fsdp, test/distributed/test_fake_pg.py::TestFakePG::test_fake_pg_tracing, test/distributed/test_fake_pg.py::TestFakePG::test_fsdp_fake_e2e, test/distributed/test_fake_pg.py::TestFakePG::test_fsdp_tp_fake_e2e, test/distributed/test_fake_pg.py::TestFakePG::test_recv, test/distributed/test_fake_pg.py::TestFakePG::test_reduce_scatter, test/distributed/test_fake_pg.py::TestFakePG::test_scatter, test/distributed/test_fake_pg.py::TestFakePG::test_send 2024-08-20T22:00:20.7769987Z 2024-08-20T22:00:20.7770604Z Running distributed/pipelining/test_backward 1/1 ... [2024-08-20 22:00:20.775783] 2024-08-20T22:00:20.7771420Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:00:20.7773621Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_backward.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:00:20.776239] 2024-08-20T22:00:24.7484956Z 2024-08-20T22:00:24.7487568Z distributed/pipelining/test_backward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_backward_1.1_45bbedc9b1ada6f5_.log 2024-08-20T22:00:24.7491356Z Running 1 items in this shard: test/distributed/pipelining/test_backward.py::StageBackwardTests::test_stage_backward 2024-08-20T22:00:24.7492442Z 2024-08-20T22:00:24.7493100Z Running distributed/fsdp/test_checkpoint_wrapper 1/1 ... [2024-08-20 22:00:24.748752] 2024-08-20T22:00:24.7493952Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:00:24.7497028Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_checkpoint_wrapper.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:00:24.749196] 2024-08-20T22:00:28.9719972Z 2024-08-20T22:00:28.9722106Z distributed/fsdp/test_checkpoint_wrapper 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_checkpoint_wrapper_1.1_85265c3d83008866_.log 2024-08-20T22:00:28.9729389Z Running 8 items in this shard: test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_apply_activation_checkpointing, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_args_kwargs, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_cpu_offload, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_kwarg_support, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_parity, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_forward_missing_attributes, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_fqn, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_load_activation_checkpointed_module 2024-08-20T22:00:28.9735113Z 2024-08-20T22:00:28.9735665Z Running distributed/test_symmetric_memory 1/1 ... [2024-08-20 22:00:28.972288] 2024-08-20T22:00:28.9736457Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:00:28.9738640Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_symmetric_memory.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:00:28.972724] 2024-08-20T22:00:32.5943320Z 2024-08-20T22:00:32.5945535Z distributed/test_symmetric_memory 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_symmetric_memory_1.1_7f26551526830141_.log 2024-08-20T22:00:32.5960605Z Running 20 items in this shard: test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_cuda_nvlink_connectivity_detection, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_empty_strided_p2p, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_empty_strided_p2p_persistent, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_gather_dim_0, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_gather_dim_1, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_0, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_1, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_matmul_reduce_scatter_scatter_dim_0, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_matmul_reduce_scatter_scatter_dim_1, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_0, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_1, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_all_gather_symm_mem_input_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_all_gather_symm_mem_input_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_avg_symm_mem_input_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_avg_symm_mem_input_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_sum_symm_mem_input_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_sum_symm_mem_input_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_optimal_layout_dim_0, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_optimal_layout_dim_1, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_optimal_layout_dim_2 2024-08-20T22:00:32.5975910Z 2024-08-20T22:00:32.5976574Z Running distributed/flight_recorder/test_fr_analysis 1/1 ... [2024-08-20 22:00:32.594611] 2024-08-20T22:00:32.5977437Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:00:32.5979910Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/flight_recorder/test_fr_analysis.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:00:32.595050] 2024-08-20T22:00:36.4304664Z 2024-08-20T22:00:36.4307044Z distributed/flight_recorder/test_fr_analysis 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.flight_recorder.test_fr_analysis_1.1_bd2167f53d19fde1_.log 2024-08-20T22:00:36.4309492Z Running 1 items in this shard: test/distributed/flight_recorder/test_fr_analysis.py::FlightRecorderEventTest::test_match_one_event 2024-08-20T22:00:36.4310508Z 2024-08-20T22:00:36.4311137Z Running distributed/_shard/sharded_tensor/test_logger 1/1 ... [2024-08-20 22:00:36.430736] 2024-08-20T22:00:36.4311989Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:00:36.4315936Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_logger.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:00:36.431166] 2024-08-20T22:00:40.2528197Z 2024-08-20T22:00:40.2530433Z distributed/_shard/sharded_tensor/test_logger 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.test_logger_1.1_4b3310b7010c2aa7_.log 2024-08-20T22:00:40.2532699Z Running 1 items in this shard: test/distributed/_shard/sharded_tensor/test_logger.py::ShardingSpecLoggerTest::test_get_or_create_logger 2024-08-20T22:00:40.2533738Z 2024-08-20T22:00:40.2534267Z Running distributed/_tensor/test_dtensor 1/1 ... [2024-08-20 22:00:40.253103] 2024-08-20T22:00:40.2535040Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:00:40.2539540Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:00:40.253531] 2024-08-20T22:03:03.8151011Z 2024-08-20T22:03:03.8154994Z distributed/_tensor/test_dtensor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_1.1_008bab50afdf08c7_.log 2024-08-20T22:03:03.8174010Z Running 32 items in this shard: test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_async_output, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_constructor, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_new_empty_strided, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_properties, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_save_load, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_spec_hash, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_spec_read_only_after_set, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_stride, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_negative_dim, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_then_to_local, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_uneven_sharding, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_uneven_sharding_raise_error, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_full_tensor_grad_hint, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_full_tensor_sync, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_meta_dtensor, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_modules_w_meta_dtensor, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_to_local, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_to_local_grad_hint, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_auto_implicit_replication, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_default_value_sub_mesh, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_device_mesh_nd, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_2d_mesh, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_api_device_mesh_context_manager, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_device_mesh_device_conversion, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_spec_local_shard_offset, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_from_local_sub_mesh, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_implicit_replication, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_metadata_consistency_check, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_redistribute_sub_mesh, test/distributed/_tensor/test_dtensor.py::TestDTensorPlacementTypes::test_split_tensor_1D, test/distributed/_tensor/test_dtensor.py::DTensorLogTest::test_dtensor_log 2024-08-20T22:03:03.8191082Z 2024-08-20T22:03:03.8191706Z Running distributed/_tensor/test_random_ops 1/1 ... [2024-08-20 22:03:03.815591] 2024-08-20T22:03:03.8192489Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:03:03.8194678Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_random_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:03:03.816105] 2024-08-20T22:03:37.4960759Z 2024-08-20T22:03:37.4962670Z distributed/_tensor/test_random_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_random_ops_1.1_f692d262105ee15d_.log 2024-08-20T22:03:37.4969897Z Running 7 items in this shard: test/distributed/_tensor/test_random_ops.py::DistTensorRandomInitTest::test_init_ops, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_dropout_1d, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_rand_1d, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_uniform_2d, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_manual_seed, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_meta_tensor_init, test/distributed/_tensor/test_random_ops.py::DistTensorRandomOpTest::test_rng_tracker_init 2024-08-20T22:03:37.4974610Z 2024-08-20T22:03:37.4975288Z Running distributed/_tensor/test_experimental_ops 1/1 ... [2024-08-20 22:03:37.496306] 2024-08-20T22:03:37.4976145Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:03:37.4978381Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_experimental_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:03:37.496758] 2024-08-20T22:03:55.1463840Z 2024-08-20T22:03:55.1466035Z distributed/_tensor/test_experimental_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_experimental_ops_1.1_03f6d0fd47302058_.log 2024-08-20T22:03:55.1469608Z Running 3 items in this shard: test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_bernoulli, test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_nll, test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_slice 2024-08-20T22:03:55.1471894Z 2024-08-20T22:03:55.1472457Z Running distributed/fsdp/test_fsdp_fx 1/1 ... [2024-08-20 22:03:55.146559] 2024-08-20T22:03:55.1473226Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:03:55.1475385Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fx.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:03:55.147076] 2024-08-20T22:03:59.0190813Z 2024-08-20T22:03:59.0192955Z distributed/fsdp/test_fsdp_fx 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_fx_1.1_d13d1eb22ea37849_.log 2024-08-20T22:03:59.0195033Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_fx.py::TestSymbolicTracing::test_symbolic_tracing_outputs 2024-08-20T22:03:59.0195985Z 2024-08-20T22:03:59.0198502Z Running distributed/_tensor/test_tensor_ops 1/1 ... [2024-08-20 22:03:59.019495] 2024-08-20T22:03:59.0199337Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:03:59.0204223Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_tensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:03:59.019983] 2024-08-20T22:05:58.6683659Z 2024-08-20T22:05:58.6685641Z distributed/_tensor/test_tensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_tensor_ops_1.1_c0f2a6f34f3f2d0c_.log 2024-08-20T22:05:58.6700224Z Running 25 items in this shard: test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_aten_contiguous, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_clone, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_contiguous, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_detach, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_dtensor_dtype_conversion, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_empty_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_equal, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_fill_inplace, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_fill_inplace_partial_sum, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_full_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_gather, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_index, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_inplace_op, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_new_empty_strided, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_new_full, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_ones_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_ones_like_partial_sum, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_op_out_variant, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_scatter, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_slice, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_stack, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_where_type_promotion, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zero_inplace, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zeros_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zeros_like_partial_sum 2024-08-20T22:05:58.6713315Z 2024-08-20T22:05:58.6714073Z Running distributed/_composable/fsdp/test_fully_shard_overlap 1/1 ... [2024-08-20 22:05:58.668651] 2024-08-20T22:05:58.6714996Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:05:58.6717318Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_overlap.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:05:58.669091] 2024-08-20T22:06:13.3609791Z 2024-08-20T22:06:13.3611987Z distributed/_composable/fsdp/test_fully_shard_overlap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_overlap_1.1_1eea5ef98c0da6ab_.log 2024-08-20T22:06:13.3615625Z Running 2 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_overlap.py::TestFullyShardOverlap::test_fully_shard_post_optim_event_overlap, test/distributed/_composable/fsdp/test_fully_shard_overlap.py::TestFullyShardOverlap::test_fully_shard_training_overlap 2024-08-20T22:06:13.3617648Z 2024-08-20T22:06:13.3618362Z Running distributed/_tensor/experimental/test_tp_transform 1/1 ... [2024-08-20 22:06:13.361273] 2024-08-20T22:06:13.3619267Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:06:13.3621843Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/experimental/test_tp_transform.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:06:13.361782] 2024-08-20T22:06:33.1651939Z 2024-08-20T22:06:33.1654046Z distributed/_tensor/experimental/test_tp_transform 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.experimental.test_tp_transform_1.1_a5f86a89759679b3_.log 2024-08-20T22:06:33.1658057Z Running 3 items in this shard: test/distributed/_tensor/experimental/test_tp_transform.py::TensorParallelTest::test_tp_transform_e2e, test/distributed/_tensor/experimental/test_tp_transform.py::TensorParallelTest::test_tp_transform_no_bias, test/distributed/_tensor/experimental/test_tp_transform.py::TensorParallelTest::test_tp_transform_with_uncovered_op 2024-08-20T22:06:33.1660732Z 2024-08-20T22:06:33.1661340Z Running distributed/test_compute_comm_reordering 1/1 ... [2024-08-20 22:06:33.165458] 2024-08-20T22:06:33.1662173Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:06:33.1664411Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_compute_comm_reordering.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:06:33.165902] 2024-08-20T22:06:41.0985902Z 2024-08-20T22:06:41.0989072Z distributed/test_compute_comm_reordering 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_compute_comm_reordering_1.1_da4d082e22eae4b6_.log 2024-08-20T22:06:41.1001359Z Running 7 items in this shard: test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_grouped_scheduler_node, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_nccl_heuristics, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_raise_comms, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap_custom_runtime_estimation, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits_raise_comms 2024-08-20T22:06:41.1010356Z 2024-08-20T22:06:41.1011027Z Running distributed/checkpoint/e2e/test_fsdp_ep 1/1 ... [2024-08-20 22:06:41.099195] 2024-08-20T22:06:41.1011869Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:06:41.1014107Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/e2e/test_fsdp_ep.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:06:41.099851] 2024-08-20T22:06:48.6800674Z 2024-08-20T22:06:48.6802878Z distributed/checkpoint/e2e/test_fsdp_ep 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.e2e.test_fsdp_ep_1.1_524f19244548fbc6_.log 2024-08-20T22:06:48.6804943Z Running 1 items in this shard: test/distributed/checkpoint/e2e/test_fsdp_ep.py::TestFSDPWithEP::test_e2e 2024-08-20T22:06:48.6806097Z 2024-08-20T22:06:48.6807233Z Running distributed/_tools/test_mem_tracker 1/1 ... [2024-08-20 22:06:48.680354] 2024-08-20T22:06:48.6808068Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:06:48.6812800Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_mem_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:06:48.680819] 2024-08-20T22:06:54.1553827Z 2024-08-20T22:06:54.1555453Z distributed/_tools/test_mem_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_mem_tracker_1.1_0bd7665b71726769_.log 2024-08-20T22:06:54.1558976Z Running 3 items in this shard: test/distributed/_tools/test_mem_tracker.py::TestMemTracker::test_cuda_tracker_equivalence, test/distributed/_tools/test_mem_tracker.py::TestMemTracker::test_tracker_attribution, test/distributed/_tools/test_mem_tracker.py::TestMemTracker::test_tracker_with_activation_checkpointing 2024-08-20T22:06:54.1561068Z 2024-08-20T22:06:54.1561763Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2024-08-20 22:06:54.155679] 2024-08-20T22:06:54.1562943Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:06:54.1565651Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:06:54.156116] 2024-08-20T22:10:08.5233582Z 2024-08-20T22:10:08.5235778Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_ccf9ca308586f1ef_.log 2024-08-20T22:10:08.5308746Z Running 20 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_training.py::test_compiled_fsdp, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardForwardInputs::test_root_move_forward_input_to_device, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardRegisteredParams::test_param_registration_after_backward, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardRegisteredParams::test_param_registration_after_forward, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardCastAfterInit::test_to_float64_after_init, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_explicit_prefetching, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_multi_forward_module, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_non_root_forward_backward, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_post_optim_event, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_multi_group, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_multi_group_cpu_offload_eager, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_single_group, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCompose::test_train_parity_with_activation_checkpointing, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardSharedParams::test_train_parity_with_shared_params, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardGradientAccumulation::test_1f1b_microbatching, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardGradientAccumulation::test_gradient_accumulation, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardNDTraining::test_2d_mlp_with_nd_mesh, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardHSDP3DTraining::test_3d_mlp_with_nd_mesh, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardHSDPTraining::test_train_parity_hsdp, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardCustomForwardMethod::test_register_fsdp_forward_method 2024-08-20T22:10:08.5657059Z 2024-08-20T22:10:09.0793691Z Running test batch 'tests to run' cost 683.72 seconds 2024-08-20T22:10:09.5090870Z 2024-08-20T22:10:09.5091538Z real 11m29.008s 2024-08-20T22:10:09.5091987Z user 23m26.785s 2024-08-20T22:10:09.5095130Z sys 8m43.693s 2024-08-20T22:10:09.5095552Z + assert_git_not_dirty 2024-08-20T22:10:09.5096855Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-08-20T22:10:09.5097990Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *xla* ]] 2024-08-20T22:10:09.5098884Z ++ git status --porcelain 2024-08-20T22:10:09.5099386Z ++ grep -v '?? third_party' 2024-08-20T22:10:12.4742279Z ++ true 2024-08-20T22:10:12.4743459Z + git_status= 2024-08-20T22:10:12.4744493Z + [[ -n '' ]] 2024-08-20T22:10:12.4745257Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-08-20T22:10:12.4746485Z + [[ 2 == 1 ]] 2024-08-20T22:10:12.4747069Z + [[ 2 == 1 ]] 2024-08-20T22:10:12.4747419Z + cleanup_workspace 2024-08-20T22:10:12.4748474Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-20T22:10:12.4749889Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-20T22:10:12.4751058Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-20T22:10:12.4751858Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-20T22:10:12.4752898Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-20T22:10:12.4754074Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-20T22:10:12.4754950Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-20T22:10:13.0757938Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-20T22:10:13.0758612Z cat test/**/*_toprint.log || true 2024-08-20T22:10:13.0770096Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:13.0770708Z env: 2024-08-20T22:10:13.0771045Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:13.0771609Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:13.0772540Z DOCKER_CONTAINER_ID: bf521d92eaa63669a33ef60d81f6abab162029657da975c0da2d1de2c74fc019 2024-08-20T22:10:13.0773355Z ##[endgroup] 2024-08-20T22:10:13.0864760Z cat: 'test/**/*_toprint.log': No such file or directory 2024-08-20T22:10:13.0906432Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-20T22:10:13.0907046Z kill "$MONITOR_SCRIPT_PID" 2024-08-20T22:10:13.0913773Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:13.0914399Z env: 2024-08-20T22:10:13.0914740Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:13.0915283Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:13.0916218Z DOCKER_CONTAINER_ID: bf521d92eaa63669a33ef60d81f6abab162029657da975c0da2d1de2c74fc019 2024-08-20T22:10:13.0917065Z MONITOR_SCRIPT_PID: 116430 2024-08-20T22:10:13.0917488Z ##[endgroup] 2024-08-20T22:10:13.1159929Z Prepare all required actions 2024-08-20T22:10:13.1160521Z Getting action download info 2024-08-20T22:10:13.3293799Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-20T22:10:13.5300808Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-20T22:10:13.5301380Z with: 2024-08-20T22:10:13.5302042Z file-suffix: test-distributed-2-3-amz2023.linux.8xlarge.nvidia.gpu_29025850735 2024-08-20T22:10:13.5302877Z s3-bucket: gha-artifacts 2024-08-20T22:10:13.5303277Z env: 2024-08-20T22:10:13.5303605Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:13.5304162Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:13.5305099Z DOCKER_CONTAINER_ID: bf521d92eaa63669a33ef60d81f6abab162029657da975c0da2d1de2c74fc019 2024-08-20T22:10:13.5305905Z ##[endgroup] 2024-08-20T22:10:13.5341168Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-20T22:10:13.5341981Z # Remove any previous test jsons if they exist 2024-08-20T22:10:13.5342604Z rm -f test-jsons-*.zip 2024-08-20T22:10:13.5343253Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-20T22:10:13.5350156Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:13.5350757Z env: 2024-08-20T22:10:13.5351092Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:13.5351654Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:13.5352572Z DOCKER_CONTAINER_ID: bf521d92eaa63669a33ef60d81f6abab162029657da975c0da2d1de2c74fc019 2024-08-20T22:10:13.5353716Z FILE_SUFFIX: test-distributed-2-3-amz2023.linux.8xlarge.nvidia.gpu_29025850735 2024-08-20T22:10:13.5354588Z ##[endgroup] 2024-08-20T22:10:13.5610481Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-20T22:10:13.5648682Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-20T22:10:13.5649590Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-20T22:10:13.5657133Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-20T22:10:13.5663849Z adding: test/slow_tests.json (deflated 82%) 2024-08-20T22:10:13.5666670Z adding: test/test-reports/td_exclusions-a16b71a6dad9baba93bd.json (deflated 86%) 2024-08-20T22:10:13.5681742Z adding: test/.pytorch-disabled-tests.json (deflated 88%) 2024-08-20T22:10:13.5717341Z ##[group]Run # Remove any previous test reports if they exist 2024-08-20T22:10:13.5718133Z # Remove any previous test reports if they exist 2024-08-20T22:10:13.5718802Z rm -f test-reports-*.zip 2024-08-20T22:10:13.5719536Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-20T22:10:13.5726522Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:13.5727149Z env: 2024-08-20T22:10:13.5727484Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:13.5728044Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:13.5728977Z DOCKER_CONTAINER_ID: bf521d92eaa63669a33ef60d81f6abab162029657da975c0da2d1de2c74fc019 2024-08-20T22:10:13.5730134Z FILE_SUFFIX: test-distributed-2-3-amz2023.linux.8xlarge.nvidia.gpu_29025850735 2024-08-20T22:10:13.5730949Z ##[endgroup] 2024-08-20T22:10:13.5955613Z adding: test/test-reports/python-pytest/distributed.test_dynamo_distributed/distributed.test_dynamo_distributed-076c2491acd935a5.xml (deflated 91%) 2024-08-20T22:10:13.5957933Z adding: test/test-reports/python-pytest/distributed.pipelining.test_transformer/distributed.pipelining.test_transformer-067b0f27939c927f.xml (deflated 40%) 2024-08-20T22:10:13.5960179Z adding: test/test-reports/python-pytest/distributed._tools.test_mod_tracker/distributed._tools.test_mod_tracker-cfc667b003913c3b.xml (deflated 65%) 2024-08-20T22:10:13.5962386Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_nested_dict/distributed.checkpoint.test_nested_dict-e92a9602ac67fb98.xml (deflated 51%) 2024-08-20T22:10:13.5964723Z adding: test/test-reports/python-pytest/distributed.test_fake_pg/distributed.test_fake_pg-d503aabab5f42460.xml (deflated 78%) 2024-08-20T22:10:13.5966845Z adding: test/test-reports/python-pytest/distributed.pipelining.test_backward/distributed.pipelining.test_backward-de7e8e6fe82b0f67.xml (deflated 37%) 2024-08-20T22:10:13.5969716Z adding: test/test-reports/python-pytest/distributed.fsdp.test_checkpoint_wrapper/distributed.fsdp.test_checkpoint_wrapper-27972ce035fe17f5.xml (deflated 74%) 2024-08-20T22:10:13.5972181Z adding: test/test-reports/python-pytest/distributed.test_symmetric_memory/distributed.test_symmetric_memory-4445ef262f6e2ba6.xml (deflated 93%) 2024-08-20T22:10:13.5974475Z adding: test/test-reports/python-pytest/distributed.flight_recorder.test_fr_analysis/distributed.flight_recorder.test_fr_analysis-ac45be59aaa5741e.xml (deflated 36%) 2024-08-20T22:10:13.5976887Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.test_logger/distributed._shard.sharded_tensor.test_logger-dec7199cfae2a286.xml (deflated 37%) 2024-08-20T22:10:13.5979119Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor/distributed._tensor.test_dtensor-7fbadb6042319c47.xml (deflated 85%) 2024-08-20T22:10:13.5981226Z adding: test/test-reports/python-pytest/distributed._tensor.test_random_ops/distributed._tensor.test_random_ops-56bd873fa8b55f76.xml (deflated 72%) 2024-08-20T22:10:13.5983414Z adding: test/test-reports/python-pytest/distributed._tensor.test_experimental_ops/distributed._tensor.test_experimental_ops-46619facb343596c.xml (deflated 56%) 2024-08-20T22:10:13.5985546Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_fx/distributed.fsdp.test_fsdp_fx-6a525bb274677f1a.xml (deflated 37%) 2024-08-20T22:10:13.5987604Z adding: test/test-reports/python-pytest/distributed._tensor.test_tensor_ops/distributed._tensor.test_tensor_ops-a275f10a4692222f.xml (deflated 83%) 2024-08-20T22:10:13.5989998Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_overlap/distributed._composable.fsdp.test_fully_shard_overlap-97f596a4125c7f3f.xml (deflated 52%) 2024-08-20T22:10:13.5992595Z adding: test/test-reports/python-pytest/distributed._tensor.experimental.test_tp_transform/distributed._tensor.experimental.test_tp_transform-add820c65effb157.xml (deflated 59%) 2024-08-20T22:10:13.5995029Z adding: test/test-reports/python-pytest/distributed.test_compute_comm_reordering/distributed.test_compute_comm_reordering-02c97d66e3bdee7f.xml (deflated 83%) 2024-08-20T22:10:13.5997332Z adding: test/test-reports/python-pytest/distributed.checkpoint.e2e.test_fsdp_ep/distributed.checkpoint.e2e.test_fsdp_ep-70408dc591a73029.xml (deflated 50%) 2024-08-20T22:10:13.5999569Z adding: test/test-reports/python-pytest/distributed._tools.test_mem_tracker/distributed._tools.test_mem_tracker-4bb262bfc4b4b2a2.xml (deflated 58%) 2024-08-20T22:10:13.6001977Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_training/distributed._composable.fsdp.test_fully_shard_training-018393b83f34df10.xml (deflated 81%) 2024-08-20T22:10:13.6035496Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-20T22:10:13.6036277Z # Remove any previous usage logs if they exist 2024-08-20T22:10:13.6036905Z rm -f logs-*.zip 2024-08-20T22:10:13.6037738Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-20T22:10:13.6038700Z # so check to see if the file exists first 2024-08-20T22:10:13.6039318Z if [ -f 'usage_log.txt' ]; then 2024-08-20T22:10:13.6039968Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-20T22:10:13.6040570Z fi 2024-08-20T22:10:13.6040991Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-20T22:10:13.6041700Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-20T22:10:13.6042306Z fi 2024-08-20T22:10:13.6048669Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:13.6049287Z env: 2024-08-20T22:10:13.6049784Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:13.6050352Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:13.6051285Z DOCKER_CONTAINER_ID: bf521d92eaa63669a33ef60d81f6abab162029657da975c0da2d1de2c74fc019 2024-08-20T22:10:13.6052560Z FILE_SUFFIX: test-distributed-2-3-amz2023.linux.8xlarge.nvidia.gpu_29025850735 2024-08-20T22:10:13.6053344Z ##[endgroup] 2024-08-20T22:10:13.6110957Z adding: usage_log.txt (deflated 92%) 2024-08-20T22:10:13.6345182Z adding: test/test-reports/distributed.test_dynamo_distributed_1.1_1cf361ef7cc99d01_.log (deflated 85%) 2024-08-20T22:10:13.6346769Z adding: test/test-reports/distributed.pipelining.test_transformer_1.1_ca12b2c9456d111c_.log (deflated 54%) 2024-08-20T22:10:13.6348295Z adding: test/test-reports/distributed._tools.test_mod_tracker_1.1_fe29e931c78a3104_.log (deflated 64%) 2024-08-20T22:10:13.6349825Z adding: test/test-reports/distributed.checkpoint.test_nested_dict_1.1_dc19e716f04a7ca4_.log (deflated 59%) 2024-08-20T22:10:13.6351265Z adding: test/test-reports/distributed.test_fake_pg_1.1_469e2e41162ce46a_.log (deflated 73%) 2024-08-20T22:10:13.6352709Z adding: test/test-reports/distributed.pipelining.test_backward_1.1_45bbedc9b1ada6f5_.log (deflated 54%) 2024-08-20T22:10:13.6354323Z adding: test/test-reports/distributed.fsdp.test_checkpoint_wrapper_1.1_85265c3d83008866_.log (deflated 71%) 2024-08-20T22:10:13.6355803Z adding: test/test-reports/distributed.test_symmetric_memory_1.1_7f26551526830141_.log (deflated 86%) 2024-08-20T22:10:13.6357338Z adding: test/test-reports/distributed.flight_recorder.test_fr_analysis_1.1_bd2167f53d19fde1_.log (deflated 55%) 2024-08-20T22:10:13.6358948Z adding: test/test-reports/distributed._shard.sharded_tensor.test_logger_1.1_4b3310b7010c2aa7_.log (deflated 55%) 2024-08-20T22:10:13.6360473Z adding: test/test-reports/distributed._tensor.test_dtensor_1.1_008bab50afdf08c7_.log (deflated 88%) 2024-08-20T22:10:13.6361921Z adding: test/test-reports/distributed._tensor.test_random_ops_1.1_f692d262105ee15d_.log (deflated 69%) 2024-08-20T22:10:13.6363441Z adding: test/test-reports/distributed._tensor.test_experimental_ops_1.1_03f6d0fd47302058_.log (deflated 62%) 2024-08-20T22:10:13.6364927Z adding: test/test-reports/distributed.fsdp.test_fsdp_fx_1.1_d13d1eb22ea37849_.log (deflated 53%) 2024-08-20T22:10:13.6366377Z adding: test/test-reports/distributed._tensor.test_tensor_ops_1.1_c0f2a6f34f3f2d0c_.log (deflated 78%) 2024-08-20T22:10:13.6367982Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_overlap_1.1_1eea5ef98c0da6ab_.log (deflated 62%) 2024-08-20T22:10:13.6370131Z adding: test/test-reports/distributed._tensor.experimental.test_tp_transform_1.1_a5f86a89759679b3_.log (deflated 64%) 2024-08-20T22:10:13.6371757Z adding: test/test-reports/distributed.test_compute_comm_reordering_1.1_da4d082e22eae4b6_.log (deflated 74%) 2024-08-20T22:10:13.6373329Z adding: test/test-reports/distributed.checkpoint.e2e.test_fsdp_ep_1.1_524f19244548fbc6_.log (deflated 61%) 2024-08-20T22:10:13.6374833Z adding: test/test-reports/distributed._tools.test_mem_tracker_1.1_0bd7665b71726769_.log (deflated 62%) 2024-08-20T22:10:13.6708506Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_ccf9ca308586f1ef_.log (deflated 96%) 2024-08-20T22:10:13.6743536Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-20T22:10:13.6744416Z # Remove any previous debugging artifacts if they exist 2024-08-20T22:10:13.6745097Z rm -f debug-*.zip 2024-08-20T22:10:13.6745561Z if [ -d 'test/debug' ]; then 2024-08-20T22:10:13.6746163Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-20T22:10:13.6746746Z fi 2024-08-20T22:10:13.6753204Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:13.6753829Z env: 2024-08-20T22:10:13.6754202Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:13.6754762Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:13.6755864Z DOCKER_CONTAINER_ID: bf521d92eaa63669a33ef60d81f6abab162029657da975c0da2d1de2c74fc019 2024-08-20T22:10:13.6757023Z FILE_SUFFIX: test-distributed-2-3-amz2023.linux.8xlarge.nvidia.gpu_29025850735 2024-08-20T22:10:13.6758399Z ##[endgroup] 2024-08-20T22:10:13.6863228Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:10:13.6863762Z with: 2024-08-20T22:10:13.6864104Z s3-bucket: gha-artifacts 2024-08-20T22:10:13.6864641Z s3-prefix: pytorch/pytorch/10479309237/1/artifact 2024-08-20T22:10:13.6865209Z retention-days: 14 2024-08-20T22:10:13.6865616Z if-no-files-found: warn 2024-08-20T22:10:13.6866051Z path: test-jsons-*.zip 2024-08-20T22:10:13.6866448Z name: artifact 2024-08-20T22:10:13.6866813Z region: us-east-1 2024-08-20T22:10:13.6867175Z env: 2024-08-20T22:10:13.6867483Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:13.6868034Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:13.6869386Z DOCKER_CONTAINER_ID: bf521d92eaa63669a33ef60d81f6abab162029657da975c0da2d1de2c74fc019 2024-08-20T22:10:13.6870220Z ##[endgroup] 2024-08-20T22:10:14.1314278Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T22:10:14.1315168Z With the provided path, there will be 1 file uploaded 2024-08-20T22:10:14.1315967Z Uploading to s3 prefix: pytorch/pytorch/10479309237/1/artifact 2024-08-20T22:10:14.1368667Z Starting upload of test-jsons-test-distributed-2-3-amz2023.linux.8xlarge.nvidia.gpu_29025850735.zip 2024-08-20T22:10:14.2804849Z Finished upload of test-jsons-test-distributed-2-3-amz2023.linux.8xlarge.nvidia.gpu_29025850735.zip 2024-08-20T22:10:14.3002626Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:10:14.3003175Z with: 2024-08-20T22:10:14.3003496Z s3-bucket: gha-artifacts 2024-08-20T22:10:14.3004030Z s3-prefix: pytorch/pytorch/10479309237/1/artifact 2024-08-20T22:10:14.3004613Z retention-days: 14 2024-08-20T22:10:14.3005005Z if-no-files-found: error 2024-08-20T22:10:14.3005447Z path: test-reports-*.zip 2024-08-20T22:10:14.3005890Z name: artifact 2024-08-20T22:10:14.3006238Z region: us-east-1 2024-08-20T22:10:14.3006600Z env: 2024-08-20T22:10:14.3006907Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:14.3007459Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:14.3008406Z DOCKER_CONTAINER_ID: bf521d92eaa63669a33ef60d81f6abab162029657da975c0da2d1de2c74fc019 2024-08-20T22:10:14.3009217Z ##[endgroup] 2024-08-20T22:10:14.7487824Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T22:10:14.7489362Z With the provided path, there will be 1 file uploaded 2024-08-20T22:10:14.7490129Z Uploading to s3 prefix: pytorch/pytorch/10479309237/1/artifact 2024-08-20T22:10:14.7542156Z Starting upload of test-reports-test-distributed-2-3-amz2023.linux.8xlarge.nvidia.gpu_29025850735.zip 2024-08-20T22:10:14.9143396Z Finished upload of test-reports-test-distributed-2-3-amz2023.linux.8xlarge.nvidia.gpu_29025850735.zip 2024-08-20T22:10:14.9338479Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:10:14.9339018Z with: 2024-08-20T22:10:14.9339377Z s3-bucket: gha-artifacts 2024-08-20T22:10:14.9339917Z s3-prefix: pytorch/pytorch/10479309237/1/artifact 2024-08-20T22:10:14.9340501Z retention-days: 14 2024-08-20T22:10:14.9340909Z if-no-files-found: ignore 2024-08-20T22:10:14.9341352Z path: logs-*.zip 2024-08-20T22:10:14.9341726Z name: artifact 2024-08-20T22:10:14.9342079Z region: us-east-1 2024-08-20T22:10:14.9342443Z env: 2024-08-20T22:10:14.9342757Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:14.9343316Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:14.9344255Z DOCKER_CONTAINER_ID: bf521d92eaa63669a33ef60d81f6abab162029657da975c0da2d1de2c74fc019 2024-08-20T22:10:14.9345083Z ##[endgroup] 2024-08-20T22:10:15.3755788Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T22:10:15.3756770Z With the provided path, there will be 1 file uploaded 2024-08-20T22:10:15.3757547Z Uploading to s3 prefix: pytorch/pytorch/10479309237/1/artifact 2024-08-20T22:10:15.3809453Z Starting upload of logs-test-distributed-2-3-amz2023.linux.8xlarge.nvidia.gpu_29025850735.zip 2024-08-20T22:10:15.5727634Z Finished upload of logs-test-distributed-2-3-amz2023.linux.8xlarge.nvidia.gpu_29025850735.zip 2024-08-20T22:10:15.5920765Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:10:15.5921299Z with: 2024-08-20T22:10:15.5921644Z s3-bucket: gha-artifacts 2024-08-20T22:10:15.5922185Z s3-prefix: pytorch/pytorch/10479309237/1/artifact 2024-08-20T22:10:15.5922750Z retention-days: 14 2024-08-20T22:10:15.5923161Z if-no-files-found: ignore 2024-08-20T22:10:15.5923604Z path: debug-*.zip 2024-08-20T22:10:15.5923965Z name: artifact 2024-08-20T22:10:15.5924337Z region: us-east-1 2024-08-20T22:10:15.5924702Z env: 2024-08-20T22:10:15.5925017Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:15.5925576Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:15.5926513Z DOCKER_CONTAINER_ID: bf521d92eaa63669a33ef60d81f6abab162029657da975c0da2d1de2c74fc019 2024-08-20T22:10:15.5927329Z ##[endgroup] 2024-08-20T22:10:16.0287380Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-20T22:10:16.0478340Z ##[group]Run # shellcheck disable=SC2156 2024-08-20T22:10:16.0478972Z # shellcheck disable=SC2156 2024-08-20T22:10:16.0479965Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-20T22:10:16.0487551Z shell: /usr/bin/bash -e {0} 2024-08-20T22:10:16.0487985Z env: 2024-08-20T22:10:16.0488314Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:16.0488861Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:16.0489799Z DOCKER_CONTAINER_ID: bf521d92eaa63669a33ef60d81f6abab162029657da975c0da2d1de2c74fc019 2024-08-20T22:10:16.0490614Z ##[endgroup] 2024-08-20T22:10:16.3309361Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-20T22:10:16.3310060Z with: 2024-08-20T22:10:16.3310366Z env: 2024-08-20T22:10:16.3310695Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:16.3311235Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:16.3312169Z DOCKER_CONTAINER_ID: bf521d92eaa63669a33ef60d81f6abab162029657da975c0da2d1de2c74fc019 2024-08-20T22:10:16.3313008Z ##[endgroup] 2024-08-20T22:10:16.3338628Z ##[group]Run set -eou pipefail 2024-08-20T22:10:16.3339124Z set -eou pipefail 2024-08-20T22:10:16.3339546Z  2024-08-20T22:10:16.3340169Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-20T22:10:16.3340947Z for _ in $(seq 1440); do 2024-08-20T22:10:16.3341525Z  # Break if no ssh session exists anymore 2024-08-20T22:10:16.3342135Z  if [ "$(who)" = "" ]; then 2024-08-20T22:10:16.3342615Z  break 2024-08-20T22:10:16.3343046Z  fi 2024-08-20T22:10:16.3343390Z  echo "." 2024-08-20T22:10:16.3343785Z  sleep 5 2024-08-20T22:10:16.3344162Z done 2024-08-20T22:10:16.3350789Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:16.3351401Z env: 2024-08-20T22:10:16.3351733Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:16.3352267Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:16.3353203Z DOCKER_CONTAINER_ID: bf521d92eaa63669a33ef60d81f6abab162029657da975c0da2d1de2c74fc019 2024-08-20T22:10:16.3354014Z ##[endgroup] 2024-08-20T22:10:16.3382761Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-20T22:10:16.3441974Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:10:16.3442940Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:10:16.3443679Z # shellcheck disable=SC2046 2024-08-20T22:10:16.3444256Z docker stop $(docker ps -q) || true 2024-08-20T22:10:16.3444841Z # Prune all of the docker images 2024-08-20T22:10:16.3445398Z docker system prune -af 2024-08-20T22:10:16.3452004Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:16.3452629Z env: 2024-08-20T22:10:16.3452964Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:16.3453510Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:16.3454715Z DOCKER_CONTAINER_ID: bf521d92eaa63669a33ef60d81f6abab162029657da975c0da2d1de2c74fc019 2024-08-20T22:10:16.3455535Z ##[endgroup] 2024-08-20T22:10:17.0418235Z bf521d92eaa6 2024-08-20T22:10:17.5413582Z Deleted Containers: 2024-08-20T22:10:17.5414309Z bf521d92eaa63669a33ef60d81f6abab162029657da975c0da2d1de2c74fc019 2024-08-20T22:10:17.5414909Z 2024-08-20T22:10:23.6078422Z Deleted Images: 2024-08-20T22:10:23.6080219Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:10:23.6082796Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9@sha256:0b3aa0836d4f18656c98dcf355b46cbae2123243424ebbed0929206c3d798126 2024-08-20T22:10:23.6084583Z deleted: sha256:35f4945dfeab4592c8657de7107f96a9fd9a51999f582a0064937246789c0825 2024-08-20T22:10:23.6085621Z deleted: sha256:5aeec7923eaf143c8c860346742d9e9c0a627a72c06c4a8574f03d7359497d8a 2024-08-20T22:10:23.6086690Z deleted: sha256:8c2e3c359e6f8f863100522d9503272ad2a22af4d3245cb32988cf7f75dff9bf 2024-08-20T22:10:23.6087736Z deleted: sha256:b9c8bef934615ea09715897fa2252d4ff9a33ae4b0b99d65f4314edb98a08508 2024-08-20T22:10:23.6088754Z deleted: sha256:7325e907409be83e224427131a7e02928ab8ca8880e53b975b309a17c9887b3d 2024-08-20T22:10:23.6089794Z deleted: sha256:93c3c0295ffeb06e6b45fd31a03c7b9de49d6a6404a889643dcb02a29956a46d 2024-08-20T22:10:23.6090890Z deleted: sha256:75434bb2c96c820ddef9e3cd15729787375a35257f82c07e9a2e49a63ccd258b 2024-08-20T22:10:23.6092246Z deleted: sha256:2b55c8d634372339b274a7068c1e8058ff9ca9b828c020d97462adddfbbab68e 2024-08-20T22:10:23.6093298Z deleted: sha256:f4242713ef5734c93522c188d00d66c11c2ea30e1edc576f7557feec4a71fb44 2024-08-20T22:10:23.6094331Z deleted: sha256:e3f0028f9a27ad44915c7ce616833089a5ac41163b1f3fd441c29448c9967bd6 2024-08-20T22:10:23.6095362Z deleted: sha256:157d6992f3a76ca9422961a443176f6c953d6d2609d8f26abc95e283177d8838 2024-08-20T22:10:23.6096404Z deleted: sha256:80725860ab404ed4a3e2d4f1c9179efb9628ddaaead438f0e2dfd45860821659 2024-08-20T22:10:23.6097455Z deleted: sha256:12612899d16f7cab2f6bcce2c8c2ce9a6aabe0f8c57d6a9ec3df95aebc24eb66 2024-08-20T22:10:23.6098520Z deleted: sha256:f892c2afba839178fe4f4480ffa99124c5788c76959cef3c751d31ffbc56532e 2024-08-20T22:10:23.6099565Z deleted: sha256:3783df37fa2f17df24597d0c9b48925712f7ec5cdb75993e04a750249242d945 2024-08-20T22:10:23.6100586Z deleted: sha256:03700104a8d899077ca0cf2dcca4902d103f736f5605fc2dcb1228b4fbb01cf8 2024-08-20T22:10:23.6101900Z deleted: sha256:c22e6b22d7d1206421c03a90e2a0d202a5b0293700ea39051d3062ab4e02212f 2024-08-20T22:10:23.6102949Z deleted: sha256:6a5fd513d78286d2a97e6118bdfaf09d345513de1afe00f7214930aab4e182ce 2024-08-20T22:10:23.6103981Z deleted: sha256:12d6046ca585372ab7221c75bf1ff9fbad344c584bfac3bf48490124569b09ce 2024-08-20T22:10:23.6105038Z deleted: sha256:11a342b5bdf2902b76dc6b741fa09aeada64dc62a4634c30e0b92dd49ecaa3e6 2024-08-20T22:10:23.6106094Z deleted: sha256:9c19056af41bd6f7d0316312696e9fb6ec9e16b9b2916bebb487ed1302846fca 2024-08-20T22:10:23.6107155Z deleted: sha256:dff9ad772a710e9d7a74de2fb2e60c8d87630855c5a75c7348b20d817de188c9 2024-08-20T22:10:23.6108181Z deleted: sha256:21911a7fa8e0ff3a30aa732134f3d24cb0e339bc36725f9980a7687e520e356a 2024-08-20T22:10:23.6109228Z deleted: sha256:f40a21fc1dad1aa6a566142c92a08afe87864068e6c360b6ade83ec521803286 2024-08-20T22:10:23.6110264Z deleted: sha256:617a6a309e07100390ac08e5ea2680a47e1f79fa72907e8b08ff46309a46f1ba 2024-08-20T22:10:23.6111287Z deleted: sha256:8d26fabf19fef49bc3a331f43fdaa661678b87d53109fc7540bfb9b44ce810bf 2024-08-20T22:10:23.6112464Z deleted: sha256:a2d242ed225d8f09315e3ee4597c3feb02f088337f59c95ddf440784d2be4f52 2024-08-20T22:10:23.6113903Z deleted: sha256:e10875c7907612ac7bd831be770f133f6eccaa347118b765c5d9ec776512f0b2 2024-08-20T22:10:23.6115023Z deleted: sha256:b5fa471f84592d75a2cf8625dcf9a285b9d76a7e7edccff81d385b66a46f9532 2024-08-20T22:10:23.6116106Z deleted: sha256:e42f988c8d3d938d4a1e3975407f64e4f962f3cabc1122a8fe98828492e34657 2024-08-20T22:10:23.6117385Z deleted: sha256:9507391f7f3611eb7c6f95061219d63ee3ecfd4cc81b3de44126c6d3c4667551 2024-08-20T22:10:23.6118416Z deleted: sha256:5e72605653a900048cf413ca242bb695bc9932595763521ed44f8fbe7166166a 2024-08-20T22:10:23.6119417Z deleted: sha256:435005331c11e4210ce1b81221a7054aee7ccaeb202bb2e09b45f2510f6ab254 2024-08-20T22:10:23.6120454Z deleted: sha256:a2f04d898ab3e490e7247db9696a9726e1f1f85012b46115a89aedab38930710 2024-08-20T22:10:23.6121483Z deleted: sha256:07b864569ca9924353696ae1560e08d337c4867cb398ce0e552e04c37d4cf6e5 2024-08-20T22:10:23.6122498Z deleted: sha256:907fa71a16fcb35456269eb13d0128ce7763243eaf3449c44cdd36b5b49312e1 2024-08-20T22:10:23.6123565Z deleted: sha256:f79790dfbd9f1c7e3fc5d45ffd290c41176ec11dffaf7a8c63c2c58b797fcdb6 2024-08-20T22:10:23.6124621Z deleted: sha256:406d461e2aa914e38d6eef5723bb633966284960223c93a97cd00be8a1cc0faf 2024-08-20T22:10:23.6125645Z deleted: sha256:119a4673c9c39860d87ebbc4b72e29b12aa2f3242fabcf48cb24a5c8cfcd1704 2024-08-20T22:10:23.6126691Z deleted: sha256:624417752cf64b2297d09ab2e429cc9c87774134cc1b34bf2e5c2b79cd34ecc4 2024-08-20T22:10:23.6127748Z deleted: sha256:2bcfc72dea6a6bc48e2290c98fa34228cfe20302176ff01ce6b84971d811079a 2024-08-20T22:10:23.6128824Z deleted: sha256:d4aee3900a65d7e4f9a427d9bae22e836280b85f289fd5b80adc4ed2cd3aff53 2024-08-20T22:10:23.6129845Z deleted: sha256:2a255798af4e419932e17c11a82527d26470216c9bc57ab8807a735dbd2c93ae 2024-08-20T22:10:23.6130870Z deleted: sha256:fa19e724b62cc0bbe219419c371baf2471725f88a55292672c1aa24358db4a9b 2024-08-20T22:10:23.6132049Z deleted: sha256:9c5a64873ca39cb9da2e93ad3ec1eb31211c8c088c3a90be1541ec5ac891ac4e 2024-08-20T22:10:23.6133101Z deleted: sha256:8421b1b55b2087ef547bb0133e57f5859efb3c9429d011de45772b63c923dd90 2024-08-20T22:10:23.6134154Z deleted: sha256:66066e4af5674b8211099bfebb5a5ab99b3d9181fcfc1dc67b54a0eff5ded07a 2024-08-20T22:10:23.6135214Z deleted: sha256:dad58a2cd6e9c8234f343c2e5ca4cd2a317b729f8048049d34151b15c83b0dae 2024-08-20T22:10:23.6136253Z deleted: sha256:3bc136283d22fe55b48989019c8484e37c075a7a98fae0a5b7f827623cb2fd5b 2024-08-20T22:10:23.6137285Z deleted: sha256:2543cfb072205917a252ea669db6e7fe93ac6bf424262d6d67b6bf24c1edfe55 2024-08-20T22:10:23.6138328Z deleted: sha256:a0abc6809e0930ad53392ef3443966cb3f6bc7645d3e1968f2d18cb0648d1240 2024-08-20T22:10:23.6139369Z deleted: sha256:0068bc6311b72b957fcc8bec156923eef76c142d408a73e201ff797dcf126f5a 2024-08-20T22:10:23.6140407Z deleted: sha256:3bb2c93e3ca7da8a708900f677778ff0abe6bcb9666d9480bdad78acd6793a59 2024-08-20T22:10:23.6141507Z deleted: sha256:d3a2bdf80e0add5b55bffcb4cf9bd55c993adad4fb66a4ec8864e0d8a118bb0a 2024-08-20T22:10:23.6142578Z deleted: sha256:c021994940e99e271b90c5bf290f821f73a14768aa34bfae3755c04685c88251 2024-08-20T22:10:23.6143582Z deleted: sha256:688ff5904c74620c1b2bd34c392425856fe1f759f94bf0419d051997c4fb23ad 2024-08-20T22:10:23.6144623Z deleted: sha256:d397bdc2172f15f11a5f57d5f57064aee1b038d71c3ca9e967ebf47dcbd5dec0 2024-08-20T22:10:23.6145695Z deleted: sha256:a70ac2bc18340667dfc9895cf91a48cc3c2b5a6c2acdc898bae02810b111c7d7 2024-08-20T22:10:23.6146754Z deleted: sha256:0fbec293f1fa0d21f414351f21bd86a114facc2f3bc74715393f4f33c9b2c89e 2024-08-20T22:10:23.6147810Z deleted: sha256:4a26099a629bbf706fb951dcb1381af8361bf5f3e736e82ea839a4a27788d0b3 2024-08-20T22:10:23.6148854Z deleted: sha256:b6284de21252ed91fd7dea71b02196645a8e239d6188ecd9f080d9b4a2026b77 2024-08-20T22:10:23.6149900Z deleted: sha256:adca9f30c5b25b7904016386497a34e3699d12b61be1cabda8703a4f6f51adaf 2024-08-20T22:10:23.6150912Z deleted: sha256:b1007278b864200bed03e3738e6a554a7913b41e9246c86925126b02f19d2512 2024-08-20T22:10:23.6151960Z deleted: sha256:ac9c37335cbc79df52a9de95491fe9ffa89b9fe730c29153ae1d8c93a046bda9 2024-08-20T22:10:23.6153031Z deleted: sha256:d0e212b9d05b634be5d1abfbfcc4fd090277954332f2b8e519c695a979bfd0dd 2024-08-20T22:10:23.6154080Z deleted: sha256:abcbe99d7f9c9df7e3c594587238e23d28f96a5fc92c2a9dd1d183ff7af4800c 2024-08-20T22:10:23.6155196Z deleted: sha256:7740288981d55c53dc1d781eb5a229ccbd5c81c56584c4d4217d7fbb512de04b 2024-08-20T22:10:23.6156334Z deleted: sha256:020ed2c4861bb111c010888e7c4ca3b2c1125b9d7b8a29148de859f786ecea66 2024-08-20T22:10:23.6157346Z deleted: sha256:89edf8c906678c217b9a88160e1561991f53909c3fd27a4a5e1f0ca0f7ff7888 2024-08-20T22:10:23.6158405Z deleted: sha256:d5ba67f8f9b0c2b6da06d2d5d9d5fedc9f17cbe1a5e13a1a4033a4619f4528ba 2024-08-20T22:10:23.6159457Z deleted: sha256:5d61311e1a293c0c9642e34871719525dfeec7028e4b45e08344a469e3fe24bf 2024-08-20T22:10:23.6160495Z deleted: sha256:b89bb70f745a94c0edd5e974aee294e160735f76fd666a1a06a0b9b1be3caf63 2024-08-20T22:10:23.6161539Z deleted: sha256:9e5477f1f21a38c3dfe21a56a7d77d1b1a42453aed3e11729afcbb9d5731770b 2024-08-20T22:10:23.6162609Z deleted: sha256:b76f4af91cffe2bb7b3f09960f5bdbb26f741419defda65c451c4407fc5b0a92 2024-08-20T22:10:23.6163675Z deleted: sha256:3f8748a00ebd394c0e9d9d481972d764dff58896841ef0150b3f7efaac0f4a4a 2024-08-20T22:10:23.6164688Z deleted: sha256:13989c771e3ba4d40f618344e55e5cb7062a35041b5efd69e2f740ba819f46ee 2024-08-20T22:10:23.6165750Z deleted: sha256:9ebb27837f6c67ad8c17e9af4e63c008faae03fa226c3641dd8bbbfeefb106c4 2024-08-20T22:10:23.6166797Z deleted: sha256:8c16907d735950f8093582bc3cf392b70e002456bf9c4224de5a5b86b78b3ea2 2024-08-20T22:10:23.6167811Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-08-20T22:10:23.6168455Z 2024-08-20T22:10:23.6168638Z Total reclaimed space: 34.72GB 2024-08-20T22:10:23.6256602Z Post job cleanup. 2024-08-20T22:10:23.6460146Z Post job cleanup. 2024-08-20T22:10:23.7650202Z [command]/usr/bin/git version 2024-08-20T22:10:23.7709616Z git version 2.40.1 2024-08-20T22:10:23.7772299Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/233b6bd2-2bb8-430b-a0f9-cc09193e0c8e' before making global git config changes 2024-08-20T22:10:23.7773897Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T22:10:23.7779827Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:23.7825180Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T22:10:23.7866552Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T22:10:23.8214781Z Entering 'android/libs/fbjni' 2024-08-20T22:10:23.8277524Z Entering 'third_party/FP16' 2024-08-20T22:10:23.8340211Z Entering 'third_party/FXdiv' 2024-08-20T22:10:23.8404306Z Entering 'third_party/NNPACK' 2024-08-20T22:10:23.8466184Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:10:23.8530346Z Entering 'third_party/XNNPACK' 2024-08-20T22:10:23.8611891Z Entering 'third_party/benchmark' 2024-08-20T22:10:23.8675862Z Entering 'third_party/cpp-httplib' 2024-08-20T22:10:23.8736522Z Entering 'third_party/cpuinfo' 2024-08-20T22:10:23.8801126Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:10:23.8863504Z Entering 'third_party/cutlass' 2024-08-20T22:10:23.8935370Z Entering 'third_party/eigen' 2024-08-20T22:10:23.9001837Z Entering 'third_party/fbgemm' 2024-08-20T22:10:23.9064379Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:10:23.9124813Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:10:23.9186961Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:10:23.9255161Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:10:23.9317051Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:10:23.9381304Z Entering 'third_party/flatbuffers' 2024-08-20T22:10:23.9445491Z Entering 'third_party/fmt' 2024-08-20T22:10:23.9508639Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:10:23.9570519Z Entering 'third_party/gloo' 2024-08-20T22:10:23.9632813Z Entering 'third_party/googletest' 2024-08-20T22:10:23.9696913Z Entering 'third_party/ideep' 2024-08-20T22:10:23.9758320Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:10:23.9829806Z Entering 'third_party/ittapi' 2024-08-20T22:10:23.9893184Z Entering 'third_party/kineto' 2024-08-20T22:10:23.9954610Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:10:24.0016763Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:10:24.0081332Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:10:24.0144394Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:10:24.0208025Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:10:24.0268397Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:10:24.0333172Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:10:24.0395392Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:10:24.0458493Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:10:24.0522932Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:10:24.0587018Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:10:24.0648054Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:10:24.0711833Z Entering 'third_party/mimalloc' 2024-08-20T22:10:24.0774225Z Entering 'third_party/nccl/nccl' 2024-08-20T22:10:24.0836080Z Entering 'third_party/nlohmann' 2024-08-20T22:10:24.0900389Z Entering 'third_party/onnx' 2024-08-20T22:10:24.0981283Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:10:24.1042657Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:10:24.1108727Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:10:24.1174121Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:10:24.1236484Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:10:24.1298824Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:10:24.1359827Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:10:24.1423343Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:10:24.1485689Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:10:24.1546329Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:10:24.1607689Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:10:24.1673160Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:10:24.1736138Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:10:24.1820308Z Entering 'third_party/pocketfft' 2024-08-20T22:10:24.1882991Z Entering 'third_party/protobuf' 2024-08-20T22:10:24.1948227Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:10:24.2010784Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:10:24.2074907Z Entering 'third_party/psimd' 2024-08-20T22:10:24.2135960Z Entering 'third_party/pthreadpool' 2024-08-20T22:10:24.2197782Z Entering 'third_party/pybind11' 2024-08-20T22:10:24.2260713Z Entering 'third_party/python-peachpy' 2024-08-20T22:10:24.2323352Z Entering 'third_party/sleef' 2024-08-20T22:10:24.2385463Z Entering 'third_party/tensorpipe' 2024-08-20T22:10:24.2446947Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:10:24.2509086Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:10:24.2569788Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:10:24.2631266Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:10:24.2692456Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:10:24.2777534Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T22:10:24.2807778Z http.https://github.com/.extraheader 2024-08-20T22:10:24.2819764Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-20T22:10:24.2860651Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T22:10:24.3200401Z Entering 'android/libs/fbjni' 2024-08-20T22:10:24.3242250Z http.https://github.com/.extraheader 2024-08-20T22:10:24.3280666Z Entering 'third_party/FP16' 2024-08-20T22:10:24.3322434Z http.https://github.com/.extraheader 2024-08-20T22:10:24.3361707Z Entering 'third_party/FXdiv' 2024-08-20T22:10:24.3404056Z http.https://github.com/.extraheader 2024-08-20T22:10:24.3441341Z Entering 'third_party/NNPACK' 2024-08-20T22:10:24.3486108Z http.https://github.com/.extraheader 2024-08-20T22:10:24.3523718Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:10:24.3566149Z http.https://github.com/.extraheader 2024-08-20T22:10:24.3605526Z Entering 'third_party/XNNPACK' 2024-08-20T22:10:24.3647074Z http.https://github.com/.extraheader 2024-08-20T22:10:24.3704423Z Entering 'third_party/benchmark' 2024-08-20T22:10:24.3746108Z http.https://github.com/.extraheader 2024-08-20T22:10:24.3784179Z Entering 'third_party/cpp-httplib' 2024-08-20T22:10:24.3827034Z http.https://github.com/.extraheader 2024-08-20T22:10:24.3864680Z Entering 'third_party/cpuinfo' 2024-08-20T22:10:24.3907286Z http.https://github.com/.extraheader 2024-08-20T22:10:24.3946763Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:10:24.3990824Z http.https://github.com/.extraheader 2024-08-20T22:10:24.4028468Z Entering 'third_party/cutlass' 2024-08-20T22:10:24.4070459Z http.https://github.com/.extraheader 2024-08-20T22:10:24.4118572Z Entering 'third_party/eigen' 2024-08-20T22:10:24.4160563Z http.https://github.com/.extraheader 2024-08-20T22:10:24.4201256Z Entering 'third_party/fbgemm' 2024-08-20T22:10:24.4243480Z http.https://github.com/.extraheader 2024-08-20T22:10:24.4283145Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:10:24.4324699Z http.https://github.com/.extraheader 2024-08-20T22:10:24.4361848Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:10:24.4404555Z http.https://github.com/.extraheader 2024-08-20T22:10:24.4441964Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:10:24.4483685Z http.https://github.com/.extraheader 2024-08-20T22:10:24.4529345Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:10:24.4571740Z http.https://github.com/.extraheader 2024-08-20T22:10:24.4609066Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:10:24.4650015Z http.https://github.com/.extraheader 2024-08-20T22:10:24.4689469Z Entering 'third_party/flatbuffers' 2024-08-20T22:10:24.4730988Z http.https://github.com/.extraheader 2024-08-20T22:10:24.4771973Z Entering 'third_party/fmt' 2024-08-20T22:10:24.4813815Z http.https://github.com/.extraheader 2024-08-20T22:10:24.4851337Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:10:24.4893776Z http.https://github.com/.extraheader 2024-08-20T22:10:24.4931221Z Entering 'third_party/gloo' 2024-08-20T22:10:24.4973961Z http.https://github.com/.extraheader 2024-08-20T22:10:24.5011494Z Entering 'third_party/googletest' 2024-08-20T22:10:24.5053330Z http.https://github.com/.extraheader 2024-08-20T22:10:24.5091740Z Entering 'third_party/ideep' 2024-08-20T22:10:24.5133351Z http.https://github.com/.extraheader 2024-08-20T22:10:24.5169930Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:10:24.5211779Z http.https://github.com/.extraheader 2024-08-20T22:10:24.5256471Z Entering 'third_party/ittapi' 2024-08-20T22:10:24.5299244Z http.https://github.com/.extraheader 2024-08-20T22:10:24.5336372Z Entering 'third_party/kineto' 2024-08-20T22:10:24.5379883Z http.https://github.com/.extraheader 2024-08-20T22:10:24.5418411Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:10:24.5459942Z http.https://github.com/.extraheader 2024-08-20T22:10:24.5499294Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:10:24.5541844Z http.https://github.com/.extraheader 2024-08-20T22:10:24.5582486Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:10:24.5624170Z http.https://github.com/.extraheader 2024-08-20T22:10:24.5662704Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:10:24.5705040Z http.https://github.com/.extraheader 2024-08-20T22:10:24.5743934Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:10:24.5786136Z http.https://github.com/.extraheader 2024-08-20T22:10:24.5825061Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:10:24.5866891Z http.https://github.com/.extraheader 2024-08-20T22:10:24.5909577Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:10:24.5950796Z http.https://github.com/.extraheader 2024-08-20T22:10:24.5990027Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:10:24.6031157Z http.https://github.com/.extraheader 2024-08-20T22:10:24.6070223Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:10:24.6111553Z http.https://github.com/.extraheader 2024-08-20T22:10:24.6152697Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:10:24.6195506Z http.https://github.com/.extraheader 2024-08-20T22:10:24.6235119Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:10:24.6277484Z http.https://github.com/.extraheader 2024-08-20T22:10:24.6314640Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:10:24.6356173Z http.https://github.com/.extraheader 2024-08-20T22:10:24.6397907Z Entering 'third_party/mimalloc' 2024-08-20T22:10:24.6439806Z http.https://github.com/.extraheader 2024-08-20T22:10:24.6478210Z Entering 'third_party/nccl/nccl' 2024-08-20T22:10:24.6519994Z http.https://github.com/.extraheader 2024-08-20T22:10:24.6557714Z Entering 'third_party/nlohmann' 2024-08-20T22:10:24.6600446Z http.https://github.com/.extraheader 2024-08-20T22:10:24.6639050Z Entering 'third_party/onnx' 2024-08-20T22:10:24.6682356Z http.https://github.com/.extraheader 2024-08-20T22:10:24.6738650Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:10:24.6781376Z http.https://github.com/.extraheader 2024-08-20T22:10:24.6818740Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:10:24.6860532Z http.https://github.com/.extraheader 2024-08-20T22:10:24.6901738Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:10:24.6943917Z http.https://github.com/.extraheader 2024-08-20T22:10:24.6985012Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:10:24.7025934Z http.https://github.com/.extraheader 2024-08-20T22:10:24.7064073Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:10:24.7106132Z http.https://github.com/.extraheader 2024-08-20T22:10:24.7143491Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:10:24.7185912Z http.https://github.com/.extraheader 2024-08-20T22:10:24.7222858Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:10:24.7264254Z http.https://github.com/.extraheader 2024-08-20T22:10:24.7303732Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:10:24.7344712Z http.https://github.com/.extraheader 2024-08-20T22:10:24.7382832Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:10:24.7423802Z http.https://github.com/.extraheader 2024-08-20T22:10:24.7461273Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:10:24.7503917Z http.https://github.com/.extraheader 2024-08-20T22:10:24.7540892Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:10:24.7583483Z http.https://github.com/.extraheader 2024-08-20T22:10:24.7623625Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:10:24.7664524Z http.https://github.com/.extraheader 2024-08-20T22:10:24.7705508Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:10:24.7746687Z http.https://github.com/.extraheader 2024-08-20T22:10:24.7809939Z Entering 'third_party/pocketfft' 2024-08-20T22:10:24.7851817Z http.https://github.com/.extraheader 2024-08-20T22:10:24.7889802Z Entering 'third_party/protobuf' 2024-08-20T22:10:24.7931687Z http.https://github.com/.extraheader 2024-08-20T22:10:24.7973223Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:10:24.8015125Z http.https://github.com/.extraheader 2024-08-20T22:10:24.8052094Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:10:24.8095162Z http.https://github.com/.extraheader 2024-08-20T22:10:24.8134542Z Entering 'third_party/psimd' 2024-08-20T22:10:24.8177254Z http.https://github.com/.extraheader 2024-08-20T22:10:24.8214705Z Entering 'third_party/pthreadpool' 2024-08-20T22:10:24.8256930Z http.https://github.com/.extraheader 2024-08-20T22:10:24.8295145Z Entering 'third_party/pybind11' 2024-08-20T22:10:24.8336777Z http.https://github.com/.extraheader 2024-08-20T22:10:24.8375080Z Entering 'third_party/python-peachpy' 2024-08-20T22:10:24.8416510Z http.https://github.com/.extraheader 2024-08-20T22:10:24.8453723Z Entering 'third_party/sleef' 2024-08-20T22:10:24.8497011Z http.https://github.com/.extraheader 2024-08-20T22:10:24.8534696Z Entering 'third_party/tensorpipe' 2024-08-20T22:10:24.8577309Z http.https://github.com/.extraheader 2024-08-20T22:10:24.8614502Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:10:24.8655922Z http.https://github.com/.extraheader 2024-08-20T22:10:24.8694526Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:10:24.8736185Z http.https://github.com/.extraheader 2024-08-20T22:10:24.8773928Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:10:24.8815756Z http.https://github.com/.extraheader 2024-08-20T22:10:24.8853606Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:10:24.8896364Z http.https://github.com/.extraheader 2024-08-20T22:10:24.8932773Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:10:24.8974998Z http.https://github.com/.extraheader 2024-08-20T22:10:24.9123543Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-20T22:10:24.9147845Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-20T22:10:24.9154201Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:24.9154842Z ##[endgroup] 2024-08-20T22:10:33.1120146Z Cleaning up orphan processes