2024-08-20T21:47:47.3751966Z Current runner version: '2.319.1' 2024-08-20T21:47:47.3760711Z Runner name: 'i-050c06cb9ff8794b9' 2024-08-20T21:47:47.3761854Z Runner group name: 'Default' 2024-08-20T21:47:47.3762893Z Machine name: 'ip-10-0-37-115' 2024-08-20T21:47:47.3785771Z Testing runner upgrade compatibility 2024-08-20T21:47:47.5449148Z ##[group]GITHUB_TOKEN Permissions 2024-08-20T21:47:47.5451801Z Actions: read 2024-08-20T21:47:47.5452577Z Attestations: read 2024-08-20T21:47:47.5453207Z Checks: read 2024-08-20T21:47:47.5453788Z Contents: read 2024-08-20T21:47:47.5454429Z Deployments: read 2024-08-20T21:47:47.5455058Z Discussions: read 2024-08-20T21:47:47.5455686Z Issues: read 2024-08-20T21:47:47.5456279Z Metadata: read 2024-08-20T21:47:47.5456883Z Packages: read 2024-08-20T21:47:47.5457499Z Pages: read 2024-08-20T21:47:47.5458095Z PullRequests: read 2024-08-20T21:47:47.5458833Z RepositoryProjects: read 2024-08-20T21:47:47.5459561Z SecurityEvents: read 2024-08-20T21:47:47.5460218Z Statuses: read 2024-08-20T21:47:47.5461093Z ##[endgroup] 2024-08-20T21:47:47.5465764Z Secret source: Actions 2024-08-20T21:47:47.5466681Z Prepare workflow directory 2024-08-20T21:47:48.0295144Z Prepare all required actions 2024-08-20T21:47:48.0532154Z Getting action download info 2024-08-20T21:47:48.2425492Z Download action repository 'pytorch/test-infra@main' (SHA:0c3a2634aaa2f638c8f640e743f03d696ce1191f) 2024-08-20T21:47:48.6668675Z Download action repository 'pytorch/pytorch@main' (SHA:2e1830c7c8dfc1141c89232e0356671ac07aa979) 2024-08-20T21:47:52.7476958Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-20T21:47:52.9228748Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-20T21:47:53.2681124Z Getting action download info 2024-08-20T21:47:53.3513220Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-20T21:47:53.5414112Z Getting action download info 2024-08-20T21:47:53.6912329Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-20T21:47:53.8438346Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/133712/merge (f2fb9405c2fa9f9502a76363091cce6fd8179736) 2024-08-20T21:47:53.8442213Z ##[group] Inputs 2024-08-20T21:47:53.8442808Z build-environment: linux-focal-cuda11.8-py3.10-gcc9 2024-08-20T21:47:53.8445080Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}]} 2024-08-20T21:47:53.8448033Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:47:53.8449424Z sync-tag: 2024-08-20T21:47:53.8450603Z timeout-minutes: 360 2024-08-20T21:47:53.8451038Z use-gha: 2024-08-20T21:47:53.8451392Z dashboard-tag: 2024-08-20T21:47:53.8451776Z s3-bucket: gha-artifacts 2024-08-20T21:47:53.8452222Z aws-role-to-assume: 2024-08-20T21:47:53.8452645Z ##[endgroup] 2024-08-20T21:47:53.8453953Z Complete job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, amz2023.linux.8xlarge.nvidia.gpu) 2024-08-20T21:47:53.9428278Z A job started hook has been configured by the self-hosted runner administrator 2024-08-20T21:47:53.9630019Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-20T21:47:53.9642539Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:47:53.9643216Z ##[endgroup] 2024-08-20T21:47:55.9447349Z Runner Type: amz2023.linux.8xlarge.nvidia.gpu 2024-08-20T21:47:55.9448044Z Instance Type: g3.8xlarge 2024-08-20T21:47:55.9448930Z AMI Name: al2023-ami-2023.5.20240701.0-kernel-6.1-x86_64 2024-08-20T21:47:55.9449659Z AMI ID: ami-06c68f701d8090592 2024-08-20T21:48:03.1916782Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-20T21:48:03.1917577Z with: 2024-08-20T21:48:03.1918433Z github-secret: *** 2024-08-20T21:48:03.1919654Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T21:48:03.1920943Z activate-with-label: false 2024-08-20T21:48:03.1921393Z label: with-ssh 2024-08-20T21:48:03.1921768Z remove-existing-keys: true 2024-08-20T21:48:03.1922221Z fail-silently: true 2024-08-20T21:48:03.1922607Z env: 2024-08-20T21:48:03.1922927Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:48:03.1923353Z ##[endgroup] 2024-08-20T21:48:03.3118964Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-20T21:48:03.6340676Z Grabbing public ssh keys from https://github.com/pytorchmergebot.keys 2024-08-20T21:48:03.7047523Z No SSH keys found for user pytorchmergebot 2024-08-20T21:48:03.7048380Z Grabbing public ssh keys from https://github.com/XuehaiPan.keys 2024-08-20T21:48:03.7736949Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-20T21:48:03.7759988Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-20T21:48:03.7800940Z Login using: ssh ec2-user@ec2-18-212-166-120.compute-1.amazonaws.com 2024-08-20T21:48:03.7802109Z All testing is done inside the container, to start an interactive session run: 2024-08-20T21:48:03.7803166Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T21:48:03.7947212Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-20T21:48:03.7947936Z with: 2024-08-20T21:48:03.7948283Z submodules: recursive 2024-08-20T21:48:03.7948701Z fetch-depth: 0 2024-08-20T21:48:03.7949041Z env: 2024-08-20T21:48:03.7949364Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:48:03.7949793Z ##[endgroup] 2024-08-20T21:48:03.8214428Z ##[group]Run retry () { 2024-08-20T21:48:03.8214924Z retry () { 2024-08-20T21:48:03.8215558Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-20T21:48:03.8216260Z } 2024-08-20T21:48:03.8216621Z echo "${GITHUB_WORKSPACE}" 2024-08-20T21:48:03.8217148Z if [ -z "${NO_SUDO}" ]; then 2024-08-20T21:48:03.8217750Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T21:48:03.8218306Z else 2024-08-20T21:48:03.8218728Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T21:48:03.8219258Z fi 2024-08-20T21:48:03.8219679Z mkdir "${GITHUB_WORKSPACE}" 2024-08-20T21:48:03.8229044Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:48:03.8229691Z env: 2024-08-20T21:48:03.8230035Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:48:03.8230441Z NO_SUDO: 2024-08-20T21:48:03.8230784Z ##[endgroup] 2024-08-20T21:48:03.8265318Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T21:48:06.8428958Z ##[group]Run malfet/checkout@silent-checkout 2024-08-20T21:48:06.8429505Z with: 2024-08-20T21:48:06.8429891Z ref: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:48:06.8430448Z fetch-depth: 0 2024-08-20T21:48:06.8430831Z submodules: recursive 2024-08-20T21:48:06.8431231Z quiet-checkout: true 2024-08-20T21:48:06.8431661Z repository: pytorch/pytorch 2024-08-20T21:48:06.8432271Z token: *** 2024-08-20T21:48:06.8432608Z ssh-strict: true 2024-08-20T21:48:06.8433008Z persist-credentials: true 2024-08-20T21:48:06.8433444Z clean: true 2024-08-20T21:48:06.8433839Z sparse-checkout-cone-mode: true 2024-08-20T21:48:06.8434318Z lfs: false 2024-08-20T21:48:06.8434686Z set-safe-directory: true 2024-08-20T21:48:06.8435086Z env: 2024-08-20T21:48:06.8435418Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:48:06.8435839Z ##[endgroup] 2024-08-20T21:48:06.9743514Z Syncing repository: pytorch/pytorch 2024-08-20T21:48:06.9745675Z ##[group]Getting Git version info 2024-08-20T21:48:06.9746642Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T21:48:06.9748119Z [command]/usr/bin/git version 2024-08-20T21:48:06.9748575Z git version 2.40.1 2024-08-20T21:48:06.9760580Z ##[endgroup] 2024-08-20T21:48:06.9782912Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/d2f10999-0930-4c1d-acc5-0620640eb1b9' before making global git config changes 2024-08-20T21:48:06.9784524Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T21:48:06.9790011Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T21:48:06.9840029Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T21:48:06.9846043Z ##[group]Initializing the repository 2024-08-20T21:48:06.9850284Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T21:48:06.9885669Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-20T21:48:06.9886744Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-20T21:48:06.9887755Z hint: of your new repositories, which will suppress this warning, call: 2024-08-20T21:48:06.9888447Z hint: 2024-08-20T21:48:06.9889189Z hint: git config --global init.defaultBranch 2024-08-20T21:48:06.9889778Z hint: 2024-08-20T21:48:06.9890445Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-20T21:48:06.9891549Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-20T21:48:06.9892571Z hint: 2024-08-20T21:48:06.9893001Z hint: git branch -m 2024-08-20T21:48:06.9893951Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-20T21:48:06.9902653Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-20T21:48:06.9940660Z ##[endgroup] 2024-08-20T21:48:06.9941412Z ##[group]Disabling automatic garbage collection 2024-08-20T21:48:06.9945385Z [command]/usr/bin/git config --local gc.auto 0 2024-08-20T21:48:06.9980645Z ##[endgroup] 2024-08-20T21:48:06.9981277Z ##[group]Setting up auth 2024-08-20T21:48:06.9990414Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T21:48:07.0028029Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T21:48:07.0360706Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T21:48:07.0395980Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T21:48:07.0720418Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T21:48:07.0775382Z ##[endgroup] 2024-08-20T21:48:07.0776020Z ##[group]Fetching the repository 2024-08-20T21:48:07.0784467Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-20T21:48:09.7142224Z remote: Enumerating objects: 1008453 2024-08-20T21:48:09.7142957Z remote: Enumerating objects: 1010156, done. 2024-08-20T21:48:09.7144022Z remote: Counting objects: 0% (1/1703) 2024-08-20T21:48:09.7144644Z remote: Counting objects: 1% (18/1703) 2024-08-20T21:48:09.7145310Z remote: Counting objects: 2% (35/1703) 2024-08-20T21:48:09.7145966Z remote: Counting objects: 3% (52/1703) 2024-08-20T21:48:09.7146583Z remote: Counting objects: 4% (69/1703) 2024-08-20T21:48:09.7147214Z remote: Counting objects: 5% (86/1703) 2024-08-20T21:48:09.7147844Z remote: Counting objects: 6% (103/1703) 2024-08-20T21:48:09.7148463Z remote: Counting objects: 7% (120/1703) 2024-08-20T21:48:09.7149525Z remote: Counting objects: 8% (137/1703) 2024-08-20T21:48:09.7150155Z remote: Counting objects: 9% (154/1703) 2024-08-20T21:48:09.7150787Z remote: Counting objects: 10% (171/1703) 2024-08-20T21:48:09.7151410Z remote: Counting objects: 11% (188/1703) 2024-08-20T21:48:09.7152037Z remote: Counting objects: 12% (205/1703) 2024-08-20T21:48:09.7152678Z remote: Counting objects: 13% (222/1703) 2024-08-20T21:48:09.7153286Z remote: Counting objects: 14% (239/1703) 2024-08-20T21:48:09.7153920Z remote: Counting objects: 15% (256/1703) 2024-08-20T21:48:09.7154556Z remote: Counting objects: 16% (273/1703) 2024-08-20T21:48:09.7155154Z remote: Counting objects: 17% (290/1703) 2024-08-20T21:48:09.7155788Z remote: Counting objects: 18% (307/1703) 2024-08-20T21:48:09.7156414Z remote: Counting objects: 19% (324/1703) 2024-08-20T21:48:09.7157048Z remote: Counting objects: 20% (341/1703) 2024-08-20T21:48:09.7157681Z remote: Counting objects: 21% (358/1703) 2024-08-20T21:48:09.7158311Z remote: Counting objects: 22% (375/1703) 2024-08-20T21:48:09.7158935Z remote: Counting objects: 23% (392/1703) 2024-08-20T21:48:09.7159540Z remote: Counting objects: 24% (409/1703) 2024-08-20T21:48:09.7160164Z remote: Counting objects: 25% (426/1703) 2024-08-20T21:48:09.7160800Z remote: Counting objects: 26% (443/1703) 2024-08-20T21:48:09.7161402Z remote: Counting objects: 27% (460/1703) 2024-08-20T21:48:09.7162223Z remote: Counting objects: 28% (477/1703) 2024-08-20T21:48:09.7162880Z remote: Counting objects: 29% (494/1703) 2024-08-20T21:48:09.7163485Z remote: Counting objects: 30% (511/1703) 2024-08-20T21:48:09.7164117Z remote: Counting objects: 31% (528/1703) 2024-08-20T21:48:09.7164749Z remote: Counting objects: 32% (545/1703) 2024-08-20T21:48:09.7165373Z remote: Counting objects: 33% (562/1703) 2024-08-20T21:48:09.7165995Z remote: Counting objects: 34% (580/1703) 2024-08-20T21:48:09.7166617Z remote: Counting objects: 35% (597/1703) 2024-08-20T21:48:09.7167218Z remote: Counting objects: 36% (614/1703) 2024-08-20T21:48:09.7167842Z remote: Counting objects: 37% (631/1703) 2024-08-20T21:48:09.7168464Z remote: Counting objects: 38% (648/1703) 2024-08-20T21:48:09.7169064Z remote: Counting objects: 39% (665/1703) 2024-08-20T21:48:09.7169684Z remote: Counting objects: 40% (682/1703) 2024-08-20T21:48:09.7170324Z remote: Counting objects: 41% (699/1703) 2024-08-20T21:48:09.7170952Z remote: Counting objects: 42% (716/1703) 2024-08-20T21:48:09.7171558Z remote: Counting objects: 43% (733/1703) 2024-08-20T21:48:09.7172185Z remote: Counting objects: 44% (750/1703) 2024-08-20T21:48:09.7172805Z remote: Counting objects: 45% (767/1703) 2024-08-20T21:48:09.7173462Z remote: Counting objects: 46% (784/1703) 2024-08-20T21:48:09.7174091Z remote: Counting objects: 47% (801/1703) 2024-08-20T21:48:09.7174721Z remote: Counting objects: 48% (818/1703) 2024-08-20T21:48:09.7175321Z remote: Counting objects: 49% (835/1703) 2024-08-20T21:48:09.7175945Z remote: Counting objects: 50% (852/1703) 2024-08-20T21:48:09.7176570Z remote: Counting objects: 51% (869/1703) 2024-08-20T21:48:09.7177194Z remote: Counting objects: 52% (886/1703) 2024-08-20T21:48:09.7177794Z remote: Counting objects: 53% (903/1703) 2024-08-20T21:48:09.7178425Z remote: Counting objects: 54% (920/1703) 2024-08-20T21:48:09.7179051Z remote: Counting objects: 55% (937/1703) 2024-08-20T21:48:09.7179652Z remote: Counting objects: 56% (954/1703) 2024-08-20T21:48:09.7180283Z remote: Counting objects: 57% (971/1703) 2024-08-20T21:48:09.7180908Z remote: Counting objects: 58% (988/1703) 2024-08-20T21:48:09.7181647Z remote: Counting objects: 59% (1005/1703) 2024-08-20T21:48:09.7182305Z remote: Counting objects: 60% (1022/1703) 2024-08-20T21:48:09.7182961Z remote: Counting objects: 61% (1039/1703) 2024-08-20T21:48:09.7183597Z remote: Counting objects: 62% (1056/1703) 2024-08-20T21:48:09.7184259Z remote: Counting objects: 63% (1073/1703) 2024-08-20T21:48:09.7184906Z remote: Counting objects: 64% (1090/1703) 2024-08-20T21:48:09.7185554Z remote: Counting objects: 65% (1107/1703) 2024-08-20T21:48:09.7186189Z remote: Counting objects: 66% (1124/1703) 2024-08-20T21:48:09.7186840Z remote: Counting objects: 67% (1142/1703) 2024-08-20T21:48:09.7187487Z remote: Counting objects: 68% (1159/1703) 2024-08-20T21:48:09.7188116Z remote: Counting objects: 69% (1176/1703) 2024-08-20T21:48:09.7188759Z remote: Counting objects: 70% (1193/1703) 2024-08-20T21:48:09.7189405Z remote: Counting objects: 71% (1210/1703) 2024-08-20T21:48:09.7190042Z remote: Counting objects: 72% (1227/1703) 2024-08-20T21:48:09.7190692Z remote: Counting objects: 73% (1244/1703) 2024-08-20T21:48:09.7191339Z remote: Counting objects: 74% (1261/1703) 2024-08-20T21:48:09.7191981Z remote: Counting objects: 75% (1278/1703) 2024-08-20T21:48:09.7192610Z remote: Counting objects: 76% (1295/1703) 2024-08-20T21:48:09.7193256Z remote: Counting objects: 77% (1312/1703) 2024-08-20T21:48:09.7193898Z remote: Counting objects: 78% (1329/1703) 2024-08-20T21:48:09.7195457Z remote: Counting objects: 79% (1346/1703) 2024-08-20T21:48:09.7196159Z remote: Counting objects: 80% (1363/1703) 2024-08-20T21:48:09.7196806Z remote: Counting objects: 81% (1380/1703) 2024-08-20T21:48:09.7197437Z remote: Counting objects: 82% (1397/1703) 2024-08-20T21:48:09.7198086Z remote: Counting objects: 83% (1414/1703) 2024-08-20T21:48:09.7198730Z remote: Counting objects: 84% (1431/1703) 2024-08-20T21:48:09.7199366Z remote: Counting objects: 85% (1448/1703) 2024-08-20T21:48:09.7200011Z remote: Counting objects: 86% (1465/1703) 2024-08-20T21:48:09.7200654Z remote: Counting objects: 87% (1482/1703) 2024-08-20T21:48:09.7201279Z remote: Counting objects: 88% (1499/1703) 2024-08-20T21:48:09.7201923Z remote: Counting objects: 89% (1516/1703) 2024-08-20T21:48:09.7202568Z remote: Counting objects: 90% (1533/1703) 2024-08-20T21:48:09.7203193Z remote: Counting objects: 91% (1550/1703) 2024-08-20T21:48:09.7203841Z remote: Counting objects: 92% (1567/1703) 2024-08-20T21:48:09.7204488Z remote: Counting objects: 93% (1584/1703) 2024-08-20T21:48:09.7205115Z remote: Counting objects: 94% (1601/1703) 2024-08-20T21:48:09.7205763Z remote: Counting objects: 95% (1618/1703) 2024-08-20T21:48:09.7206407Z remote: Counting objects: 96% (1635/1703) 2024-08-20T21:48:09.7207051Z remote: Counting objects: 97% (1652/1703) 2024-08-20T21:48:09.7207688Z remote: Counting objects: 98% (1669/1703) 2024-08-20T21:48:09.7208329Z remote: Counting objects: 99% (1686/1703) 2024-08-20T21:48:09.7208972Z remote: Counting objects: 100% (1703/1703) 2024-08-20T21:48:09.7209647Z remote: Counting objects: 100% (1703/1703), done. 2024-08-20T21:48:09.7357747Z remote: Compressing objects: 0% (1/853) 2024-08-20T21:48:09.7733881Z remote: Compressing objects: 1% (9/853) 2024-08-20T21:48:09.7942042Z remote: Compressing objects: 2% (18/853) 2024-08-20T21:48:09.8502652Z remote: Compressing objects: 3% (26/853) 2024-08-20T21:48:09.9258277Z remote: Compressing objects: 4% (35/853) 2024-08-20T21:48:09.9825523Z remote: Compressing objects: 5% (43/853) 2024-08-20T21:48:10.0194533Z remote: Compressing objects: 6% (52/853) 2024-08-20T21:48:10.0556901Z remote: Compressing objects: 7% (60/853) 2024-08-20T21:48:10.0838309Z remote: Compressing objects: 8% (69/853) 2024-08-20T21:48:10.1179430Z remote: Compressing objects: 9% (77/853) 2024-08-20T21:48:10.1396905Z remote: Compressing objects: 10% (86/853) 2024-08-20T21:48:10.1620080Z remote: Compressing objects: 11% (94/853) 2024-08-20T21:48:10.1836671Z remote: Compressing objects: 12% (103/853) 2024-08-20T21:48:10.1993666Z remote: Compressing objects: 13% (111/853) 2024-08-20T21:48:10.2093407Z remote: Compressing objects: 14% (120/853) 2024-08-20T21:48:10.2184524Z remote: Compressing objects: 15% (128/853) 2024-08-20T21:48:10.2279011Z remote: Compressing objects: 16% (137/853) 2024-08-20T21:48:10.2349679Z remote: Compressing objects: 17% (146/853) 2024-08-20T21:48:10.2414149Z remote: Compressing objects: 18% (154/853) 2024-08-20T21:48:10.2444521Z remote: Compressing objects: 19% (163/853) 2024-08-20T21:48:10.2474884Z remote: Compressing objects: 20% (171/853) 2024-08-20T21:48:10.2500913Z remote: Compressing objects: 21% (180/853) 2024-08-20T21:48:10.2505701Z remote: Compressing objects: 22% (188/853) 2024-08-20T21:48:10.2509527Z remote: Compressing objects: 23% (197/853) 2024-08-20T21:48:10.2514646Z remote: Compressing objects: 24% (205/853) 2024-08-20T21:48:10.2517453Z remote: Compressing objects: 25% (214/853) 2024-08-20T21:48:10.2524960Z remote: Compressing objects: 26% (222/853) 2024-08-20T21:48:10.2533677Z remote: Compressing objects: 27% (231/853) 2024-08-20T21:48:10.2548136Z remote: Compressing objects: 28% (239/853) 2024-08-20T21:48:10.2561888Z remote: Compressing objects: 29% (248/853) 2024-08-20T21:48:10.2565506Z remote: Compressing objects: 30% (256/853) 2024-08-20T21:48:10.2580017Z remote: Compressing objects: 31% (265/853) 2024-08-20T21:48:10.2585489Z remote: Compressing objects: 32% (273/853) 2024-08-20T21:48:10.2595353Z remote: Compressing objects: 33% (282/853) 2024-08-20T21:48:10.2601795Z remote: Compressing objects: 34% (291/853) 2024-08-20T21:48:10.2608162Z remote: Compressing objects: 35% (299/853) 2024-08-20T21:48:10.2613170Z remote: Compressing objects: 36% (308/853) 2024-08-20T21:48:10.2623272Z remote: Compressing objects: 37% (316/853) 2024-08-20T21:48:10.2629634Z remote: Compressing objects: 38% (325/853) 2024-08-20T21:48:10.2636689Z remote: Compressing objects: 39% (333/853) 2024-08-20T21:48:10.2644299Z remote: Compressing objects: 40% (342/853) 2024-08-20T21:48:10.2646990Z remote: Compressing objects: 41% (350/853) 2024-08-20T21:48:10.2656976Z remote: Compressing objects: 42% (359/853) 2024-08-20T21:48:10.2659562Z remote: Compressing objects: 43% (367/853) 2024-08-20T21:48:10.2665110Z remote: Compressing objects: 44% (376/853) 2024-08-20T21:48:10.2672547Z remote: Compressing objects: 45% (384/853) 2024-08-20T21:48:10.2677468Z remote: Compressing objects: 46% (393/853) 2024-08-20T21:48:10.2682463Z remote: Compressing objects: 47% (401/853) 2024-08-20T21:48:10.2685941Z remote: Compressing objects: 48% (410/853) 2024-08-20T21:48:10.2689165Z remote: Compressing objects: 49% (418/853) 2024-08-20T21:48:10.2692704Z remote: Compressing objects: 50% (427/853) 2024-08-20T21:48:10.2694897Z remote: Compressing objects: 51% (436/853) 2024-08-20T21:48:10.2698893Z remote: Compressing objects: 52% (444/853) 2024-08-20T21:48:10.2701706Z remote: Compressing objects: 53% (453/853) 2024-08-20T21:48:10.2703940Z remote: Compressing objects: 54% (461/853) 2024-08-20T21:48:10.2705425Z remote: Compressing objects: 55% (470/853) 2024-08-20T21:48:10.2707826Z remote: Compressing objects: 56% (478/853) 2024-08-20T21:48:10.2709677Z remote: Compressing objects: 57% (487/853) 2024-08-20T21:48:10.2710351Z remote: Compressing objects: 58% (495/853) 2024-08-20T21:48:10.2711233Z remote: Compressing objects: 59% (504/853) 2024-08-20T21:48:10.2711899Z remote: Compressing objects: 60% (512/853) 2024-08-20T21:48:10.2712571Z remote: Compressing objects: 61% (521/853) 2024-08-20T21:48:10.2713236Z remote: Compressing objects: 62% (529/853) 2024-08-20T21:48:10.2713888Z remote: Compressing objects: 63% (538/853) 2024-08-20T21:48:10.2714547Z remote: Compressing objects: 64% (546/853) 2024-08-20T21:48:10.2717074Z remote: Compressing objects: 65% (555/853) 2024-08-20T21:48:10.2724524Z remote: Compressing objects: 66% (563/853) 2024-08-20T21:48:10.2729488Z remote: Compressing objects: 67% (572/853) 2024-08-20T21:48:10.2733398Z remote: Compressing objects: 68% (581/853) 2024-08-20T21:48:10.2741058Z remote: Compressing objects: 69% (589/853) 2024-08-20T21:48:10.2742351Z remote: Compressing objects: 70% (598/853) 2024-08-20T21:48:10.2745898Z remote: Compressing objects: 71% (606/853) 2024-08-20T21:48:10.2749763Z remote: Compressing objects: 72% (615/853) 2024-08-20T21:48:10.2752875Z remote: Compressing objects: 73% (623/853) 2024-08-20T21:48:10.2756735Z remote: Compressing objects: 74% (632/853) 2024-08-20T21:48:10.2759348Z remote: Compressing objects: 75% (640/853) 2024-08-20T21:48:10.2762766Z remote: Compressing objects: 76% (649/853) 2024-08-20T21:48:10.2765353Z remote: Compressing objects: 77% (657/853) 2024-08-20T21:48:10.2767915Z remote: Compressing objects: 78% (666/853) 2024-08-20T21:48:10.2769327Z remote: Compressing objects: 79% (674/853) 2024-08-20T21:48:10.2771411Z remote: Compressing objects: 80% (683/853) 2024-08-20T21:48:10.2773907Z remote: Compressing objects: 81% (691/853) 2024-08-20T21:48:10.2775229Z remote: Compressing objects: 82% (700/853) 2024-08-20T21:48:10.2777928Z remote: Compressing objects: 83% (708/853) 2024-08-20T21:48:10.2779348Z remote: Compressing objects: 84% (717/853) 2024-08-20T21:48:10.2781322Z remote: Compressing objects: 85% (726/853) 2024-08-20T21:48:10.2783320Z remote: Compressing objects: 86% (734/853) 2024-08-20T21:48:10.2784344Z remote: Compressing objects: 87% (743/853) 2024-08-20T21:48:10.2786916Z remote: Compressing objects: 88% (751/853) 2024-08-20T21:48:10.2788345Z remote: Compressing objects: 89% (760/853) 2024-08-20T21:48:10.2791175Z remote: Compressing objects: 90% (768/853) 2024-08-20T21:48:10.2791840Z remote: Compressing objects: 91% (777/853) 2024-08-20T21:48:10.2792506Z remote: Compressing objects: 92% (785/853) 2024-08-20T21:48:10.2793319Z remote: Compressing objects: 93% (794/853) 2024-08-20T21:48:10.2794187Z remote: Compressing objects: 94% (802/853) 2024-08-20T21:48:10.2796051Z remote: Compressing objects: 95% (811/853) 2024-08-20T21:48:10.2797124Z remote: Compressing objects: 96% (819/853) 2024-08-20T21:48:10.2798031Z remote: Compressing objects: 97% (828/853) 2024-08-20T21:48:10.2799392Z remote: Compressing objects: 98% (836/853) 2024-08-20T21:48:10.2800587Z remote: Compressing objects: 99% (845/853) 2024-08-20T21:48:10.2801256Z remote: Compressing objects: 100% (853/853) 2024-08-20T21:48:10.2801971Z remote: Compressing objects: 100% (853/853), done. 2024-08-20T21:48:37.9443363Z remote: Total 1010156 (delta 1122), reused 1319 (delta 847), pack-reused 1008453 (from 1) 2024-08-20T21:48:59.8345361Z [command]/usr/bin/git rev-parse --verify --quiet 40ec5f6ddd9787aca0449b24128343ff4c4a88b3^{object} 2024-08-20T21:48:59.8378109Z 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:48:59.8383982Z ##[endgroup] 2024-08-20T21:48:59.8384747Z ##[group]Determining the checkout info 2024-08-20T21:48:59.8386559Z ##[endgroup] 2024-08-20T21:48:59.8387198Z ##[group]Checking out the ref 2024-08-20T21:48:59.8391297Z [command]/usr/bin/git checkout --quiet --force 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:49:01.7335482Z ##[endgroup] 2024-08-20T21:49:01.7336258Z ##[group]Setting up auth for fetching submodules 2024-08-20T21:49:01.7341465Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T21:49:01.7403619Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-20T21:49:01.7440626Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-20T21:49:01.7475983Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-20T21:49:01.7508159Z ##[endgroup] 2024-08-20T21:49:01.7510321Z ##[group]Fetching submodules 2024-08-20T21:49:01.7514115Z [command]/usr/bin/git submodule sync --recursive 2024-08-20T21:49:01.7871328Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-20T21:49:01.8212737Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-20T21:49:01.8214989Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-20T21:49:01.8218947Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-20T21:49:01.8222647Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-20T21:49:01.8227795Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-20T21:49:01.8230218Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-20T21:49:01.8272957Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-20T21:49:01.8275011Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-20T21:49:01.8276811Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-20T21:49:01.8278705Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-20T21:49:01.8280460Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-20T21:49:01.8282101Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-20T21:49:01.8283705Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-20T21:49:01.8285440Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-20T21:49:01.8287097Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-20T21:49:01.8288855Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:49:01.8290673Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-20T21:49:01.8292411Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-20T21:49:01.8296060Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-20T21:49:01.8301715Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-20T21:49:01.8307473Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-20T21:49:01.8313317Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-20T21:49:01.8319179Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-20T21:49:01.8325204Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-20T21:49:01.8331431Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-20T21:49:01.8338349Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-20T21:49:01.8344719Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-20T21:49:01.8351307Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-20T21:49:01.8357747Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-20T21:49:01.8364778Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-20T21:49:01.8371382Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-20T21:49:01.8378458Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-20T21:49:01.8385368Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-20T21:49:01.8394726Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-20T21:49:01.8427674Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-20T21:49:02.1762286Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-20T21:49:02.3881585Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-20T21:49:02.5613793Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-20T21:49:02.8316915Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-20T21:49:04.8674420Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-20T21:49:17.1574951Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-20T21:49:17.6039764Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-20T21:49:18.0375514Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-20T21:49:18.7135136Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-20T21:49:20.1597849Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-20T21:49:22.6995256Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-20T21:49:29.5504864Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-20T21:49:31.1457080Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-20T21:49:33.0165485Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-20T21:49:34.3883026Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-20T21:49:34.9099880Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-20T21:49:35.2643340Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-20T21:49:36.5427219Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-20T21:49:36.9752585Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-20T21:49:37.2399960Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-20T21:49:39.0999986Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-20T21:49:39.8618824Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-20T21:49:40.5453637Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-20T21:49:47.4482976Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-20T21:49:49.5738395Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-20T21:49:53.2711439Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-20T21:49:53.9620038Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-20T21:50:04.3614591Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-20T21:50:04.5865122Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-20T21:50:04.7862121Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-20T21:50:05.7905378Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-20T21:50:06.1077269Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-20T21:50:06.7815762Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-20T21:50:07.2537567Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-20T21:50:07.2683278Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-20T21:50:07.2791915Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-20T21:50:07.3122947Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-20T21:50:07.3583223Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-20T21:50:08.6757381Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-20T21:50:08.7058813Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-20T21:50:08.7612171Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-20T21:50:08.8867022Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-20T21:50:08.9297694Z Submodule path 'third_party/cudnn_frontend': checked out '23511ba176243f27b3b275da1fb3814ea805a171' 2024-08-20T21:50:09.6138310Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-20T21:50:09.9355878Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-20T21:50:10.0419982Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-20T21:50:10.0440809Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:50:10.0444360Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:50:10.0448044Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:50:10.0451937Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:50:10.0456393Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:50:10.0486846Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-20T21:50:11.1381139Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-20T21:50:11.7793501Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-20T21:50:14.2240177Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-20T21:50:15.3684477Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-20T21:50:15.7711129Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-20T21:50:15.8933920Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-20T21:50:16.4558923Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-20T21:50:16.5340839Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-20T21:50:16.5485174Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-20T21:50:16.7060427Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-20T21:50:16.7559057Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-20T21:50:16.8043554Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-20T21:50:16.8379866Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-20T21:50:16.8959961Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T21:50:16.9115110Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-20T21:50:16.9134919Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-20T21:50:16.9165397Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-20T21:50:31.2869634Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-20T21:50:31.3093579Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-20T21:50:31.4208691Z Submodule path 'third_party/kineto': checked out 'd9753139d181b9ff42872465aac0e5d3018be415' 2024-08-20T21:50:31.4230766Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:50:31.4233598Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:50:31.4238143Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:50:31.4268291Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-20T21:50:32.0803768Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-20T21:50:33.3644848Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-20T21:50:34.5934343Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-20T21:50:34.5954178Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:50:34.5957802Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:50:34.5961576Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:50:34.5965525Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:50:34.5970027Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:50:34.5974465Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:50:34.5978725Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:50:34.5983012Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:50:34.6014112Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-20T21:50:35.6201326Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-20T21:50:36.0723552Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-20T21:50:37.3856736Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-20T21:50:37.6932198Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-20T21:50:38.2566571Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-20T21:50:39.4556666Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-20T21:50:46.3845121Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-20T21:50:46.8322372Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-20T21:50:46.8563722Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-20T21:50:46.9032834Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-20T21:50:46.9200306Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-20T21:50:46.9219724Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:50:46.9249951Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-20T21:50:47.2440364Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-20T21:50:47.2673262Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-20T21:50:47.3195407Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-20T21:50:47.4556934Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-20T21:50:47.4762735Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-20T21:50:47.5257265Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-20T21:50:47.5981764Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-20T21:50:47.6445127Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-20T21:50:47.6745020Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-20T21:50:47.8086299Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-20T21:50:48.3124389Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-20T21:50:48.3164910Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-20T21:50:48.3167910Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-20T21:50:48.3198963Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-20T21:50:48.8224778Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-20T21:50:49.8569536Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-20T21:50:49.9006055Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-20T21:50:49.9960374Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-20T21:50:49.9984372Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:50:49.9987835Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:50:49.9991886Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:50:49.9996008Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:50:50.0000025Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:50:50.0004140Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:50:50.0009065Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:50:50.0013154Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:50:50.0045498Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-20T21:50:50.4984628Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-20T21:50:51.6634869Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-20T21:50:52.0077565Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-20T21:50:58.9112199Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-20T21:50:59.1987301Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-20T21:50:59.4094733Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-20T21:50:59.7231168Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-20T21:51:07.1424466Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-20T21:51:07.1945290Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-20T21:51:07.2138507Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-20T21:51:07.3468603Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-20T21:51:07.3628229Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-20T21:51:07.3813388Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-20T21:51:07.4017503Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-20T21:51:07.4036540Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:51:07.4040732Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:51:07.4069720Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-20T21:51:09.4022487Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-20T21:51:10.9668230Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-20T21:51:11.0250743Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T21:51:11.7490334Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-20T21:51:11.7629016Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-20T21:51:12.1101041Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-20T21:51:12.1126608Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:51:12.1130958Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-20T21:51:12.1161965Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-20T21:51:12.6257270Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-20T21:51:13.7799825Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-20T21:51:13.8714538Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-20T21:51:13.8821205Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-20T21:51:13.8973807Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-20T21:51:13.9438803Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-20T21:51:13.9800014Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-20T21:51:14.0333503Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-20T21:51:14.0686008Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-20T21:51:14.0706022Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:51:14.0709823Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:51:14.0713786Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:51:14.0717535Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:51:14.0747342Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-20T21:51:15.2214140Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-20T21:51:15.4658660Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-20T21:51:16.7338472Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-20T21:51:17.7834865Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-20T21:51:17.8031453Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-20T21:51:17.8890275Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-20T21:51:17.9249163Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-20T21:51:17.9268280Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:51:17.9298558Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-20T21:51:18.1288739Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-20T21:51:18.1327768Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-20T21:51:18.1669491Z Entering 'android/libs/fbjni' 2024-08-20T21:51:18.1717429Z Entering 'third_party/FP16' 2024-08-20T21:51:18.1765772Z Entering 'third_party/FXdiv' 2024-08-20T21:51:18.1813050Z Entering 'third_party/NNPACK' 2024-08-20T21:51:18.1861978Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T21:51:18.1908787Z Entering 'third_party/XNNPACK' 2024-08-20T21:51:18.1973816Z Entering 'third_party/benchmark' 2024-08-20T21:51:18.2022000Z Entering 'third_party/cpp-httplib' 2024-08-20T21:51:18.2069612Z Entering 'third_party/cpuinfo' 2024-08-20T21:51:18.2117322Z Entering 'third_party/cudnn_frontend' 2024-08-20T21:51:18.2167655Z Entering 'third_party/cutlass' 2024-08-20T21:51:18.2224403Z Entering 'third_party/eigen' 2024-08-20T21:51:18.2274336Z Entering 'third_party/fbgemm' 2024-08-20T21:51:18.2321706Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:51:18.2369243Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:51:18.2416392Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:51:18.2470861Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:51:18.2519417Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:51:18.2568079Z Entering 'third_party/flatbuffers' 2024-08-20T21:51:18.2618483Z Entering 'third_party/fmt' 2024-08-20T21:51:18.2667271Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:51:18.2714450Z Entering 'third_party/gloo' 2024-08-20T21:51:18.2762868Z Entering 'third_party/googletest' 2024-08-20T21:51:18.2810245Z Entering 'third_party/ideep' 2024-08-20T21:51:18.2856617Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T21:51:18.2911549Z Entering 'third_party/ittapi' 2024-08-20T21:51:18.2960285Z Entering 'third_party/kineto' 2024-08-20T21:51:18.3007482Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:51:18.3055796Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:51:18.3103490Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:51:18.3151825Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:51:18.3198517Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:51:18.3245564Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:51:18.3294552Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:51:18.3343014Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:51:18.3389979Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:51:18.3439365Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:51:18.3487994Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:51:18.3534394Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:51:18.3583491Z Entering 'third_party/mimalloc' 2024-08-20T21:51:18.3631247Z Entering 'third_party/nccl/nccl' 2024-08-20T21:51:18.3678778Z Entering 'third_party/nlohmann' 2024-08-20T21:51:18.3726620Z Entering 'third_party/onnx' 2024-08-20T21:51:18.3793738Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T21:51:18.3841176Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T21:51:18.3890987Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T21:51:18.3942154Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:51:18.3987953Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:51:18.4034264Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:51:18.4081510Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:51:18.4130899Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:51:18.4177593Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:51:18.4223724Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:51:18.4272593Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:51:18.4322051Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:51:18.4372469Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:51:18.4440950Z Entering 'third_party/pocketfft' 2024-08-20T21:51:18.4487768Z Entering 'third_party/protobuf' 2024-08-20T21:51:18.4540019Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:51:18.4586743Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T21:51:18.4635608Z Entering 'third_party/psimd' 2024-08-20T21:51:18.4683521Z Entering 'third_party/pthreadpool' 2024-08-20T21:51:18.4730114Z Entering 'third_party/pybind11' 2024-08-20T21:51:18.4777713Z Entering 'third_party/python-peachpy' 2024-08-20T21:51:18.4824838Z Entering 'third_party/sleef' 2024-08-20T21:51:18.4873373Z Entering 'third_party/tensorpipe' 2024-08-20T21:51:18.4921023Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:51:18.4967905Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:51:18.5013385Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:51:18.5061239Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:51:18.5106692Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:51:18.5170479Z ##[endgroup] 2024-08-20T21:51:18.5173427Z ##[group]Persisting credentials for submodules 2024-08-20T21:51:18.5177093Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-20T21:51:18.5518736Z Entering 'android/libs/fbjni' 2024-08-20T21:51:18.5581823Z Entering 'third_party/FP16' 2024-08-20T21:51:18.5645068Z Entering 'third_party/FXdiv' 2024-08-20T21:51:18.5706731Z Entering 'third_party/NNPACK' 2024-08-20T21:51:18.5769808Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T21:51:18.5831395Z Entering 'third_party/XNNPACK' 2024-08-20T21:51:18.5911878Z Entering 'third_party/benchmark' 2024-08-20T21:51:18.5974477Z Entering 'third_party/cpp-httplib' 2024-08-20T21:51:18.6036529Z Entering 'third_party/cpuinfo' 2024-08-20T21:51:18.6099405Z Entering 'third_party/cudnn_frontend' 2024-08-20T21:51:18.6161501Z Entering 'third_party/cutlass' 2024-08-20T21:51:18.6232561Z Entering 'third_party/eigen' 2024-08-20T21:51:18.6297546Z Entering 'third_party/fbgemm' 2024-08-20T21:51:18.6360144Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:51:18.6421043Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:51:18.6482849Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:51:18.6553383Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:51:18.6614496Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:51:18.6677666Z Entering 'third_party/flatbuffers' 2024-08-20T21:51:18.6744489Z Entering 'third_party/fmt' 2024-08-20T21:51:18.6806391Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:51:18.6868815Z Entering 'third_party/gloo' 2024-08-20T21:51:18.6930377Z Entering 'third_party/googletest' 2024-08-20T21:51:18.6993081Z Entering 'third_party/ideep' 2024-08-20T21:51:18.7053770Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T21:51:18.7121302Z Entering 'third_party/ittapi' 2024-08-20T21:51:18.7184760Z Entering 'third_party/kineto' 2024-08-20T21:51:18.7246471Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:51:18.7308180Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:51:18.7372638Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:51:18.7434674Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:51:18.7497773Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:51:18.7558563Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:51:18.7622032Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:51:18.7684291Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:51:18.7747015Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:51:18.7810261Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:51:18.7875413Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:51:18.7936782Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:51:18.8001037Z Entering 'third_party/mimalloc' 2024-08-20T21:51:18.8063044Z Entering 'third_party/nccl/nccl' 2024-08-20T21:51:18.8124727Z Entering 'third_party/nlohmann' 2024-08-20T21:51:18.8189115Z Entering 'third_party/onnx' 2024-08-20T21:51:18.8267132Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T21:51:18.8328879Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T21:51:18.8394113Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T21:51:18.8457006Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:51:18.8517804Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:51:18.8579069Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:51:18.8640031Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:51:18.8702298Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:51:18.8763030Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:51:18.8823974Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:51:18.8885997Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:51:18.8950724Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:51:18.9013844Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:51:18.9097447Z Entering 'third_party/pocketfft' 2024-08-20T21:51:18.9158767Z Entering 'third_party/protobuf' 2024-08-20T21:51:18.9225065Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:51:18.9286941Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T21:51:18.9351081Z Entering 'third_party/psimd' 2024-08-20T21:51:18.9412508Z Entering 'third_party/pthreadpool' 2024-08-20T21:51:18.9474703Z Entering 'third_party/pybind11' 2024-08-20T21:51:18.9536411Z Entering 'third_party/python-peachpy' 2024-08-20T21:51:18.9598572Z Entering 'third_party/sleef' 2024-08-20T21:51:18.9662267Z Entering 'third_party/tensorpipe' 2024-08-20T21:51:18.9723142Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:51:18.9784289Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:51:18.9845792Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:51:18.9906910Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:51:18.9967874Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:51:19.0048129Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-20T21:51:19.0387575Z Entering 'android/libs/fbjni' 2024-08-20T21:51:19.0445894Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-20T21:51:19.0463666Z Entering 'third_party/FP16' 2024-08-20T21:51:19.0521675Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-20T21:51:19.0540507Z Entering 'third_party/FXdiv' 2024-08-20T21:51:19.0598270Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-20T21:51:19.0616739Z Entering 'third_party/NNPACK' 2024-08-20T21:51:19.0675298Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-20T21:51:19.0693395Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T21:51:19.0751790Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-20T21:51:19.0769940Z Entering 'third_party/XNNPACK' 2024-08-20T21:51:19.0827487Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-20T21:51:19.0863872Z Entering 'third_party/benchmark' 2024-08-20T21:51:19.0921824Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-20T21:51:19.0941064Z Entering 'third_party/cpp-httplib' 2024-08-20T21:51:19.0998674Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-20T21:51:19.1016637Z Entering 'third_party/cpuinfo' 2024-08-20T21:51:19.1074720Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T21:51:19.1093702Z Entering 'third_party/cudnn_frontend' 2024-08-20T21:51:19.1152280Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-20T21:51:19.1170248Z Entering 'third_party/cutlass' 2024-08-20T21:51:19.1229470Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-20T21:51:19.1258014Z Entering 'third_party/eigen' 2024-08-20T21:51:19.1315155Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-20T21:51:19.1335738Z Entering 'third_party/fbgemm' 2024-08-20T21:51:19.1396083Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-20T21:51:19.1413863Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:51:19.1472438Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-20T21:51:19.1490116Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:51:19.1550249Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T21:51:19.1568853Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:51:19.1627351Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-20T21:51:19.1653604Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:51:19.1712372Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:51:19.1730230Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:51:19.1789117Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-20T21:51:19.1808167Z Entering 'third_party/flatbuffers' 2024-08-20T21:51:19.1866615Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-20T21:51:19.1887555Z Entering 'third_party/fmt' 2024-08-20T21:51:19.1945729Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-20T21:51:19.1963976Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:51:19.2022276Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-20T21:51:19.2042031Z Entering 'third_party/gloo' 2024-08-20T21:51:19.2099545Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-20T21:51:19.2117810Z Entering 'third_party/googletest' 2024-08-20T21:51:19.2175956Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:51:19.2194207Z Entering 'third_party/ideep' 2024-08-20T21:51:19.2252745Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-20T21:51:19.2269212Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T21:51:19.2327485Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-20T21:51:19.2354655Z Entering 'third_party/ittapi' 2024-08-20T21:51:19.2411715Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-20T21:51:19.2430006Z Entering 'third_party/kineto' 2024-08-20T21:51:19.2489066Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-20T21:51:19.2506931Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:51:19.2567742Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-20T21:51:19.2585174Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:51:19.2644551Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-20T21:51:19.2663966Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:51:19.2722372Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-20T21:51:19.2741781Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:51:19.2801033Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-20T21:51:19.2819848Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:51:19.2879950Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-20T21:51:19.2896581Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:51:19.2957402Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-20T21:51:19.2976915Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:51:19.3037868Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-20T21:51:19.3057120Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:51:19.3116361Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:51:19.3134677Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:51:19.3194083Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-20T21:51:19.3213003Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:51:19.3272304Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-20T21:51:19.3292267Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:51:19.3351768Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-20T21:51:19.3369526Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:51:19.3427613Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-20T21:51:19.3447859Z Entering 'third_party/mimalloc' 2024-08-20T21:51:19.3505617Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-20T21:51:19.3523976Z Entering 'third_party/nccl/nccl' 2024-08-20T21:51:19.3582908Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-20T21:51:19.3601043Z Entering 'third_party/nlohmann' 2024-08-20T21:51:19.3660233Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-20T21:51:19.3679646Z Entering 'third_party/onnx' 2024-08-20T21:51:19.3739279Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-20T21:51:19.3774743Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T21:51:19.3832801Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-20T21:51:19.3851450Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T21:51:19.3909425Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-20T21:51:19.3930188Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T21:51:19.3989956Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-20T21:51:19.4009168Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:51:19.4068601Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-20T21:51:19.4086278Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:51:19.4144240Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:51:19.4161967Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:51:19.4220330Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-20T21:51:19.4238820Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:51:19.4296098Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-20T21:51:19.4315351Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:51:19.4374513Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-20T21:51:19.4392128Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:51:19.4450468Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-20T21:51:19.4468030Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:51:19.4526386Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-20T21:51:19.4544656Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:51:19.4602645Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-20T21:51:19.4622433Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:51:19.4681324Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-20T21:51:19.4700604Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:51:19.4759771Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-20T21:51:19.4799891Z Entering 'third_party/pocketfft' 2024-08-20T21:51:19.4858486Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-20T21:51:19.4876029Z Entering 'third_party/protobuf' 2024-08-20T21:51:19.4935273Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-20T21:51:19.4957018Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:51:19.5015250Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-20T21:51:19.5033334Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T21:51:19.5092509Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:51:19.5111824Z Entering 'third_party/psimd' 2024-08-20T21:51:19.5170121Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-20T21:51:19.5187616Z Entering 'third_party/pthreadpool' 2024-08-20T21:51:19.5245995Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-20T21:51:19.5263529Z Entering 'third_party/pybind11' 2024-08-20T21:51:19.5321132Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-20T21:51:19.5339821Z Entering 'third_party/python-peachpy' 2024-08-20T21:51:19.5398502Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-20T21:51:19.5416589Z Entering 'third_party/sleef' 2024-08-20T21:51:19.5475018Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-20T21:51:19.5493599Z Entering 'third_party/tensorpipe' 2024-08-20T21:51:19.5554408Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-20T21:51:19.5571742Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:51:19.5629181Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:51:19.5647560Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:51:19.5704859Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-20T21:51:19.5722396Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:51:19.5781399Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-20T21:51:19.5799321Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:51:19.5859056Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-20T21:51:19.5875611Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:51:19.5934509Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-20T21:51:19.6993866Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-20T21:51:19.7334769Z Entering 'android/libs/fbjni' 2024-08-20T21:51:19.7382908Z Entering 'third_party/FP16' 2024-08-20T21:51:19.7430194Z Entering 'third_party/FXdiv' 2024-08-20T21:51:19.7478723Z Entering 'third_party/NNPACK' 2024-08-20T21:51:19.7525981Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T21:51:19.7575471Z Entering 'third_party/XNNPACK' 2024-08-20T21:51:19.7639731Z Entering 'third_party/benchmark' 2024-08-20T21:51:19.7687468Z Entering 'third_party/cpp-httplib' 2024-08-20T21:51:19.7734872Z Entering 'third_party/cpuinfo' 2024-08-20T21:51:19.7784052Z Entering 'third_party/cudnn_frontend' 2024-08-20T21:51:19.7832250Z Entering 'third_party/cutlass' 2024-08-20T21:51:19.7889017Z Entering 'third_party/eigen' 2024-08-20T21:51:19.7941470Z Entering 'third_party/fbgemm' 2024-08-20T21:51:19.7989453Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:51:19.8036577Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:51:19.8084988Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:51:19.8140723Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:51:19.8187399Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:51:19.8236045Z Entering 'third_party/flatbuffers' 2024-08-20T21:51:19.8286885Z Entering 'third_party/fmt' 2024-08-20T21:51:19.8334334Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:51:19.8382768Z Entering 'third_party/gloo' 2024-08-20T21:51:19.8430198Z Entering 'third_party/googletest' 2024-08-20T21:51:19.8479015Z Entering 'third_party/ideep' 2024-08-20T21:51:19.8525508Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T21:51:19.8580589Z Entering 'third_party/ittapi' 2024-08-20T21:51:19.8628116Z Entering 'third_party/kineto' 2024-08-20T21:51:19.8676360Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:51:19.8723549Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:51:19.8773158Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:51:19.8821092Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:51:19.8869731Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:51:19.8916395Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:51:19.8968517Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:51:19.9015626Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:51:19.9064315Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:51:19.9112588Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:51:19.9163322Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:51:19.9210202Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:51:19.9258877Z Entering 'third_party/mimalloc' 2024-08-20T21:51:19.9306760Z Entering 'third_party/nccl/nccl' 2024-08-20T21:51:19.9354887Z Entering 'third_party/nlohmann' 2024-08-20T21:51:19.9403642Z Entering 'third_party/onnx' 2024-08-20T21:51:19.9468381Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T21:51:19.9515987Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T21:51:19.9567574Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T21:51:19.9617777Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:51:19.9665511Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:51:19.9711885Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:51:19.9760287Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:51:19.9809142Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:51:19.9857126Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:51:19.9905069Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:51:19.9952158Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:51:20.0001461Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:51:20.0051116Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:51:20.0120369Z Entering 'third_party/pocketfft' 2024-08-20T21:51:20.0169138Z Entering 'third_party/protobuf' 2024-08-20T21:51:20.0220444Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:51:20.0268631Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T21:51:20.0317780Z Entering 'third_party/psimd' 2024-08-20T21:51:20.0367706Z Entering 'third_party/pthreadpool' 2024-08-20T21:51:20.0419229Z Entering 'third_party/pybind11' 2024-08-20T21:51:20.0469850Z Entering 'third_party/python-peachpy' 2024-08-20T21:51:20.0517348Z Entering 'third_party/sleef' 2024-08-20T21:51:20.0568139Z Entering 'third_party/tensorpipe' 2024-08-20T21:51:20.0615771Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:51:20.0663718Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:51:20.0710434Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:51:20.0757616Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:51:20.0803306Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:51:20.0869825Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-20T21:51:20.1210805Z Entering 'android/libs/fbjni' 2024-08-20T21:51:20.1258220Z Entering 'third_party/FP16' 2024-08-20T21:51:20.1305636Z Entering 'third_party/FXdiv' 2024-08-20T21:51:20.1353035Z Entering 'third_party/NNPACK' 2024-08-20T21:51:20.1400409Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T21:51:20.1449556Z Entering 'third_party/XNNPACK' 2024-08-20T21:51:20.1515493Z Entering 'third_party/benchmark' 2024-08-20T21:51:20.1563812Z Entering 'third_party/cpp-httplib' 2024-08-20T21:51:20.1611889Z Entering 'third_party/cpuinfo' 2024-08-20T21:51:20.1660858Z Entering 'third_party/cudnn_frontend' 2024-08-20T21:51:20.1708671Z Entering 'third_party/cutlass' 2024-08-20T21:51:20.1766444Z Entering 'third_party/eigen' 2024-08-20T21:51:20.1816728Z Entering 'third_party/fbgemm' 2024-08-20T21:51:20.1866373Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:51:20.1912983Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:51:20.1961568Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:51:20.2015859Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:51:20.2064537Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:51:20.2112798Z Entering 'third_party/flatbuffers' 2024-08-20T21:51:20.2163367Z Entering 'third_party/fmt' 2024-08-20T21:51:20.2210882Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:51:20.2259600Z Entering 'third_party/gloo' 2024-08-20T21:51:20.2307217Z Entering 'third_party/googletest' 2024-08-20T21:51:20.2356131Z Entering 'third_party/ideep' 2024-08-20T21:51:20.2402182Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T21:51:20.2458481Z Entering 'third_party/ittapi' 2024-08-20T21:51:20.2505861Z Entering 'third_party/kineto' 2024-08-20T21:51:20.2553989Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:51:20.2601766Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:51:20.2651435Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:51:20.2699034Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:51:20.2747616Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:51:20.2794070Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:51:20.2845214Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:51:20.2892577Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:51:20.2940802Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:51:20.2989097Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:51:20.3039286Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:51:20.3086611Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:51:20.3135352Z Entering 'third_party/mimalloc' 2024-08-20T21:51:20.3184235Z Entering 'third_party/nccl/nccl' 2024-08-20T21:51:20.3232108Z Entering 'third_party/nlohmann' 2024-08-20T21:51:20.3281793Z Entering 'third_party/onnx' 2024-08-20T21:51:20.3348817Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T21:51:20.3396252Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T21:51:20.3448006Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T21:51:20.3497194Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:51:20.3545191Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:51:20.3592826Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:51:20.3639579Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:51:20.3687482Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:51:20.3734098Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:51:20.3780853Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:51:20.3827219Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:51:20.3877313Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:51:20.3926251Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:51:20.3996237Z Entering 'third_party/pocketfft' 2024-08-20T21:51:20.4045081Z Entering 'third_party/protobuf' 2024-08-20T21:51:20.4096492Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:51:20.4143587Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T21:51:20.4192151Z Entering 'third_party/psimd' 2024-08-20T21:51:20.4240003Z Entering 'third_party/pthreadpool' 2024-08-20T21:51:20.4287114Z Entering 'third_party/pybind11' 2024-08-20T21:51:20.4335535Z Entering 'third_party/python-peachpy' 2024-08-20T21:51:20.4382819Z Entering 'third_party/sleef' 2024-08-20T21:51:20.4429875Z Entering 'third_party/tensorpipe' 2024-08-20T21:51:20.4478962Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:51:20.4525949Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:51:20.4574094Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:51:20.4620628Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:51:20.4668249Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:51:20.4729985Z ##[endgroup] 2024-08-20T21:51:20.4774385Z [command]/usr/bin/git log -1 --format='%H' 2024-08-20T21:51:20.4804432Z '40ec5f6ddd9787aca0449b24128343ff4c4a88b3' 2024-08-20T21:51:20.5040076Z Prepare all required actions 2024-08-20T21:51:20.5040673Z Getting action download info 2024-08-20T21:51:20.6374253Z ##[group]Run ./.github/actions/setup-linux 2024-08-20T21:51:20.6374781Z env: 2024-08-20T21:51:20.6375118Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:20.6375524Z ##[endgroup] 2024-08-20T21:51:20.6457648Z ##[group]Run set -euo pipefail 2024-08-20T21:51:20.6480340Z set -euo pipefail 2024-08-20T21:51:20.6480932Z function get_ec2_metadata() { 2024-08-20T21:51:20.6481655Z  # Pulled from instance metadata endpoint for EC2 2024-08-20T21:51:20.6482842Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-20T21:51:20.6484123Z  category=$1 2024-08-20T21:51:20.6484832Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-20T21:51:20.6485686Z  runner_name_str=i-050c06cb9ff8794b9 2024-08-20T21:51:20.6486347Z  if [[ -f /.inarc ]]; then 2024-08-20T21:51:20.6487045Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-20T21:51:20.6487767Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-20T21:51:20.6488636Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-20T21:51:20.6489394Z  else 2024-08-20T21:51:20.6489989Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-20T21:51:20.6490707Z  fi 2024-08-20T21:51:20.6491047Z } 2024-08-20T21:51:20.6491465Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-20T21:51:20.6492198Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-20T21:51:20.6493029Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-20T21:51:20.6493719Z echo "system info $(uname -a)" 2024-08-20T21:51:20.6503336Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:20.6503962Z env: 2024-08-20T21:51:20.6504277Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:20.6504699Z ##[endgroup] 2024-08-20T21:51:20.6618265Z ami-id: ami-06c68f701d8090592 2024-08-20T21:51:20.6682446Z instance-id: i-050c06cb9ff8794b9 2024-08-20T21:51:20.6743632Z instance-type: g3.8xlarge 2024-08-20T21:51:20.6757121Z system info Linux ip-10-0-37-115.ec2.internal 6.1.94-99.176.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-20T21:51:20.6786224Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T21:51:20.6787358Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T21:51:20.6794099Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:20.6794719Z env: 2024-08-20T21:51:20.6795033Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:20.6795452Z ##[endgroup] 2024-08-20T21:51:20.6878904Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-20T21:51:20.6879661Z if systemctl is-active --quiet docker; then 2024-08-20T21:51:20.6880336Z  echo "Docker daemon is running..."; 2024-08-20T21:51:20.6880894Z else 2024-08-20T21:51:20.6881494Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-20T21:51:20.6882227Z fi 2024-08-20T21:51:20.6888506Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:20.6889124Z env: 2024-08-20T21:51:20.6889452Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:20.6889858Z ##[endgroup] 2024-08-20T21:51:20.6971373Z Docker daemon is running... 2024-08-20T21:51:20.7029893Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T21:51:20.7030609Z with: 2024-08-20T21:51:20.7030979Z shell: bash 2024-08-20T21:51:20.7031319Z timeout_minutes: 5 2024-08-20T21:51:20.7031705Z max_attempts: 3 2024-08-20T21:51:20.7032082Z retry_wait_seconds: 30 2024-08-20T21:51:20.7036125Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-08-20T21:51:20.7040601Z polling_interval_seconds: 1 2024-08-20T21:51:20.7041055Z warning_on_retry: true 2024-08-20T21:51:20.7041477Z continue_on_error: false 2024-08-20T21:51:20.7042050Z env: 2024-08-20T21:51:20.7042354Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:20.7042788Z AWS_RETRY_MODE: standard 2024-08-20T21:51:20.7043209Z AWS_MAX_ATTEMPTS: 5 2024-08-20T21:51:20.7043604Z AWS_DEFAULT_REGION: us-east-1 2024-08-20T21:51:20.7044065Z ##[endgroup] 2024-08-20T21:51:22.1438635Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T21:51:22.1439694Z Configure a credential helper to remove this warning. See 2024-08-20T21:51:22.1440800Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T21:51:22.1441494Z 2024-08-20T21:51:22.1441644Z Login Succeeded 2024-08-20T21:51:22.7796254Z Command completed after 1 attempt(s). 2024-08-20T21:51:22.7857424Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T21:51:22.7858329Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T21:51:22.7859150Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T21:51:22.7867081Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:22.7867691Z env: 2024-08-20T21:51:22.7868023Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:22.7868447Z ##[endgroup] 2024-08-20T21:51:22.7965994Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T21:51:22.7966955Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T21:51:22.7967695Z # shellcheck disable=SC2046 2024-08-20T21:51:22.7968263Z docker stop $(docker ps -q) || true 2024-08-20T21:51:22.7968836Z # Prune all of the docker images 2024-08-20T21:51:22.7969392Z docker system prune -af 2024-08-20T21:51:22.7975770Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:22.7976373Z env: 2024-08-20T21:51:22.7976708Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:22.7977133Z ##[endgroup] 2024-08-20T21:51:22.8323971Z "docker stop" requires at least 1 argument. 2024-08-20T21:51:22.8324831Z See 'docker stop --help'. 2024-08-20T21:51:22.8325151Z 2024-08-20T21:51:22.8325428Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-20T21:51:22.8325913Z 2024-08-20T21:51:22.8326095Z Stop one or more running containers 2024-08-20T21:51:22.8524419Z Total reclaimed space: 0B 2024-08-20T21:51:22.8569889Z ##[group]Run set +e 2024-08-20T21:51:22.8570356Z set +e 2024-08-20T21:51:22.8570707Z set -x 2024-08-20T21:51:22.8571063Z  2024-08-20T21:51:22.8571456Z PT_DOMAIN=download.pytorch.org 2024-08-20T21:51:22.8572439Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-20T21:51:22.8573810Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-20T21:51:22.8574760Z # one is returned at random 2024-08-20T21:51:22.8575416Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-20T21:51:22.8576068Z  2024-08-20T21:51:22.8576447Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T21:51:22.8577200Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-20T21:51:22.8578357Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-20T21:51:22.8579080Z  2024-08-20T21:51:22.8579470Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T21:51:22.8580140Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-20T21:51:22.8580774Z  exit 1 2024-08-20T21:51:22.8581150Z  fi 2024-08-20T21:51:22.8581471Z fi 2024-08-20T21:51:22.8581804Z  2024-08-20T21:51:22.8582240Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-20T21:51:22.8582864Z  # Clean up any old records first 2024-08-20T21:51:22.8583498Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-20T21:51:22.8584057Z fi 2024-08-20T21:51:22.8584525Z  2024-08-20T21:51:22.8585052Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-20T21:51:22.8585735Z cat /etc/hosts 2024-08-20T21:51:22.8593136Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:22.8593760Z env: 2024-08-20T21:51:22.8594101Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:22.8594513Z ##[endgroup] 2024-08-20T21:51:22.8624342Z + PT_DOMAIN=download.pytorch.org 2024-08-20T21:51:22.8630596Z ++ dig -4 +short download.pytorch.org 2024-08-20T21:51:22.8631500Z ++ tail -n1 2024-08-20T21:51:22.8888476Z + RESOLVED_IP=18.160.10.22 2024-08-20T21:51:22.8889063Z + '[' -z 18.160.10.22 ']' 2024-08-20T21:51:22.8889582Z + grep -r download.pytorch.org /etc/hosts 2024-08-20T21:51:22.8902435Z 18.165.98.97 download.pytorch.org 2024-08-20T21:51:22.8904201Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-08-20T21:51:23.2056359Z + echo '18.160.10.22 download.pytorch.org' 2024-08-20T21:51:23.2057072Z + sudo tee -a /etc/hosts 2024-08-20T21:51:23.2596446Z 18.160.10.22 download.pytorch.org 2024-08-20T21:51:23.2615476Z + cat /etc/hosts 2024-08-20T21:51:23.2625087Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-20T21:51:23.2642444Z ::1 localhost6 localhost6.localdomain6 2024-08-20T21:51:23.2643200Z 18.160.10.22 download.pytorch.org 2024-08-20T21:51:23.2848528Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-20T21:51:23.2849334Z with: 2024-08-20T21:51:23.2850555Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:23.2851972Z docker-build-dir: .ci/docker 2024-08-20T21:51:23.2852438Z working-directory: . 2024-08-20T21:51:23.2853005Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:23.2853666Z force-push: false 2024-08-20T21:51:23.2854035Z env: 2024-08-20T21:51:23.2854341Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:23.2854780Z ##[endgroup] 2024-08-20T21:51:23.2881599Z ##[group]Run set -ex 2024-08-20T21:51:23.2882049Z set -ex 2024-08-20T21:51:23.2882411Z  2024-08-20T21:51:23.2883095Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-20T21:51:23.2884396Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-20T21:51:23.2885416Z # job could then download the pre-built image as usual 2024-08-20T21:51:23.2886345Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-20T21:51:23.2887194Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:23.2887992Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:23.2888680Z  2024-08-20T21:51:23.2889320Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-20T21:51:23.2890111Z  exit 0 2024-08-20T21:51:23.2890474Z else 2024-08-20T21:51:23.2890924Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:23.2891486Z fi 2024-08-20T21:51:23.2891807Z  2024-08-20T21:51:23.2892388Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-20T21:51:23.2893469Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-20T21:51:23.2894439Z  # use it as it is, but first let's extract the tag 2024-08-20T21:51:23.2895332Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-20T21:51:23.2896247Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:23.2897130Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:23.2897816Z else 2024-08-20T21:51:23.2898354Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-20T21:51:23.2899410Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:23.2900527Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:23.2901479Z fi 2024-08-20T21:51:23.2910366Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:23.2910982Z env: 2024-08-20T21:51:23.2911315Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:23.2911757Z REPO_NAME: pytorch 2024-08-20T21:51:23.2913011Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:23.2914393Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T21:51:23.2915011Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:23.2915661Z ##[endgroup] 2024-08-20T21:51:23.2946580Z + [[ ! -d .ci/docker ]] 2024-08-20T21:51:23.2947481Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-20T21:51:23.2947988Z + echo skip=false 2024-08-20T21:51:23.2950180Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-20T21:51:23.2955393Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:23.2956912Z ++ awk -F '[:,]' '{print $2}' 2024-08-20T21:51:23.2978850Z + DOCKER_TAG=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:23.2980378Z + echo docker-tag=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:23.2982759Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:23.3018905Z ##[group]Run set +e 2024-08-20T21:51:23.3019346Z set +e 2024-08-20T21:51:23.3019724Z set -x 2024-08-20T21:51:23.3020056Z  2024-08-20T21:51:23.3020387Z login() { 2024-08-20T21:51:23.3021209Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T21:51:23.3022116Z } 2024-08-20T21:51:23.3022428Z  2024-08-20T21:51:23.3022768Z retry () { 2024-08-20T21:51:23.3023234Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T21:51:23.3023786Z } 2024-08-20T21:51:23.3024114Z  2024-08-20T21:51:23.3024475Z retry login "${DOCKER_REGISTRY}" 2024-08-20T21:51:23.3024987Z  2024-08-20T21:51:23.3025566Z # Check if image already exists, if it does then skip building it 2024-08-20T21:51:23.3026451Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-20T21:51:23.3027055Z  exit 0 2024-08-20T21:51:23.3027416Z fi 2024-08-20T21:51:23.3027745Z  2024-08-20T21:51:23.3028347Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-20T21:51:23.3029399Z # be empty. The default action would be to continue rebuild the image 2024-08-20T21:51:23.3030325Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-20T21:51:23.3031224Z  # if we're on the base branch then use the parent commit 2024-08-20T21:51:23.3031966Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-20T21:51:23.3032502Z else 2024-08-20T21:51:23.3033059Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-20T21:51:23.3033929Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-20T21:51:23.3034554Z fi 2024-08-20T21:51:23.3034866Z  2024-08-20T21:51:23.3035246Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-20T21:51:23.3035875Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:23.3036437Z  2024-08-20T21:51:23.3037772Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-20T21:51:23.3038974Z  exit 0 2024-08-20T21:51:23.3039338Z fi 2024-08-20T21:51:23.3039652Z  2024-08-20T21:51:23.3040191Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-20T21:51:23.3041430Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-20T21:51:23.3042430Z  exit 1 2024-08-20T21:51:23.3042794Z fi 2024-08-20T21:51:23.3043125Z  2024-08-20T21:51:23.3043710Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-20T21:51:23.3044915Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-20T21:51:23.3046011Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-20T21:51:23.3047234Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-20T21:51:23.3048625Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-20T21:51:23.3049440Z fi 2024-08-20T21:51:23.3049774Z  2024-08-20T21:51:23.3050338Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:23.3056999Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:23.3057623Z env: 2024-08-20T21:51:23.3057934Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:23.3058381Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T21:51:23.3058951Z BASE_REVISION: 91f3d614142df02f619e44a68e3d9e0dfeba49ec 2024-08-20T21:51:23.3060390Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:23.3061826Z DOCKER_TAG: f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:23.3062590Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:23.3063233Z ##[endgroup] 2024-08-20T21:51:23.3091369Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:23.3092497Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:23.3093923Z + aws ecr get-login-password --region us-east-1 2024-08-20T21:51:23.3096047Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:23.9705417Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T21:51:23.9707354Z Configure a credential helper to remove this warning. See 2024-08-20T21:51:23.9708461Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T21:51:23.9709150Z 2024-08-20T21:51:23.9709605Z Login Succeeded 2024-08-20T21:51:23.9720215Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:24.2065205Z { 2024-08-20T21:51:24.2065973Z "schemaVersion": 2, 2024-08-20T21:51:24.2067035Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-20T21:51:24.2111514Z "config": { 2024-08-20T21:51:24.2112464Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-20T21:51:24.2113177Z "size": 47467, 2024-08-20T21:51:24.2113881Z "digest": "sha256:35f4945dfeab4592c8657de7107f96a9fd9a51999f582a0064937246789c0825" 2024-08-20T21:51:24.2114629Z }, 2024-08-20T21:51:24.2114943Z "layers": [ 2024-08-20T21:51:24.2115496Z + exit 0 2024-08-20T21:51:24.2116076Z { 2024-08-20T21:51:24.2116943Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2117651Z "size": 28580681, 2024-08-20T21:51:24.2118345Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-08-20T21:51:24.2119131Z }, 2024-08-20T21:51:24.2119438Z { 2024-08-20T21:51:24.2119953Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2120932Z "size": 7943353, 2024-08-20T21:51:24.2121695Z "digest": "sha256:09d415c238d76b32a7ea4a6e6add9542db9a5641f7f183af70aae185d0709e58" 2024-08-20T21:51:24.2122459Z }, 2024-08-20T21:51:24.2122709Z { 2024-08-20T21:51:24.2123248Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2123942Z "size": 56226389, 2024-08-20T21:51:24.2124616Z "digest": "sha256:9fe6e2e61518cba6844870c03b285737daec35e62baf25ae7744629ed3a7b470" 2024-08-20T21:51:24.2125385Z }, 2024-08-20T21:51:24.2125676Z { 2024-08-20T21:51:24.2126197Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2126889Z "size": 187, 2024-08-20T21:51:24.2127549Z "digest": "sha256:41f16248e682693ff20b3032c1d5e5541cc87c5af898ae2ff9b24d2940e59100" 2024-08-20T21:51:24.2128295Z }, 2024-08-20T21:51:24.2128587Z { 2024-08-20T21:51:24.2129122Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2129808Z "size": 6885, 2024-08-20T21:51:24.2130486Z "digest": "sha256:95d7b781703928cf3c4eece39d800cccb76728c375fedf51ecd83833fb25e458" 2024-08-20T21:51:24.2131317Z }, 2024-08-20T21:51:24.2131596Z { 2024-08-20T21:51:24.2132292Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2133017Z "size": 1383662475, 2024-08-20T21:51:24.2133680Z "digest": "sha256:8f6c9048534734f4c873935293b7296225846ceb31c1a158400a67ea170dde7f" 2024-08-20T21:51:24.2134421Z }, 2024-08-20T21:51:24.2134709Z { 2024-08-20T21:51:24.2135222Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2135916Z "size": 62282, 2024-08-20T21:51:24.2136570Z "digest": "sha256:ab17245097e491b9368790714f9d90ed447bf0973bd677cfe6f2456d62b72a13" 2024-08-20T21:51:24.2137852Z }, 2024-08-20T21:51:24.2138156Z { 2024-08-20T21:51:24.2138691Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2139528Z "size": 1684, 2024-08-20T21:51:24.2140214Z "digest": "sha256:dfecd7e9912b76ed460b8edd5a85f1943666e38a973ab5458177cf2c7c3110e3" 2024-08-20T21:51:24.2140993Z }, 2024-08-20T21:51:24.2141270Z { 2024-08-20T21:51:24.2141800Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2142498Z "size": 1522, 2024-08-20T21:51:24.2143151Z "digest": "sha256:464a8f74544589bf7b57f9a4cadcb6681e5ed00758f6c35025e691df4e88e890" 2024-08-20T21:51:24.2143919Z }, 2024-08-20T21:51:24.2144211Z { 2024-08-20T21:51:24.2144723Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2145415Z "size": 2445884909, 2024-08-20T21:51:24.2146097Z "digest": "sha256:d67c111fa58886d8904335b5122039816c8e51af5844cb6fe4e14154d9a7c016" 2024-08-20T21:51:24.2146831Z }, 2024-08-20T21:51:24.2147123Z { 2024-08-20T21:51:24.2147654Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2148323Z "size": 85882, 2024-08-20T21:51:24.2149003Z "digest": "sha256:0723abde19c09431db80b5d0f0e003541f6232c2da5c4f35c635c861875937ae" 2024-08-20T21:51:24.2149770Z }, 2024-08-20T21:51:24.2150047Z { 2024-08-20T21:51:24.2150576Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2151267Z "size": 1822, 2024-08-20T21:51:24.2151935Z "digest": "sha256:72d2b53f7a0105206afbdce791a85d403e2413ac60a2ea3fec0bb057b69925ad" 2024-08-20T21:51:24.2152714Z }, 2024-08-20T21:51:24.2153005Z { 2024-08-20T21:51:24.2153521Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2154210Z "size": 654861296, 2024-08-20T21:51:24.2154927Z "digest": "sha256:dbc6fccd9c70404c8728c8d319bbe9cd5ef59e56b27d0a3ff9385dc2007420e2" 2024-08-20T21:51:24.2155702Z }, 2024-08-20T21:51:24.2155995Z { 2024-08-20T21:51:24.2156524Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2157199Z "size": 544, 2024-08-20T21:51:24.2157869Z "digest": "sha256:ff6a6ddc73105bc6954a81b0d5ed486362bfd4399595a378a7c6594a332069a3" 2024-08-20T21:51:24.2160167Z }, 2024-08-20T21:51:24.2160453Z { 2024-08-20T21:51:24.2160994Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2161694Z "size": 1258, 2024-08-20T21:51:24.2162373Z "digest": "sha256:fac2b1b246ed3a00ba10fdfe9dc50559f8978915b204b903bc5fb47908fb5bd5" 2024-08-20T21:51:24.2163160Z }, 2024-08-20T21:51:24.2163455Z { 2024-08-20T21:51:24.2163974Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2164671Z "size": 484, 2024-08-20T21:51:24.2165348Z "digest": "sha256:bc574f79516b484ef9cff3a621d4e8f6dd0ced47a0c7de6959a2213b313cd871" 2024-08-20T21:51:24.2166110Z }, 2024-08-20T21:51:24.2166409Z { 2024-08-20T21:51:24.2166941Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2167614Z "size": 91712209, 2024-08-20T21:51:24.2168317Z "digest": "sha256:c23d42adcadecf7e1d93290ebf26fe3936841777be574d8e2a4c71719d43f05c" 2024-08-20T21:51:24.2169108Z }, 2024-08-20T21:51:24.2169384Z { 2024-08-20T21:51:24.2169917Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2170609Z "size": 3390, 2024-08-20T21:51:24.2171416Z "digest": "sha256:59b0a331db9f6d35dab3eef54ae05d703663a7cfe6ea9ec139fdc8c2221834da" 2024-08-20T21:51:24.2172229Z }, 2024-08-20T21:51:24.2172527Z { 2024-08-20T21:51:24.2173048Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2173743Z "size": 1909, 2024-08-20T21:51:24.2174424Z "digest": "sha256:478a9a2bdefc98de190384a54d89524fb0d691b7155ffba4df69a35a50e4afae" 2024-08-20T21:51:24.2175187Z }, 2024-08-20T21:51:24.2175480Z { 2024-08-20T21:51:24.2176010Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2176685Z "size": 699, 2024-08-20T21:51:24.2177355Z "digest": "sha256:28e54511f1b116a9d632caa0c4be2cd1b6ca7542f815560ec598b81d6d956d31" 2024-08-20T21:51:24.2178132Z }, 2024-08-20T21:51:24.2178407Z { 2024-08-20T21:51:24.2178939Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2179817Z "size": 2891829542, 2024-08-20T21:51:24.2180500Z "digest": "sha256:5247f18de8815b263c1380f613ce6f8b836db6766c3b137ef30b18fac324a4a3" 2024-08-20T21:51:24.2181273Z }, 2024-08-20T21:51:24.2181565Z { 2024-08-20T21:51:24.2182079Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2182768Z "size": 380, 2024-08-20T21:51:24.2183388Z "digest": "sha256:38760708e2d79494fe04b599b375145b8968797e5a063609cb007de5037ff0cf" 2024-08-20T21:51:24.2184116Z }, 2024-08-20T21:51:24.2184405Z { 2024-08-20T21:51:24.2184930Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2185599Z "size": 12876, 2024-08-20T21:51:24.2186292Z "digest": "sha256:863dbde6ed0c5edba3f86f2907198ab33eed3dd4eed51375189fd722a66b051a" 2024-08-20T21:51:24.2187078Z }, 2024-08-20T21:51:24.2187360Z { 2024-08-20T21:51:24.2187889Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2188579Z "size": 805, 2024-08-20T21:51:24.2189237Z "digest": "sha256:f283ab568a00b37f29b9d76f77c2f99a05b336ee3b1ffff8577b6bd7df2e95f3" 2024-08-20T21:51:24.2190036Z }, 2024-08-20T21:51:24.2190314Z { 2024-08-20T21:51:24.2190854Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2191544Z "size": 106, 2024-08-20T21:51:24.2192187Z "digest": "sha256:3e39f30b1d5c27e19c987aa8f8fbb36778a68cd534872e718053f71997457dab" 2024-08-20T21:51:24.2192957Z }, 2024-08-20T21:51:24.2193248Z { 2024-08-20T21:51:24.2193758Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2194446Z "size": 503, 2024-08-20T21:51:24.2195096Z "digest": "sha256:521c3d72534c80920fa11f85e7276b2130851936dba93947a4dc647eaf8855ba" 2024-08-20T21:51:24.2195829Z }, 2024-08-20T21:51:24.2196120Z { 2024-08-20T21:51:24.2196754Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2197429Z "size": 121477349, 2024-08-20T21:51:24.2198130Z "digest": "sha256:a975a4480a6d2abb7c44d3b8fe0e198c4d2666336f05f12936968ac5ac02fbe3" 2024-08-20T21:51:24.2198900Z }, 2024-08-20T21:51:24.2199175Z { 2024-08-20T21:51:24.2199711Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2200399Z "size": 109, 2024-08-20T21:51:24.2201045Z "digest": "sha256:1f2c278212ad728862a858a586ea5b666cea7384cb3b51900fc12afdd1fb0508" 2024-08-20T21:51:24.2201810Z }, 2024-08-20T21:51:24.2202100Z { 2024-08-20T21:51:24.2202612Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2203304Z "size": 489, 2024-08-20T21:51:24.2203971Z "digest": "sha256:b721dd867e80edbea20e5691ce6254378a75d20d38436ffc551dbb91453aefc5" 2024-08-20T21:51:24.2204723Z }, 2024-08-20T21:51:24.2205015Z { 2024-08-20T21:51:24.2205544Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2206224Z "size": 297, 2024-08-20T21:51:24.2206896Z "digest": "sha256:7a74b7ad649d3f0cc23bef1d20005cc436d2f7d478b2e81519d40ad7aa8ea251" 2024-08-20T21:51:24.2207673Z }, 2024-08-20T21:51:24.2207954Z { 2024-08-20T21:51:24.2208576Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2209280Z "size": 103, 2024-08-20T21:51:24.2209949Z "digest": "sha256:b53e1ad1deec07d1a5f093f57f5048627e112a68a5f512f9d9d229d0a91b9b2b" 2024-08-20T21:51:24.2210730Z }, 2024-08-20T21:51:24.2211018Z { 2024-08-20T21:51:24.2211532Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2212221Z "size": 1472, 2024-08-20T21:51:24.2212900Z "digest": "sha256:016dc3faa7efaa82e2cb84df6d34535024761fcfe67540e7e1f82686190841ee" 2024-08-20T21:51:24.2213664Z }, 2024-08-20T21:51:24.2213957Z { 2024-08-20T21:51:24.2214485Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2215170Z "size": 594729536, 2024-08-20T21:51:24.2215861Z "digest": "sha256:384dce3a4ba2bb506140986d6a45133d58345f055cbf72d018cff59716419012" 2024-08-20T21:51:24.2216623Z }, 2024-08-20T21:51:24.2216900Z { 2024-08-20T21:51:24.2217419Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2218114Z "size": 164, 2024-08-20T21:51:24.2218760Z "digest": "sha256:75461ad9f7bb4e53c4a6bd26539108b012c321304219c48a8852353fc2260613" 2024-08-20T21:51:24.2219518Z }, 2024-08-20T21:51:24.2219808Z { 2024-08-20T21:51:24.2220320Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2221016Z "size": 562, 2024-08-20T21:51:24.2221687Z "digest": "sha256:eb2062102d1137b2c485af726affb99a0f62f1a159a6c6ff0fbc936dac18596d" 2024-08-20T21:51:24.2222442Z }, 2024-08-20T21:51:24.2222733Z { 2024-08-20T21:51:24.2223262Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2223939Z "size": 35806203, 2024-08-20T21:51:24.2224634Z "digest": "sha256:3cc17f105239b758d0e1ff69aa738939b0a89561da055484a0a9e2602e5a543f" 2024-08-20T21:51:24.2225391Z }, 2024-08-20T21:51:24.2225665Z { 2024-08-20T21:51:24.2226195Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2226881Z "size": 104, 2024-08-20T21:51:24.2227537Z "digest": "sha256:585501be07997851ac265aba0a5b71e2c2e9afb4cc9c30da46cd9f2f5ffe063d" 2024-08-20T21:51:24.2228305Z }, 2024-08-20T21:51:24.2228595Z { 2024-08-20T21:51:24.2229107Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2229795Z "size": 424, 2024-08-20T21:51:24.2230486Z "digest": "sha256:6c49da68c9b4f370ed5ecfbb742ff5cad0ccc3d855e71fc10fdf603cabe9d87b" 2024-08-20T21:51:24.2231319Z }, 2024-08-20T21:51:24.2231617Z { 2024-08-20T21:51:24.2232151Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2232829Z "size": 20262169, 2024-08-20T21:51:24.2233500Z "digest": "sha256:497539689260668c0d109b8e22341ac13b1c482fd9c0b2e8371b8c88bd9fc766" 2024-08-20T21:51:24.2234401Z }, 2024-08-20T21:51:24.2234681Z { 2024-08-20T21:51:24.2235221Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2235917Z "size": 438, 2024-08-20T21:51:24.2236580Z "digest": "sha256:5843bb8b81bb68867db16a80d2ebb31396b14b3fa08b419ce9ab07b0791eb662" 2024-08-20T21:51:24.2237879Z }, 2024-08-20T21:51:24.2238185Z { 2024-08-20T21:51:24.2238705Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2239403Z "size": 699, 2024-08-20T21:51:24.2240075Z "digest": "sha256:28e54511f1b116a9d632caa0c4be2cd1b6ca7542f815560ec598b81d6d956d31" 2024-08-20T21:51:24.2240825Z }, 2024-08-20T21:51:24.2241119Z { 2024-08-20T21:51:24.2241650Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2242319Z "size": 141, 2024-08-20T21:51:24.2242969Z "digest": "sha256:222174e4e2b25c47241345129f9ec8c9c219ef78391d65f6089d89339be6ddb7" 2024-08-20T21:51:24.2243727Z }, 2024-08-20T21:51:24.2244004Z { 2024-08-20T21:51:24.2244535Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2245226Z "size": 135, 2024-08-20T21:51:24.2246022Z "digest": "sha256:8d55064a9f207c06db9da40e295a3430ac0b288cfdf6968903dddb487789b380" 2024-08-20T21:51:24.2246818Z }, 2024-08-20T21:51:24.2247116Z { 2024-08-20T21:51:24.2247640Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2248334Z "size": 32, 2024-08-20T21:51:24.2249018Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:51:24.2249778Z }, 2024-08-20T21:51:24.2250071Z { 2024-08-20T21:51:24.2250605Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2251277Z "size": 194, 2024-08-20T21:51:24.2251944Z "digest": "sha256:25114d5bae5b7775201e9a1f79c06cc0ab7ca238a8357e83018a63d782aef5d5" 2024-08-20T21:51:24.2252716Z }, 2024-08-20T21:51:24.2252990Z { 2024-08-20T21:51:24.2253525Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2254215Z "size": 565, 2024-08-20T21:51:24.2254862Z "digest": "sha256:7dd58e631031354e323ac12b1e8223aa4b198de49bd1ac0d10d11ee97933d4a8" 2024-08-20T21:51:24.2255633Z }, 2024-08-20T21:51:24.2255933Z { 2024-08-20T21:51:24.2256446Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2257138Z "size": 43162887, 2024-08-20T21:51:24.2257824Z "digest": "sha256:b125137ca92e3bd051d165ca69937cd4939aa82b5095a4379e3b940187fdb4fe" 2024-08-20T21:51:24.2258568Z }, 2024-08-20T21:51:24.2258861Z { 2024-08-20T21:51:24.2259388Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2260060Z "size": 104, 2024-08-20T21:51:24.2260736Z "digest": "sha256:12dfa73284bb2bfe40228d03e187cbf9f4d59c5300419db1d5d562ad9ca4a5cd" 2024-08-20T21:51:24.2261511Z }, 2024-08-20T21:51:24.2261795Z { 2024-08-20T21:51:24.2262324Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2263016Z "size": 1379, 2024-08-20T21:51:24.2263664Z "digest": "sha256:045427066d9a1cb897779196ada05b7b790c1dce05bead327f658f20933e9a76" 2024-08-20T21:51:24.2264426Z }, 2024-08-20T21:51:24.2264721Z { 2024-08-20T21:51:24.2265241Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2265936Z "size": 699, 2024-08-20T21:51:24.2266605Z "digest": "sha256:28e54511f1b116a9d632caa0c4be2cd1b6ca7542f815560ec598b81d6d956d31" 2024-08-20T21:51:24.2267354Z }, 2024-08-20T21:51:24.2267645Z { 2024-08-20T21:51:24.2268170Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2268841Z "size": 139, 2024-08-20T21:51:24.2269526Z "digest": "sha256:b6428379fcaafd1025a47060dccafde4f6ac9d22475fff6ce5e51fc531bd9d98" 2024-08-20T21:51:24.2270307Z }, 2024-08-20T21:51:24.2270582Z { 2024-08-20T21:51:24.2271115Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2271939Z "size": 119, 2024-08-20T21:51:24.2272602Z "digest": "sha256:7f8afc2d089702d4d576f8a6a8841b21e66c0acbf3b735f76a4fe1c4cb0f1fa2" 2024-08-20T21:51:24.2273386Z }, 2024-08-20T21:51:24.2273677Z { 2024-08-20T21:51:24.2274197Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2274889Z "size": 4327084570, 2024-08-20T21:51:24.2275593Z "digest": "sha256:ae2172ebb76524e0aabfc93971b239448922f1f340cb20dbe61460dbd250f202" 2024-08-20T21:51:24.2276367Z }, 2024-08-20T21:51:24.2276643Z { 2024-08-20T21:51:24.2277173Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2277846Z "size": 173, 2024-08-20T21:51:24.2278530Z "digest": "sha256:c6cba7526eea3b7a7febd01cb06e32f421ab15e17a3692967040d95eb0675cca" 2024-08-20T21:51:24.2279316Z }, 2024-08-20T21:51:24.2279591Z { 2024-08-20T21:51:24.2280121Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2280816Z "size": 907, 2024-08-20T21:51:24.2281487Z "digest": "sha256:5be7fac2e2a0876700857b9032bc92b1f3b8470f6bcff815e18101d7d71c4054" 2024-08-20T21:51:24.2282246Z }, 2024-08-20T21:51:24.2282536Z { 2024-08-20T21:51:24.2283051Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2283832Z "size": 699, 2024-08-20T21:51:24.2284523Z "digest": "sha256:28e54511f1b116a9d632caa0c4be2cd1b6ca7542f815560ec598b81d6d956d31" 2024-08-20T21:51:24.2285296Z }, 2024-08-20T21:51:24.2285574Z { 2024-08-20T21:51:24.2286104Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2286798Z "size": 134, 2024-08-20T21:51:24.2287459Z "digest": "sha256:eea60c1ffc991e92d764514ee25756132c369fa2fc0de806c6f1f8eb64718a5b" 2024-08-20T21:51:24.2288237Z }, 2024-08-20T21:51:24.2288527Z { 2024-08-20T21:51:24.2289041Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2289738Z "size": 32, 2024-08-20T21:51:24.2290408Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:51:24.2291172Z }, 2024-08-20T21:51:24.2291463Z { 2024-08-20T21:51:24.2291994Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2292668Z "size": 158, 2024-08-20T21:51:24.2293353Z "digest": "sha256:e4b1fa4f961f28a8004b20020e9f3e14ddda223a8cf6273a7ec5dfc7b58a9b04" 2024-08-20T21:51:24.2294128Z }, 2024-08-20T21:51:24.2294403Z { 2024-08-20T21:51:24.2294934Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2295623Z "size": 1841, 2024-08-20T21:51:24.2296298Z "digest": "sha256:ff20aefbe592a1fd39676d339cf9ebce003fde8a0b5702b8301d756bdee912c7" 2024-08-20T21:51:24.2297083Z }, 2024-08-20T21:51:24.2297373Z { 2024-08-20T21:51:24.2297885Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2298576Z "size": 7529768, 2024-08-20T21:51:24.2299266Z "digest": "sha256:39f017df093b119d169de8c649be9cf342a3075430b27bf2388dc9ccdefafac8" 2024-08-20T21:51:24.2300015Z }, 2024-08-20T21:51:24.2300305Z { 2024-08-20T21:51:24.2300821Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2301487Z "size": 163, 2024-08-20T21:51:24.2302151Z "digest": "sha256:dcb4b1c5a6f604842cbe094756013849c141d2c73b9764babb9b3cf35684a797" 2024-08-20T21:51:24.2302921Z }, 2024-08-20T21:51:24.2303197Z { 2024-08-20T21:51:24.2303727Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2304413Z "size": 7943, 2024-08-20T21:51:24.2305061Z "digest": "sha256:862b4f5c5cd13e6c7641578efd673953421540d1b64af3e8a362b709a3ef443a" 2024-08-20T21:51:24.2305821Z }, 2024-08-20T21:51:24.2306098Z { 2024-08-20T21:51:24.2306612Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2307288Z "size": 8070, 2024-08-20T21:51:24.2307918Z "digest": "sha256:46546d0a53bb910aa9a28f5073146d18a700ca0d1ca7407044085170059104cf" 2024-08-20T21:51:24.2308745Z }, 2024-08-20T21:51:24.2309014Z { 2024-08-20T21:51:24.2309531Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2310204Z "size": 301, 2024-08-20T21:51:24.2310892Z "digest": "sha256:82a31e7860af47e63f76dcfa09bf02b9ed5ba7f0c2c17804cbdd4e4f9dc7ab1d" 2024-08-20T21:51:24.2311664Z }, 2024-08-20T21:51:24.2311941Z { 2024-08-20T21:51:24.2312471Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2313160Z "size": 7629503, 2024-08-20T21:51:24.2313845Z "digest": "sha256:ac2f6809f82b0e504eb4e1bfada74e84542eaff1a26c06047ffecdcabc8f2f04" 2024-08-20T21:51:24.2314635Z }, 2024-08-20T21:51:24.2314928Z { 2024-08-20T21:51:24.2315442Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2316122Z "size": 108, 2024-08-20T21:51:24.2316794Z "digest": "sha256:2b45e44613094d77c76cfbe3d12bd0a314f3cfea24cf97303d9c6ee51715a021" 2024-08-20T21:51:24.2317559Z }, 2024-08-20T21:51:24.2317845Z { 2024-08-20T21:51:24.2318366Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2319021Z "size": 54145661, 2024-08-20T21:51:24.2319677Z "digest": "sha256:403462e9ee27186eada6b3b99891b31b4394b51e12d017f8fc1c302e33444a8a" 2024-08-20T21:51:24.2320532Z }, 2024-08-20T21:51:24.2320824Z { 2024-08-20T21:51:24.2321360Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2322054Z "size": 472, 2024-08-20T21:51:24.2322687Z "digest": "sha256:889f478638723a13f19e34f1f415d0c7dd7070c02484cb105d03601eb16e45f4" 2024-08-20T21:51:24.2323438Z }, 2024-08-20T21:51:24.2323710Z { 2024-08-20T21:51:24.2324210Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2324877Z "size": 1373229330, 2024-08-20T21:51:24.2325555Z "digest": "sha256:23898c5065b1e76bbf101c2151c1f06c50633abb686f1a87ee578de1acce2eaf" 2024-08-20T21:51:24.2326307Z }, 2024-08-20T21:51:24.2326597Z { 2024-08-20T21:51:24.2327124Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2327797Z "size": 106, 2024-08-20T21:51:24.2328464Z "digest": "sha256:f1caf060fc92271565fb07a66ff67d61a3bb761687913767cebd4240754275db" 2024-08-20T21:51:24.2329232Z }, 2024-08-20T21:51:24.2329516Z { 2024-08-20T21:51:24.2330043Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2330732Z "size": 559, 2024-08-20T21:51:24.2331424Z "digest": "sha256:621b6cedbe16b797e97c910d222978f9f3077752e0386617189079966c30ee02" 2024-08-20T21:51:24.2332178Z }, 2024-08-20T21:51:24.2332465Z { 2024-08-20T21:51:24.2332980Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2333669Z "size": 36532053, 2024-08-20T21:51:24.2334358Z "digest": "sha256:b9c038a9eb4ac830af69e080b96c2d591156429f179e4e922d025624341693d4" 2024-08-20T21:51:24.2335100Z }, 2024-08-20T21:51:24.2335389Z { 2024-08-20T21:51:24.2335927Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2336601Z "size": 111, 2024-08-20T21:51:24.2337761Z "digest": "sha256:74046cd1f067b5edf32140ab31499a79acf8919e57afd4bc4debeafeb7f993ed" 2024-08-20T21:51:24.2338553Z }, 2024-08-20T21:51:24.2338836Z { 2024-08-20T21:51:24.2339378Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2340067Z "size": 32, 2024-08-20T21:51:24.2340726Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:51:24.2341503Z }, 2024-08-20T21:51:24.2341823Z { 2024-08-20T21:51:24.2342340Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2343035Z "size": 32, 2024-08-20T21:51:24.2343707Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:51:24.2344466Z }, 2024-08-20T21:51:24.2344757Z { 2024-08-20T21:51:24.2345294Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2346124Z "size": 32, 2024-08-20T21:51:24.2346798Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:51:24.2347578Z }, 2024-08-20T21:51:24.2347856Z { 2024-08-20T21:51:24.2348393Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.2349086Z "size": 32, 2024-08-20T21:51:24.2349740Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:51:24.2350516Z } 2024-08-20T21:51:24.2350811Z ] 2024-08-20T21:51:24.2351084Z } 2024-08-20T21:51:24.2482839Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T21:51:24.2483459Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T21:51:24.2484132Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-20T21:51:24.2491441Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:24.2492067Z env: 2024-08-20T21:51:24.2492399Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:24.2493720Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:24.2495052Z ##[endgroup] 2024-08-20T21:51:24.2526033Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9-f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:24.2591388Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-20T21:51:24.2592123Z with: 2024-08-20T21:51:24.2593315Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:24.2594831Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:24.2595461Z env: 2024-08-20T21:51:24.2595788Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:24.2596206Z ##[endgroup] 2024-08-20T21:51:24.2619855Z ##[group]Run set -x 2024-08-20T21:51:24.2620272Z set -x 2024-08-20T21:51:24.2620649Z set +e 2024-08-20T21:51:24.2621003Z  2024-08-20T21:51:24.2621322Z login() { 2024-08-20T21:51:24.2622147Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T21:51:24.2623059Z } 2024-08-20T21:51:24.2623370Z  2024-08-20T21:51:24.2623747Z retry () { 2024-08-20T21:51:24.2624241Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T21:51:24.2624800Z } 2024-08-20T21:51:24.2625111Z  2024-08-20T21:51:24.2625493Z retry login "${DOCKER_REGISTRY}" 2024-08-20T21:51:24.2626009Z  2024-08-20T21:51:24.2626317Z set -e 2024-08-20T21:51:24.2626906Z # ignore output since only exit code is used for conditional 2024-08-20T21:51:24.2627793Z # only pull docker image if it's not available locally 2024-08-20T21:51:24.2628768Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-20T21:51:24.2629640Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-20T21:51:24.2630187Z fi 2024-08-20T21:51:24.2636518Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:24.2637642Z env: 2024-08-20T21:51:24.2638001Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:24.2639314Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:24.2640806Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:24.2641456Z ##[endgroup] 2024-08-20T21:51:24.2668212Z + set +e 2024-08-20T21:51:24.2668855Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:24.2669684Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:24.2672441Z + aws ecr get-login-password --region us-east-1 2024-08-20T21:51:24.2673790Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:24.9427646Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T21:51:24.9428693Z Configure a credential helper to remove this warning. See 2024-08-20T21:51:24.9429844Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T21:51:24.9430532Z 2024-08-20T21:51:24.9430711Z Login Succeeded 2024-08-20T21:51:24.9441380Z + set -e 2024-08-20T21:51:24.9442936Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:24.9610218Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:24.9612650Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:25.1769886Z f6d216893d65c7b8ae43df4daaf247db808378e9: Pulling from pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9 2024-08-20T21:51:25.1771174Z 7a2c55901189: Pulling fs layer 2024-08-20T21:51:25.1771700Z 09d415c238d7: Pulling fs layer 2024-08-20T21:51:25.1772248Z 9fe6e2e61518: Pulling fs layer 2024-08-20T21:51:25.1772714Z 41f16248e682: Pulling fs layer 2024-08-20T21:51:25.1773477Z 95d7b7817039: Pulling fs layer 2024-08-20T21:51:25.1774000Z 8f6c90485347: Pulling fs layer 2024-08-20T21:51:25.1774495Z ab17245097e4: Pulling fs layer 2024-08-20T21:51:25.1774946Z dfecd7e9912b: Pulling fs layer 2024-08-20T21:51:25.1775581Z 464a8f745445: Pulling fs layer 2024-08-20T21:51:25.1776079Z d67c111fa588: Pulling fs layer 2024-08-20T21:51:25.1776524Z 0723abde19c0: Pulling fs layer 2024-08-20T21:51:25.1777003Z 72d2b53f7a01: Pulling fs layer 2024-08-20T21:51:25.1777586Z dbc6fccd9c70: Pulling fs layer 2024-08-20T21:51:25.1778050Z ff6a6ddc7310: Pulling fs layer 2024-08-20T21:51:25.1778523Z fac2b1b246ed: Pulling fs layer 2024-08-20T21:51:25.1779014Z bc574f79516b: Pulling fs layer 2024-08-20T21:51:25.1779427Z c23d42adcade: Pulling fs layer 2024-08-20T21:51:25.1779901Z 59b0a331db9f: Pulling fs layer 2024-08-20T21:51:25.1780378Z 478a9a2bdefc: Pulling fs layer 2024-08-20T21:51:25.1780815Z 28e54511f1b1: Pulling fs layer 2024-08-20T21:51:25.1781281Z dfecd7e9912b: Waiting 2024-08-20T21:51:25.1781707Z 464a8f745445: Waiting 2024-08-20T21:51:25.1782084Z 5247f18de881: Pulling fs layer 2024-08-20T21:51:25.1782560Z 38760708e2d7: Pulling fs layer 2024-08-20T21:51:25.1783038Z 863dbde6ed0c: Pulling fs layer 2024-08-20T21:51:25.1783460Z f283ab568a00: Pulling fs layer 2024-08-20T21:51:25.1783904Z d67c111fa588: Waiting 2024-08-20T21:51:25.1784331Z dbc6fccd9c70: Waiting 2024-08-20T21:51:25.1784713Z 3e39f30b1d5c: Pulling fs layer 2024-08-20T21:51:25.1785167Z 0723abde19c0: Waiting 2024-08-20T21:51:25.1785586Z ff6a6ddc7310: Waiting 2024-08-20T21:51:25.1785929Z 72d2b53f7a01: Waiting 2024-08-20T21:51:25.1786320Z fac2b1b246ed: Waiting 2024-08-20T21:51:25.1786743Z 521c3d72534c: Pulling fs layer 2024-08-20T21:51:25.1787169Z 41f16248e682: Waiting 2024-08-20T21:51:25.1787621Z c23d42adcade: Waiting 2024-08-20T21:51:25.1788038Z a975a4480a6d: Pulling fs layer 2024-08-20T21:51:25.1788482Z 95d7b7817039: Waiting 2024-08-20T21:51:25.1788851Z 59b0a331db9f: Waiting 2024-08-20T21:51:25.1789240Z 28e54511f1b1: Waiting 2024-08-20T21:51:25.1789637Z 478a9a2bdefc: Waiting 2024-08-20T21:51:25.1790012Z 3e39f30b1d5c: Waiting 2024-08-20T21:51:25.1790399Z bc574f79516b: Waiting 2024-08-20T21:51:25.1790805Z 521c3d72534c: Waiting 2024-08-20T21:51:25.1791335Z 1f2c278212ad: Pulling fs layer 2024-08-20T21:51:25.1791776Z 38760708e2d7: Waiting 2024-08-20T21:51:25.1792182Z 5247f18de881: Waiting 2024-08-20T21:51:25.1792547Z b721dd867e80: Pulling fs layer 2024-08-20T21:51:25.1793028Z f283ab568a00: Waiting 2024-08-20T21:51:25.1793497Z 8f6c90485347: Waiting 2024-08-20T21:51:25.1793891Z 7a74b7ad649d: Pulling fs layer 2024-08-20T21:51:25.1794341Z ab17245097e4: Waiting 2024-08-20T21:51:25.1794930Z b721dd867e80: Waiting 2024-08-20T21:51:25.1795329Z 1f2c278212ad: Waiting 2024-08-20T21:51:25.1795710Z b53e1ad1deec: Pulling fs layer 2024-08-20T21:51:25.1796164Z a975a4480a6d: Waiting 2024-08-20T21:51:25.1796548Z 7a74b7ad649d: Waiting 2024-08-20T21:51:25.1796929Z b53e1ad1deec: Waiting 2024-08-20T21:51:25.1797321Z 016dc3faa7ef: Pulling fs layer 2024-08-20T21:51:25.1797802Z 384dce3a4ba2: Pulling fs layer 2024-08-20T21:51:25.1798261Z 863dbde6ed0c: Waiting 2024-08-20T21:51:25.1798654Z 75461ad9f7bb: Pulling fs layer 2024-08-20T21:51:25.1799119Z eb2062102d11: Pulling fs layer 2024-08-20T21:51:25.1799569Z 016dc3faa7ef: Waiting 2024-08-20T21:51:25.1799941Z 75461ad9f7bb: Waiting 2024-08-20T21:51:25.1800353Z 3cc17f105239: Pulling fs layer 2024-08-20T21:51:25.1800802Z 384dce3a4ba2: Waiting 2024-08-20T21:51:25.1801173Z eb2062102d11: Waiting 2024-08-20T21:51:25.1801619Z 585501be0799: Pulling fs layer 2024-08-20T21:51:25.1802080Z 6c49da68c9b4: Pulling fs layer 2024-08-20T21:51:25.1802493Z 497539689260: Pulling fs layer 2024-08-20T21:51:25.1803103Z 3cc17f105239: Waiting 2024-08-20T21:51:25.1803626Z 5843bb8b81bb: Pulling fs layer 2024-08-20T21:51:25.1804177Z 222174e4e2b2: Pulling fs layer 2024-08-20T21:51:25.1804657Z 8d55064a9f20: Pulling fs layer 2024-08-20T21:51:25.1805122Z 585501be0799: Waiting 2024-08-20T21:51:25.1805473Z 222174e4e2b2: Waiting 2024-08-20T21:51:25.1805860Z 5843bb8b81bb: Waiting 2024-08-20T21:51:25.1806443Z 4f4fb700ef54: Pulling fs layer 2024-08-20T21:51:25.1806861Z 6c49da68c9b4: Waiting 2024-08-20T21:51:25.1807252Z 8d55064a9f20: Waiting 2024-08-20T21:51:25.1807661Z 25114d5bae5b: Pulling fs layer 2024-08-20T21:51:25.1808087Z 497539689260: Waiting 2024-08-20T21:51:25.1808490Z 7dd58e631031: Pulling fs layer 2024-08-20T21:51:25.1808954Z b125137ca92e: Pulling fs layer 2024-08-20T21:51:25.1809403Z 12dfa73284bb: Pulling fs layer 2024-08-20T21:51:25.1809865Z 045427066d9a: Pulling fs layer 2024-08-20T21:51:25.1810321Z 4f4fb700ef54: Waiting 2024-08-20T21:51:25.1810687Z 7dd58e631031: Waiting 2024-08-20T21:51:25.1811085Z 25114d5bae5b: Waiting 2024-08-20T21:51:25.1811476Z 12dfa73284bb: Waiting 2024-08-20T21:51:25.1811868Z b6428379fcaa: Pulling fs layer 2024-08-20T21:51:25.1812317Z b125137ca92e: Waiting 2024-08-20T21:51:25.1812706Z b6428379fcaa: Waiting 2024-08-20T21:51:25.1813099Z 7f8afc2d0897: Pulling fs layer 2024-08-20T21:51:25.1813572Z ae2172ebb765: Pulling fs layer 2024-08-20T21:51:25.1814027Z 045427066d9a: Waiting 2024-08-20T21:51:25.1814443Z c6cba7526eea: Pulling fs layer 2024-08-20T21:51:25.1814907Z 5be7fac2e2a0: Pulling fs layer 2024-08-20T21:51:25.1815349Z 7f8afc2d0897: Waiting 2024-08-20T21:51:25.1815776Z eea60c1ffc99: Pulling fs layer 2024-08-20T21:51:25.1816335Z 5be7fac2e2a0: Waiting 2024-08-20T21:51:25.1816758Z c6cba7526eea: Waiting 2024-08-20T21:51:25.1817168Z ae2172ebb765: Waiting 2024-08-20T21:51:25.1817543Z e4b1fa4f961f: Pulling fs layer 2024-08-20T21:51:25.1818019Z ff20aefbe592: Pulling fs layer 2024-08-20T21:51:25.1818469Z 39f017df093b: Pulling fs layer 2024-08-20T21:51:25.1818940Z dcb4b1c5a6f6: Pulling fs layer 2024-08-20T21:51:25.1819404Z ff20aefbe592: Waiting 2024-08-20T21:51:25.1819778Z 39f017df093b: Waiting 2024-08-20T21:51:25.1820173Z eea60c1ffc99: Waiting 2024-08-20T21:51:25.1820587Z 862b4f5c5cd1: Pulling fs layer 2024-08-20T21:51:25.1821021Z dcb4b1c5a6f6: Waiting 2024-08-20T21:51:25.1821433Z 46546d0a53bb: Pulling fs layer 2024-08-20T21:51:25.1821891Z 862b4f5c5cd1: Waiting 2024-08-20T21:51:25.1822282Z 82a31e7860af: Pulling fs layer 2024-08-20T21:51:25.1822730Z 46546d0a53bb: Waiting 2024-08-20T21:51:25.1823140Z ac2f6809f82b: Pulling fs layer 2024-08-20T21:51:25.1823591Z 2b45e4461309: Pulling fs layer 2024-08-20T21:51:25.1824056Z 403462e9ee27: Pulling fs layer 2024-08-20T21:51:25.1824537Z 889f47863872: Pulling fs layer 2024-08-20T21:51:25.1824948Z 82a31e7860af: Waiting 2024-08-20T21:51:25.1825338Z ac2f6809f82b: Waiting 2024-08-20T21:51:25.1825757Z 23898c5065b1: Pulling fs layer 2024-08-20T21:51:25.1826175Z 2b45e4461309: Waiting 2024-08-20T21:51:25.1826564Z 403462e9ee27: Waiting 2024-08-20T21:51:25.1827082Z f1caf060fc92: Pulling fs layer 2024-08-20T21:51:25.1827522Z 23898c5065b1: Waiting 2024-08-20T21:51:25.1827907Z 889f47863872: Waiting 2024-08-20T21:51:25.1828294Z 621b6cedbe16: Pulling fs layer 2024-08-20T21:51:25.1828752Z f1caf060fc92: Waiting 2024-08-20T21:51:25.1829167Z b9c038a9eb4a: Pulling fs layer 2024-08-20T21:51:25.1829601Z 621b6cedbe16: Waiting 2024-08-20T21:51:25.1830017Z 74046cd1f067: Pulling fs layer 2024-08-20T21:51:25.1830487Z b9c038a9eb4a: Waiting 2024-08-20T21:51:25.3164148Z 09d415c238d7: Download complete 2024-08-20T21:51:25.3945104Z 41f16248e682: Verifying Checksum 2024-08-20T21:51:25.3945654Z 41f16248e682: Download complete 2024-08-20T21:51:25.4660886Z 95d7b7817039: Verifying Checksum 2024-08-20T21:51:25.5219214Z 7a2c55901189: Verifying Checksum 2024-08-20T21:51:25.5219743Z 7a2c55901189: Download complete 2024-08-20T21:51:25.6062457Z ab17245097e4: Verifying Checksum 2024-08-20T21:51:25.6063079Z ab17245097e4: Download complete 2024-08-20T21:51:25.6832547Z dfecd7e9912b: Verifying Checksum 2024-08-20T21:51:25.6833170Z dfecd7e9912b: Download complete 2024-08-20T21:51:25.7588405Z 464a8f745445: Verifying Checksum 2024-08-20T21:51:25.7963165Z 464a8f745445: Download complete 2024-08-20T21:51:25.7963686Z 9fe6e2e61518: Verifying Checksum 2024-08-20T21:51:25.7964170Z 9fe6e2e61518: Download complete 2024-08-20T21:51:25.8707910Z 0723abde19c0: Verifying Checksum 2024-08-20T21:51:25.8708815Z 0723abde19c0: Download complete 2024-08-20T21:51:25.9406541Z 72d2b53f7a01: Verifying Checksum 2024-08-20T21:51:25.9407377Z 72d2b53f7a01: Download complete 2024-08-20T21:51:26.7761437Z 7a2c55901189: Pull complete 2024-08-20T21:51:27.1095737Z 09d415c238d7: Pull complete 2024-08-20T21:51:28.1377594Z 9fe6e2e61518: Pull complete 2024-08-20T21:51:28.1565981Z 41f16248e682: Pull complete 2024-08-20T21:51:28.1758297Z 95d7b7817039: Pull complete 2024-08-20T21:51:32.5422095Z dbc6fccd9c70: Verifying Checksum 2024-08-20T21:51:32.5422707Z dbc6fccd9c70: Download complete 2024-08-20T21:51:32.6310529Z ff6a6ddc7310: Verifying Checksum 2024-08-20T21:51:32.6311153Z ff6a6ddc7310: Download complete 2024-08-20T21:51:32.7022522Z fac2b1b246ed: Download complete 2024-08-20T21:51:32.7730100Z bc574f79516b: Verifying Checksum 2024-08-20T21:51:32.7730645Z bc574f79516b: Download complete 2024-08-20T21:51:33.7500337Z c23d42adcade: Verifying Checksum 2024-08-20T21:51:33.7501093Z c23d42adcade: Download complete 2024-08-20T21:51:33.8310277Z 59b0a331db9f: Verifying Checksum 2024-08-20T21:51:33.8310985Z 59b0a331db9f: Download complete 2024-08-20T21:51:33.9127011Z 478a9a2bdefc: Verifying Checksum 2024-08-20T21:51:33.9127545Z 478a9a2bdefc: Download complete 2024-08-20T21:51:33.9883951Z 28e54511f1b1: Download complete 2024-08-20T21:51:39.3640450Z 8f6c90485347: Verifying Checksum 2024-08-20T21:51:39.3641067Z 8f6c90485347: Download complete 2024-08-20T21:51:39.4372784Z 38760708e2d7: Download complete 2024-08-20T21:51:39.5181563Z 863dbde6ed0c: Download complete 2024-08-20T21:51:39.5970671Z f283ab568a00: Verifying Checksum 2024-08-20T21:51:39.5971235Z f283ab568a00: Download complete 2024-08-20T21:51:39.6650516Z 3e39f30b1d5c: Verifying Checksum 2024-08-20T21:51:39.6651205Z 3e39f30b1d5c: Download complete 2024-08-20T21:51:39.7494969Z 521c3d72534c: Verifying Checksum 2024-08-20T21:51:39.7495680Z 521c3d72534c: Download complete 2024-08-20T21:51:41.0231708Z a975a4480a6d: Verifying Checksum 2024-08-20T21:51:41.0232453Z a975a4480a6d: Download complete 2024-08-20T21:51:41.0972418Z 1f2c278212ad: Verifying Checksum 2024-08-20T21:51:41.0973034Z 1f2c278212ad: Download complete 2024-08-20T21:51:41.1813960Z b721dd867e80: Verifying Checksum 2024-08-20T21:51:41.1814616Z b721dd867e80: Download complete 2024-08-20T21:51:41.2632503Z 7a74b7ad649d: Verifying Checksum 2024-08-20T21:51:41.2633068Z 7a74b7ad649d: Download complete 2024-08-20T21:51:41.3393500Z b53e1ad1deec: Verifying Checksum 2024-08-20T21:51:41.3394102Z b53e1ad1deec: Download complete 2024-08-20T21:51:41.4086444Z 016dc3faa7ef: Verifying Checksum 2024-08-20T21:51:41.4087337Z 016dc3faa7ef: Download complete 2024-08-20T21:51:47.4106570Z 384dce3a4ba2: Verifying Checksum 2024-08-20T21:51:47.4107216Z 384dce3a4ba2: Download complete 2024-08-20T21:51:47.5168197Z 75461ad9f7bb: Verifying Checksum 2024-08-20T21:51:47.5168740Z 75461ad9f7bb: Download complete 2024-08-20T21:51:47.5981908Z eb2062102d11: Verifying Checksum 2024-08-20T21:51:47.5982454Z eb2062102d11: Download complete 2024-08-20T21:51:48.0106751Z 3cc17f105239: Verifying Checksum 2024-08-20T21:51:48.0107667Z 3cc17f105239: Download complete 2024-08-20T21:51:48.1071941Z 585501be0799: Verifying Checksum 2024-08-20T21:51:48.1072576Z 585501be0799: Download complete 2024-08-20T21:51:48.1844407Z 6c49da68c9b4: Verifying Checksum 2024-08-20T21:51:48.1845023Z 6c49da68c9b4: Download complete 2024-08-20T21:51:48.4489169Z 497539689260: Verifying Checksum 2024-08-20T21:51:48.4491102Z 497539689260: Download complete 2024-08-20T21:51:48.5267721Z 5843bb8b81bb: Verifying Checksum 2024-08-20T21:51:48.5269651Z 5843bb8b81bb: Download complete 2024-08-20T21:51:48.6073615Z 222174e4e2b2: Verifying Checksum 2024-08-20T21:51:48.6075621Z 222174e4e2b2: Download complete 2024-08-20T21:51:48.6810918Z 8d55064a9f20: Download complete 2024-08-20T21:51:48.6916840Z 4f4fb700ef54: Verifying Checksum 2024-08-20T21:51:48.6917532Z 4f4fb700ef54: Download complete 2024-08-20T21:51:48.7699839Z 25114d5bae5b: Verifying Checksum 2024-08-20T21:51:48.7700421Z 25114d5bae5b: Download complete 2024-08-20T21:51:48.8465400Z 7dd58e631031: Verifying Checksum 2024-08-20T21:51:48.8466228Z 7dd58e631031: Download complete 2024-08-20T21:51:49.3393300Z b125137ca92e: Verifying Checksum 2024-08-20T21:51:49.3394257Z b125137ca92e: Download complete 2024-08-20T21:51:49.4351254Z 12dfa73284bb: Download complete 2024-08-20T21:51:49.5219269Z 045427066d9a: Verifying Checksum 2024-08-20T21:51:49.5219807Z 045427066d9a: Download complete 2024-08-20T21:51:49.6119199Z b6428379fcaa: Verifying Checksum 2024-08-20T21:51:49.6119931Z b6428379fcaa: Download complete 2024-08-20T21:51:49.6993534Z 7f8afc2d0897: Verifying Checksum 2024-08-20T21:51:49.6994523Z 7f8afc2d0897: Download complete 2024-08-20T21:51:50.2795403Z d67c111fa588: Verifying Checksum 2024-08-20T21:51:50.2796289Z d67c111fa588: Download complete 2024-08-20T21:51:50.3595492Z c6cba7526eea: Verifying Checksum 2024-08-20T21:51:50.3596089Z c6cba7526eea: Download complete 2024-08-20T21:51:50.4289376Z 5be7fac2e2a0: Verifying Checksum 2024-08-20T21:51:50.4290098Z 5be7fac2e2a0: Download complete 2024-08-20T21:51:50.5057678Z eea60c1ffc99: Download complete 2024-08-20T21:51:50.6474173Z e4b1fa4f961f: Verifying Checksum 2024-08-20T21:51:50.6475221Z e4b1fa4f961f: Download complete 2024-08-20T21:51:50.7248961Z ff20aefbe592: Verifying Checksum 2024-08-20T21:51:50.7249613Z ff20aefbe592: Download complete 2024-08-20T21:51:50.8580107Z 39f017df093b: Verifying Checksum 2024-08-20T21:51:50.8580716Z 39f017df093b: Download complete 2024-08-20T21:51:50.9256189Z dcb4b1c5a6f6: Verifying Checksum 2024-08-20T21:51:50.9257134Z dcb4b1c5a6f6: Download complete 2024-08-20T21:51:51.0149321Z 862b4f5c5cd1: Verifying Checksum 2024-08-20T21:51:51.0150240Z 862b4f5c5cd1: Download complete 2024-08-20T21:51:51.0915249Z 46546d0a53bb: Verifying Checksum 2024-08-20T21:51:51.0916185Z 46546d0a53bb: Download complete 2024-08-20T21:51:51.1739883Z 82a31e7860af: Download complete 2024-08-20T21:51:51.3535550Z ac2f6809f82b: Verifying Checksum 2024-08-20T21:51:51.3536653Z ac2f6809f82b: Download complete 2024-08-20T21:51:51.4309807Z 2b45e4461309: Verifying Checksum 2024-08-20T21:51:51.4310776Z 2b45e4461309: Download complete 2024-08-20T21:51:52.0160196Z 403462e9ee27: Verifying Checksum 2024-08-20T21:51:52.0989304Z 403462e9ee27: Download complete 2024-08-20T21:51:52.0989872Z 889f47863872: Verifying Checksum 2024-08-20T21:51:52.0990373Z 889f47863872: Download complete 2024-08-20T21:51:56.4235923Z 8f6c90485347: Pull complete 2024-08-20T21:51:56.8721044Z ab17245097e4: Pull complete 2024-08-20T21:51:57.3164466Z dfecd7e9912b: Pull complete 2024-08-20T21:51:57.7645793Z 464a8f745445: Pull complete 2024-08-20T21:52:02.9614277Z 5247f18de881: Verifying Checksum 2024-08-20T21:52:02.9615173Z 5247f18de881: Download complete 2024-08-20T21:52:03.0491606Z f1caf060fc92: Verifying Checksum 2024-08-20T21:52:03.0492150Z f1caf060fc92: Download complete 2024-08-20T21:52:03.1269026Z 621b6cedbe16: Verifying Checksum 2024-08-20T21:52:03.1269606Z 621b6cedbe16: Download complete 2024-08-20T21:52:03.5580198Z b9c038a9eb4a: Verifying Checksum 2024-08-20T21:52:03.5580829Z b9c038a9eb4a: Download complete 2024-08-20T21:52:03.6442771Z 74046cd1f067: Verifying Checksum 2024-08-20T21:52:03.6443454Z 74046cd1f067: Download complete 2024-08-20T21:52:05.9045554Z 23898c5065b1: Verifying Checksum 2024-08-20T21:52:31.5938939Z d67c111fa588: Pull complete 2024-08-20T21:52:32.0580983Z 0723abde19c0: Pull complete 2024-08-20T21:52:32.5073624Z 72d2b53f7a01: Pull complete 2024-08-20T21:52:33.0245727Z ae2172ebb765: Verifying Checksum 2024-08-20T21:52:33.0246337Z ae2172ebb765: Download complete 2024-08-20T21:52:46.2431514Z dbc6fccd9c70: Pull complete 2024-08-20T21:52:46.6935129Z ff6a6ddc7310: Pull complete 2024-08-20T21:52:47.1514866Z fac2b1b246ed: Pull complete 2024-08-20T21:52:47.6091980Z bc574f79516b: Pull complete 2024-08-20T21:52:50.9023669Z c23d42adcade: Pull complete 2024-08-20T21:52:51.3523848Z 59b0a331db9f: Pull complete 2024-08-20T21:52:51.7218103Z 478a9a2bdefc: Pull complete 2024-08-20T21:52:52.1581856Z 28e54511f1b1: Pull complete 2024-08-20T21:53:56.3228409Z 5247f18de881: Pull complete 2024-08-20T21:53:56.3457736Z 38760708e2d7: Pull complete 2024-08-20T21:53:56.3777235Z 863dbde6ed0c: Pull complete 2024-08-20T21:53:56.3996497Z f283ab568a00: Pull complete 2024-08-20T21:53:56.4217829Z 3e39f30b1d5c: Pull complete 2024-08-20T21:53:56.4443042Z 521c3d72534c: Pull complete 2024-08-20T21:54:00.4521753Z a975a4480a6d: Pull complete 2024-08-20T21:54:00.7079763Z 1f2c278212ad: Pull complete 2024-08-20T21:54:01.0866111Z b721dd867e80: Pull complete 2024-08-20T21:54:01.3596243Z 7a74b7ad649d: Pull complete 2024-08-20T21:54:01.6098259Z b53e1ad1deec: Pull complete 2024-08-20T21:54:01.8216973Z 016dc3faa7ef: Pull complete 2024-08-20T21:54:14.0851244Z 384dce3a4ba2: Pull complete 2024-08-20T21:54:14.5435063Z 75461ad9f7bb: Pull complete 2024-08-20T21:54:14.9979897Z eb2062102d11: Pull complete 2024-08-20T21:54:16.3557024Z 3cc17f105239: Pull complete 2024-08-20T21:54:16.7188895Z 585501be0799: Pull complete 2024-08-20T21:54:16.9148600Z 6c49da68c9b4: Pull complete 2024-08-20T21:54:17.6712700Z 497539689260: Pull complete 2024-08-20T21:54:18.1167327Z 5843bb8b81bb: Pull complete 2024-08-20T21:54:19.0403305Z 222174e4e2b2: Pull complete 2024-08-20T21:54:19.4952461Z 8d55064a9f20: Pull complete 2024-08-20T21:54:19.8081679Z 4f4fb700ef54: Pull complete 2024-08-20T21:54:20.1148364Z 25114d5bae5b: Pull complete 2024-08-20T21:54:20.5145671Z 7dd58e631031: Pull complete 2024-08-20T21:54:23.6924266Z b125137ca92e: Pull complete 2024-08-20T21:54:24.1488207Z 12dfa73284bb: Pull complete 2024-08-20T21:54:24.6037580Z 045427066d9a: Pull complete 2024-08-20T21:54:25.5251836Z b6428379fcaa: Pull complete 2024-08-20T21:54:25.9554100Z 7f8afc2d0897: Pull complete 2024-08-20T21:56:00.8617721Z ae2172ebb765: Pull complete 2024-08-20T21:56:01.3212534Z c6cba7526eea: Pull complete 2024-08-20T21:56:01.7751896Z 5be7fac2e2a0: Pull complete 2024-08-20T21:56:02.6727461Z eea60c1ffc99: Pull complete 2024-08-20T21:56:03.5600801Z e4b1fa4f961f: Pull complete 2024-08-20T21:56:04.0049634Z ff20aefbe592: Pull complete 2024-08-20T21:56:04.6280279Z 39f017df093b: Pull complete 2024-08-20T21:56:05.0675321Z dcb4b1c5a6f6: Pull complete 2024-08-20T21:56:05.4759890Z 862b4f5c5cd1: Pull complete 2024-08-20T21:56:05.9205690Z 46546d0a53bb: Pull complete 2024-08-20T21:56:06.3692666Z 82a31e7860af: Pull complete 2024-08-20T21:56:08.1171727Z ac2f6809f82b: Pull complete 2024-08-20T21:56:08.5675971Z 2b45e4461309: Pull complete 2024-08-20T21:56:11.2614408Z 403462e9ee27: Pull complete 2024-08-20T21:56:11.7212608Z 889f47863872: Pull complete 2024-08-20T21:56:27.4966989Z 23898c5065b1: Pull complete 2024-08-20T21:56:27.9264041Z f1caf060fc92: Pull complete 2024-08-20T21:56:28.3840549Z 621b6cedbe16: Pull complete 2024-08-20T21:56:29.4325245Z b9c038a9eb4a: Pull complete 2024-08-20T21:56:29.7633011Z 74046cd1f067: Pull complete 2024-08-20T21:56:31.5702632Z Digest: sha256:0b3aa0836d4f18656c98dcf355b46cbae2123243424ebbed0929206c3d798126 2024-08-20T21:56:31.6641521Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:56:31.7162300Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:56:31.7235510Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T21:56:31.7236703Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T21:56:31.7246964Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:56:31.7247598Z env: 2024-08-20T21:56:31.7247919Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:56:31.7248361Z ##[endgroup] 2024-08-20T21:56:31.7476045Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-08-20T21:56:31.7476755Z with: 2024-08-20T21:56:31.7477112Z driver-version: 550.54.15 2024-08-20T21:56:31.7477536Z env: 2024-08-20T21:56:31.7477851Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:56:31.7478286Z ##[endgroup] 2024-08-20T21:56:31.7715685Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T21:56:31.7716405Z with: 2024-08-20T21:56:31.7716738Z timeout_minutes: 10 2024-08-20T21:56:31.7717122Z max_attempts: 3 2024-08-20T21:56:31.7761482Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-08-20T21:56:31.7805884Z retry_wait_seconds: 10 2024-08-20T21:56:31.7806344Z polling_interval_seconds: 1 2024-08-20T21:56:31.7806811Z warning_on_retry: true 2024-08-20T21:56:31.7807224Z continue_on_error: false 2024-08-20T21:56:31.7807635Z env: 2024-08-20T21:56:31.7807944Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:56:31.7808379Z DRIVER_VERSION: 550.54.15 2024-08-20T21:56:31.7808804Z ##[endgroup] 2024-08-20T21:56:31.8814451Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-08-20T21:56:31.8816075Z + pre_install_nvidia_driver_amzn2 2024-08-20T21:56:31.8817390Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-08-20T21:56:32.4542460Z No match for argument: nvidia-driver-latest-dkms 2024-08-20T21:56:32.4543134Z No packages marked for removal. 2024-08-20T21:56:32.4619185Z Dependencies resolved. 2024-08-20T21:56:32.4632580Z Nothing to do. 2024-08-20T21:56:32.4633265Z Complete! 2024-08-20T21:56:32.6098730Z + install_nvidia_driver_common 2024-08-20T21:56:32.6102403Z + echo 'Before installing NVIDIA driver' 2024-08-20T21:56:32.6102997Z Before installing NVIDIA driver 2024-08-20T21:56:32.6104910Z + lspci 2024-08-20T21:56:32.6301536Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-08-20T21:56:32.6302541Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T21:56:32.6303527Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-08-20T21:56:32.6304508Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-08-20T21:56:32.6305328Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-08-20T21:56:32.6306202Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T21:56:32.6307249Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-20T21:56:32.6308321Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-20T21:56:32.6309568Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-08-20T21:56:32.6310335Z + lsmod 2024-08-20T21:56:32.6344264Z Module Size Used by 2024-08-20T21:56:32.6344802Z nvidia_uvm 4706304 0 2024-08-20T21:56:32.6345283Z nvidia 54071296 1 nvidia_uvm 2024-08-20T21:56:32.6345798Z drm 602112 1 nvidia 2024-08-20T21:56:32.6346369Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T21:56:32.6346933Z backlight 24576 1 drm 2024-08-20T21:56:32.6347411Z i2c_core 106496 2 nvidia,drm 2024-08-20T21:56:32.6347947Z xt_conntrack 16384 1 2024-08-20T21:56:32.6348427Z nft_chain_nat 16384 3 2024-08-20T21:56:32.6348959Z xt_MASQUERADE 20480 1 2024-08-20T21:56:32.6349790Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T21:56:32.6350411Z nf_conntrack_netlink 57344 0 2024-08-20T21:56:32.6351157Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T21:56:32.6351951Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T21:56:32.6352537Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T21:56:32.6353077Z xfrm_user 57344 1 2024-08-20T21:56:32.6353547Z xfrm_algo 16384 1 xfrm_user 2024-08-20T21:56:32.6354088Z xt_addrtype 16384 2 2024-08-20T21:56:32.6354569Z nft_compat 20480 4 2024-08-20T21:56:32.6355122Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T21:56:32.6355860Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T21:56:32.6356549Z br_netfilter 36864 0 2024-08-20T21:56:32.6357042Z bridge 307200 1 br_netfilter 2024-08-20T21:56:32.6357587Z stp 16384 1 bridge 2024-08-20T21:56:32.6358104Z llc 16384 2 bridge,stp 2024-08-20T21:56:32.6358605Z overlay 167936 0 2024-08-20T21:56:32.6359062Z tls 114688 0 2024-08-20T21:56:32.6359523Z nls_ascii 16384 1 2024-08-20T21:56:32.6359969Z sunrpc 692224 1 2024-08-20T21:56:32.6360444Z nls_cp437 20480 1 2024-08-20T21:56:32.6360894Z vfat 24576 1 2024-08-20T21:56:32.6361327Z ata_piix 45056 0 2024-08-20T21:56:32.6361787Z fat 86016 1 vfat 2024-08-20T21:56:32.6362339Z libata 401408 1 ata_piix 2024-08-20T21:56:32.6362827Z scsi_mod 290816 1 libata 2024-08-20T21:56:32.6363351Z ghash_clmulni_intel 16384 0 2024-08-20T21:56:32.6363807Z ena 167936 0 2024-08-20T21:56:32.6364311Z scsi_common 16384 2 scsi_mod,libata 2024-08-20T21:56:32.6364868Z ptp 36864 1 ena 2024-08-20T21:56:32.6365331Z aesni_intel 393216 0 2024-08-20T21:56:32.6365780Z pps_core 24576 1 ptp 2024-08-20T21:56:32.6366254Z i8042 45056 0 2024-08-20T21:56:32.6366711Z serio 28672 3 i8042 2024-08-20T21:56:32.6367408Z crypto_simd 16384 1 aesni_intel 2024-08-20T21:56:32.6368061Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T21:56:32.6368680Z button 24576 0 2024-08-20T21:56:32.6369129Z sch_fq_codel 20480 9 2024-08-20T21:56:32.6369596Z dm_mod 188416 0 2024-08-20T21:56:32.6370048Z fuse 163840 1 2024-08-20T21:56:32.6370468Z dax 45056 1 dm_mod 2024-08-20T21:56:32.6370963Z configfs 57344 1 2024-08-20T21:56:32.6371420Z loop 36864 0 2024-08-20T21:56:32.6371856Z dmi_sysfs 20480 0 2024-08-20T21:56:32.6372313Z crc32_pclmul 16384 0 2024-08-20T21:56:32.6372773Z crc32c_intel 24576 0 2024-08-20T21:56:32.6373392Z + modinfo nvidia 2024-08-20T21:56:32.6374486Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T21:56:32.6375401Z alias: char-major-195-* 2024-08-20T21:56:32.6375870Z version: 550.54.15 2024-08-20T21:56:32.6376464Z supported: external 2024-08-20T21:56:32.6376912Z license: NVIDIA 2024-08-20T21:56:32.6377362Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T21:56:32.6377970Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T21:56:32.6378543Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T21:56:32.6379169Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T21:56:32.6379781Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T21:56:32.6380401Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T21:56:32.6381046Z depends: i2c-core,drm 2024-08-20T21:56:32.6381483Z retpoline: Y 2024-08-20T21:56:32.6381862Z name: nvidia 2024-08-20T21:56:32.6382652Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T21:56:32.6383525Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T21:56:32.6384343Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T21:56:32.6385111Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T21:56:32.6385651Z parm: NVreg_RmLogonRC:int 2024-08-20T21:56:32.6386194Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T21:56:32.6386761Z parm: NVreg_DeviceFileUID:int 2024-08-20T21:56:32.6387286Z parm: NVreg_DeviceFileGID:int 2024-08-20T21:56:32.6387839Z parm: NVreg_DeviceFileMode:int 2024-08-20T21:56:32.6388505Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T21:56:32.6389197Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T21:56:32.6389799Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T21:56:32.6390373Z parm: NVreg_EnableMSI:int 2024-08-20T21:56:32.6390870Z parm: NVreg_TCEBypassMode:int 2024-08-20T21:56:32.6391440Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T21:56:32.6392107Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T21:56:32.6392823Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T21:56:32.6393548Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T21:56:32.6394320Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T21:56:32.6395059Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T21:56:32.6395837Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T21:56:32.6396589Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T21:56:32.6397205Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T21:56:32.6397861Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T21:56:32.6398546Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T21:56:32.6399171Z parm: NVreg_MemoryPoolSize:int 2024-08-20T21:56:32.6399733Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T21:56:32.6400335Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T21:56:32.6400929Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T21:56:32.6401465Z parm: NVreg_NvLinkDisable:int 2024-08-20T21:56:32.6402218Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T21:56:32.6402883Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T21:56:32.6403463Z parm: NVreg_EnableResizableBar:int 2024-08-20T21:56:32.6404071Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T21:56:32.6404699Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T21:56:32.6405295Z parm: NVreg_RegistryDwords:charp 2024-08-20T21:56:32.6405920Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T21:56:32.6406519Z parm: NVreg_RmMsg:charp 2024-08-20T21:56:32.6407010Z parm: NVreg_GpuBlacklist:charp 2024-08-20T21:56:32.6407592Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T21:56:32.6408183Z parm: NVreg_ExcludedGpus:charp 2024-08-20T21:56:32.6408733Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T21:56:32.6409333Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T21:56:32.6409935Z parm: NVreg_ImexChannelCount:int 2024-08-20T21:56:32.6410577Z parm: rm_firmware_active:charp 2024-08-20T21:56:32.6411110Z + HAS_NVIDIA_DRIVER=0 2024-08-20T21:56:32.6411602Z ++ command -v nvidia-smi 2024-08-20T21:56:32.6412100Z + '[' -x /usr/bin/nvidia-smi ']' 2024-08-20T21:56:32.6412559Z + set +e 2024-08-20T21:56:32.6413213Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-08-20T21:56:36.8616002Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-08-20T21:56:36.8616588Z + NVIDIA_SMI_STATUS=0 2024-08-20T21:56:36.8617797Z + '[' 0 -ne 0 ']' 2024-08-20T21:56:36.8618263Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-08-20T21:56:36.8618731Z + HAS_NVIDIA_DRIVER=1 2024-08-20T21:56:36.8619692Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-08-20T21:56:36.8620641Z + set -e 2024-08-20T21:56:36.8621015Z + '[' 1 -eq 0 ']' 2024-08-20T21:56:36.8621707Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-08-20T21:56:36.8622636Z + post_install_nvidia_driver_common 2024-08-20T21:56:36.8623139Z + sudo modprobe nvidia 2024-08-20T21:56:37.0044870Z + echo 'After installing NVIDIA driver' 2024-08-20T21:56:37.0045445Z + lspci 2024-08-20T21:56:37.0045759Z After installing NVIDIA driver 2024-08-20T21:56:37.0241266Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-08-20T21:56:37.0242316Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T21:56:37.0243254Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-08-20T21:56:37.0244207Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-08-20T21:56:37.0245018Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-08-20T21:56:37.0245906Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T21:56:37.0246913Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-20T21:56:37.0247999Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-20T21:56:37.0249071Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-08-20T21:56:37.0249797Z + lsmod 2024-08-20T21:56:37.0271357Z Module Size Used by 2024-08-20T21:56:37.0271880Z nvidia_uvm 4706304 0 2024-08-20T21:56:37.0272396Z nvidia 54071296 1 nvidia_uvm 2024-08-20T21:56:37.0272926Z drm 602112 1 nvidia 2024-08-20T21:56:37.0273471Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T21:56:37.0274033Z backlight 24576 1 drm 2024-08-20T21:56:37.0274538Z i2c_core 106496 2 nvidia,drm 2024-08-20T21:56:37.0275066Z xt_conntrack 16384 1 2024-08-20T21:56:37.0275572Z nft_chain_nat 16384 3 2024-08-20T21:56:37.0276011Z xt_MASQUERADE 20480 1 2024-08-20T21:56:37.0276542Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T21:56:37.0277154Z nf_conntrack_netlink 57344 0 2024-08-20T21:56:37.0278199Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T21:56:37.0279016Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T21:56:37.0279595Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T21:56:37.0280117Z xfrm_user 57344 1 2024-08-20T21:56:37.0280650Z xfrm_algo 16384 1 xfrm_user 2024-08-20T21:56:37.0281219Z xt_addrtype 16384 2 2024-08-20T21:56:37.0281694Z nft_compat 20480 4 2024-08-20T21:56:37.0282248Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T21:56:37.0283007Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T21:56:37.0283690Z br_netfilter 36864 0 2024-08-20T21:56:37.0284181Z bridge 307200 1 br_netfilter 2024-08-20T21:56:37.0284728Z stp 16384 1 bridge 2024-08-20T21:56:37.0285248Z llc 16384 2 bridge,stp 2024-08-20T21:56:37.0285756Z overlay 167936 0 2024-08-20T21:56:37.0286378Z tls 114688 0 2024-08-20T21:56:37.0286859Z nls_ascii 16384 1 2024-08-20T21:56:37.0287298Z sunrpc 692224 1 2024-08-20T21:56:37.0287771Z nls_cp437 20480 1 2024-08-20T21:56:37.0288203Z vfat 24576 1 2024-08-20T21:56:37.0288667Z ata_piix 45056 0 2024-08-20T21:56:37.0289134Z fat 86016 1 vfat 2024-08-20T21:56:37.0289631Z libata 401408 1 ata_piix 2024-08-20T21:56:37.0290135Z scsi_mod 290816 1 libata 2024-08-20T21:56:37.0290654Z ghash_clmulni_intel 16384 0 2024-08-20T21:56:37.0291121Z ena 167936 0 2024-08-20T21:56:37.0291619Z scsi_common 16384 2 scsi_mod,libata 2024-08-20T21:56:37.0292167Z ptp 36864 1 ena 2024-08-20T21:56:37.0292611Z aesni_intel 393216 0 2024-08-20T21:56:37.0293077Z pps_core 24576 1 ptp 2024-08-20T21:56:37.0293548Z i8042 45056 0 2024-08-20T21:56:37.0294011Z serio 28672 3 i8042 2024-08-20T21:56:37.0294539Z crypto_simd 16384 1 aesni_intel 2024-08-20T21:56:37.0295178Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T21:56:37.0295800Z button 24576 0 2024-08-20T21:56:37.0296258Z sch_fq_codel 20480 9 2024-08-20T21:56:37.0296697Z dm_mod 188416 0 2024-08-20T21:56:37.0297149Z fuse 163840 1 2024-08-20T21:56:37.0297611Z dax 45056 1 dm_mod 2024-08-20T21:56:37.0298088Z configfs 57344 1 2024-08-20T21:56:37.0298542Z loop 36864 0 2024-08-20T21:56:37.0298991Z dmi_sysfs 20480 0 2024-08-20T21:56:37.0299430Z crc32_pclmul 16384 0 2024-08-20T21:56:37.0299886Z crc32c_intel 24576 0 2024-08-20T21:56:37.0300333Z + modinfo nvidia 2024-08-20T21:56:37.0301226Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T21:56:37.0302146Z alias: char-major-195-* 2024-08-20T21:56:37.0302631Z version: 550.54.15 2024-08-20T21:56:37.0303036Z supported: external 2024-08-20T21:56:37.0303456Z license: NVIDIA 2024-08-20T21:56:37.0303927Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T21:56:37.0304515Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T21:56:37.0305084Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T21:56:37.0305664Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T21:56:37.0306267Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T21:56:37.0306884Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T21:56:37.0307533Z depends: i2c-core,drm 2024-08-20T21:56:37.0307965Z retpoline: Y 2024-08-20T21:56:37.0308339Z name: nvidia 2024-08-20T21:56:37.0309131Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T21:56:37.0310007Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T21:56:37.0311000Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T21:56:37.0311756Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T21:56:37.0312291Z parm: NVreg_RmLogonRC:int 2024-08-20T21:56:37.0312829Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T21:56:37.0313395Z parm: NVreg_DeviceFileUID:int 2024-08-20T21:56:37.0313938Z parm: NVreg_DeviceFileGID:int 2024-08-20T21:56:37.0314464Z parm: NVreg_DeviceFileMode:int 2024-08-20T21:56:37.0315122Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T21:56:37.0315829Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T21:56:37.0316420Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T21:56:37.0316953Z parm: NVreg_EnableMSI:int 2024-08-20T21:56:37.0317467Z parm: NVreg_TCEBypassMode:int 2024-08-20T21:56:37.0318019Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T21:56:37.0318782Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T21:56:37.0319542Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T21:56:37.0320235Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T21:56:37.0321002Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T21:56:37.0321757Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T21:56:37.0322513Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T21:56:37.0323258Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T21:56:37.0323870Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T21:56:37.0324523Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T21:56:37.0325209Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T21:56:37.0325830Z parm: NVreg_MemoryPoolSize:int 2024-08-20T21:56:37.0326392Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T21:56:37.0326992Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T21:56:37.0327599Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T21:56:37.0328141Z parm: NVreg_NvLinkDisable:int 2024-08-20T21:56:37.0328773Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T21:56:37.0329438Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T21:56:37.0330020Z parm: NVreg_EnableResizableBar:int 2024-08-20T21:56:37.0330629Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T21:56:37.0331255Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T21:56:37.0331845Z parm: NVreg_RegistryDwords:charp 2024-08-20T21:56:37.0332474Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T21:56:37.0333076Z parm: NVreg_RmMsg:charp 2024-08-20T21:56:37.0333568Z parm: NVreg_GpuBlacklist:charp 2024-08-20T21:56:37.0334154Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T21:56:37.0334742Z parm: NVreg_ExcludedGpus:charp 2024-08-20T21:56:37.0335291Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T21:56:37.0335905Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T21:56:37.0336502Z parm: NVreg_ImexChannelCount:int 2024-08-20T21:56:37.0337521Z parm: rm_firmware_active:charp 2024-08-20T21:56:37.0338039Z + set +e 2024-08-20T21:56:37.0338476Z + nvidia-smi 2024-08-20T21:56:40.7452435Z Tue Aug 20 21:56:40 2024 2024-08-20T21:56:40.7453451Z +-----------------------------------------------------------------------------------------+ 2024-08-20T21:56:40.7454521Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T21:56:40.7455564Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T21:56:40.7456602Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T21:56:40.7457812Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T21:56:40.7458684Z | | | MIG M. | 2024-08-20T21:56:40.7459708Z |=========================================+========================+======================| 2024-08-20T21:56:40.7499978Z | 0 Tesla M60 Off | 00000000:00:1D.0 Off | 0 | 2024-08-20T21:56:40.7500866Z | N/A 26C P0 38W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-20T21:56:40.7501673Z | | | N/A | 2024-08-20T21:56:40.7502565Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T21:56:40.7546565Z | 1 Tesla M60 Off | 00000000:00:1E.0 Off | 0 | 2024-08-20T21:56:40.7547477Z | N/A 34C P0 38W / 150W | 0MiB / 7680MiB | 98% Default | 2024-08-20T21:56:40.7548268Z | | | N/A | 2024-08-20T21:56:40.7549385Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T21:56:40.7550153Z 2024-08-20T21:56:40.7551026Z +-----------------------------------------------------------------------------------------+ 2024-08-20T21:56:40.7551813Z | Processes: | 2024-08-20T21:56:40.7552711Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T21:56:40.7553591Z | ID ID Usage | 2024-08-20T21:56:40.7554305Z |=========================================================================================| 2024-08-20T21:56:40.7555128Z | No running processes found | 2024-08-20T21:56:40.7556141Z +-----------------------------------------------------------------------------------------+ 2024-08-20T21:56:41.3019154Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-08-20T21:56:43.7024382Z Tesla M60 2024-08-20T21:56:44.5802095Z + NVIDIA_SMI_STATUS=0 2024-08-20T21:56:44.5802803Z + '[' 0 -eq 0 ']' 2024-08-20T21:56:44.5803335Z + echo 'INFO: Ignoring allowed status 0' 2024-08-20T21:56:44.5803873Z + set -e 2024-08-20T21:56:44.5804233Z INFO: Ignoring allowed status 0 2024-08-20T21:56:44.5808618Z == Installing nvidia container toolkit for amzn2023 == 2024-08-20T21:56:44.5812897Z + sudo yum install -y yum-utils 2024-08-20T21:56:45.0911203Z Last metadata expiration check: 0:31:14 ago on Tue Aug 20 21:25:31 2024. 2024-08-20T21:56:45.1205506Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-08-20T21:56:45.1630570Z Dependencies resolved. 2024-08-20T21:56:45.1818046Z Nothing to do. 2024-08-20T21:56:45.1818821Z Complete! 2024-08-20T21:56:45.4219721Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-08-20T21:56:45.4221371Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T21:56:45.4223348Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T21:56:45.7714013Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T21:56:45.8502979Z + sudo yum install -y nvidia-docker2 2024-08-20T21:56:46.4920483Z nvidia-container-toolkit 8.4 kB/s | 833 B 00:00 2024-08-20T21:56:46.5227823Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-08-20T21:56:46.5649576Z Dependencies resolved. 2024-08-20T21:56:46.5835692Z Nothing to do. 2024-08-20T21:56:46.5837868Z Complete! 2024-08-20T21:56:46.7046238Z + sudo systemctl restart docker 2024-08-20T21:57:19.9057096Z Tue Aug 20 21:57:19 2024 2024-08-20T21:57:19.9058461Z +-----------------------------------------------------------------------------------------+ 2024-08-20T21:57:19.9061729Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T21:57:19.9062842Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T21:57:19.9063900Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T21:57:19.9065075Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T21:57:19.9065944Z | | | MIG M. | 2024-08-20T21:57:19.9066632Z |=========================================+========================+======================| 2024-08-20T21:57:19.9139094Z | 0 Tesla M60 On | 00000000:00:1D.0 Off | 0 | 2024-08-20T21:57:19.9140086Z | N/A 26C P0 38W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-20T21:57:19.9141125Z | | | N/A | 2024-08-20T21:57:19.9142104Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T21:57:19.9187795Z | 1 Tesla M60 On | 00000000:00:1E.0 Off | 0 | 2024-08-20T21:57:19.9188666Z | N/A 34C P0 38W / 150W | 0MiB / 7680MiB | 99% Default | 2024-08-20T21:57:19.9189527Z | | | N/A | 2024-08-20T21:57:19.9190437Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T21:57:19.9191168Z 2024-08-20T21:57:19.9191984Z +-----------------------------------------------------------------------------------------+ 2024-08-20T21:57:19.9192821Z | Processes: | 2024-08-20T21:57:19.9193718Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T21:57:19.9194622Z | ID ID Usage | 2024-08-20T21:57:19.9195301Z |=========================================================================================| 2024-08-20T21:57:19.9196123Z | No running processes found | 2024-08-20T21:57:19.9197127Z +-----------------------------------------------------------------------------------------+ 2024-08-20T21:57:20.9062758Z Command completed after 1 attempt(s). 2024-08-20T21:57:20.9175090Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T21:57:20.9176115Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T21:57:20.9176976Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-20T21:57:20.9177850Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:57:20.9189533Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:57:20.9190189Z env: 2024-08-20T21:57:20.9190544Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:57:20.9191098Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:57:20.9191717Z ##[endgroup] 2024-08-20T21:57:21.3297434Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T21:57:21.8338944Z Collecting psutil==5.9.1 2024-08-20T21:57:21.8845660Z Downloading psutil-5.9.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2024-08-20T21:57:21.9477326Z Collecting nvidia-ml-py==11.525.84 2024-08-20T21:57:21.9521500Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2024-08-20T21:57:22.0728892Z Installing collected packages: psutil, nvidia-ml-py 2024-08-20T21:57:22.2820800Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1 2024-08-20T21:57:22.5189661Z Prepare all required actions 2024-08-20T21:57:22.5190340Z Getting action download info 2024-08-20T21:57:22.6507175Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-20T21:57:22.8282137Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-20T21:57:22.9622537Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-20T21:57:22.9623135Z with: 2024-08-20T21:57:22.9623504Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-08-20T21:57:22.9624019Z s3-bucket: gha-artifacts 2024-08-20T21:57:22.9624415Z env: 2024-08-20T21:57:22.9624721Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:57:22.9625248Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:57:22.9625831Z ##[endgroup] 2024-08-20T21:57:22.9672201Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T21:57:22.9672745Z with: 2024-08-20T21:57:22.9673186Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-08-20T21:57:22.9673730Z s3-bucket: gha-artifacts 2024-08-20T21:57:22.9674162Z region: us-east-1 2024-08-20T21:57:22.9674523Z env: 2024-08-20T21:57:22.9674829Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:57:22.9675383Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:57:22.9675981Z ##[endgroup] 2024-08-20T21:57:23.5750204Z (node:59942) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T21:57:23.5751097Z 2024-08-20T21:57:23.5751421Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T21:57:23.5752344Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T21:57:23.5753472Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T21:57:23.6624582Z Found 1 objects with prefix pytorch/pytorch/10479309237/linux-focal-cuda11.8-py3.10-gcc9/ 2024-08-20T21:57:23.6625960Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T21:57:46.7416754Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T21:57:46.7429938Z Artifact download has finished successfully 2024-08-20T21:57:46.7674599Z ##[group]Run unzip -o artifacts.zip 2024-08-20T21:57:46.7675141Z unzip -o artifacts.zip 2024-08-20T21:57:46.7683683Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:57:46.7684315Z env: 2024-08-20T21:57:46.7684647Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:57:46.7685192Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:57:46.7685796Z ##[endgroup] 2024-08-20T21:57:46.7765537Z Archive: artifacts.zip 2024-08-20T21:57:46.7767107Z creating: dist/ 2024-08-20T21:57:49.2182461Z inflating: dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T21:57:49.2183339Z creating: build/custom_test_artifacts/ 2024-08-20T21:57:49.2184050Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-20T21:57:49.2185032Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-20T21:57:49.2186133Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-20T21:57:49.2193123Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T21:57:49.2194371Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-20T21:57:49.2195571Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T21:57:49.2196861Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T21:57:49.2198199Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T21:57:49.2199621Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T21:57:49.2201416Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T21:57:49.2202919Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T21:57:49.2204235Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T21:57:49.2205784Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T21:57:49.2207315Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T21:57:49.2208826Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T21:57:49.2210332Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T21:57:49.2211865Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T21:57:49.2213420Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T21:57:49.2214781Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T21:57:49.2216118Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T21:57:49.2269722Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T21:57:49.2271644Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T21:57:49.2273510Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T21:57:49.2275526Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T21:57:49.2277408Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T21:57:49.2279261Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T21:57:49.2281089Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T21:57:49.2282952Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T21:57:49.2284788Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T21:57:49.2325002Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T21:57:49.2373047Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T21:57:49.2374852Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T21:57:49.2376582Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T21:57:49.2378221Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T21:57:49.2379820Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T21:57:49.2381455Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T21:57:49.2383034Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T21:57:49.2384654Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T21:57:49.2455385Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T21:57:49.2528602Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T21:57:49.2530196Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T21:57:49.2531505Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-20T21:57:49.2532641Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-20T21:57:49.2533863Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-20T21:57:49.2535096Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-20T21:57:49.2536426Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-20T21:57:49.2538254Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-20T21:57:49.2539703Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-20T21:57:49.2541128Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-20T21:57:49.2542534Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-20T21:57:49.2543965Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-20T21:57:49.2545401Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-20T21:57:49.2546804Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-20T21:57:49.2548217Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-20T21:57:49.2564809Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-20T21:57:49.2736277Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-20T21:57:49.2737836Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-20T21:57:49.2739294Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-20T21:57:49.2740943Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-20T21:57:49.2742499Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-20T21:57:49.2743914Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-20T21:57:49.2745423Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-20T21:57:49.2746934Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-20T21:57:49.2748457Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-20T21:57:49.2749939Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-20T21:57:49.2751417Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-20T21:57:49.2770975Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-20T21:57:49.2870765Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-20T21:57:49.2872341Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T21:57:49.2873752Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-20T21:57:49.2875245Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-20T21:57:49.2876884Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-20T21:57:49.2878085Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-20T21:57:49.2879282Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-08-20T21:57:49.2880407Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-20T21:57:49.2881441Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-20T21:57:49.2882496Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-20T21:57:49.3026244Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-20T21:57:49.3102397Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-20T21:57:49.3103332Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-20T21:57:49.3104266Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-20T21:57:49.3105382Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-20T21:57:49.3112271Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T21:57:49.3113507Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-20T21:57:49.3114692Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T21:57:49.3115995Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T21:57:49.3117290Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T21:57:49.3118695Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T21:57:49.3120179Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T21:57:49.3121465Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T21:57:49.3122796Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T21:57:49.3124281Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T21:57:49.3125797Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T21:57:49.3127456Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T21:57:49.3128964Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T21:57:49.3130511Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T21:57:49.3132493Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T21:57:49.3133949Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T21:57:49.3135266Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T21:57:49.3189181Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T21:57:49.3191067Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T21:57:49.3192967Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T21:57:49.3194894Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T21:57:49.3196810Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T21:57:49.3199021Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T21:57:49.3200885Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T21:57:49.3202719Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T21:57:49.3204551Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T21:57:49.3245856Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T21:57:49.3293379Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T21:57:49.3295212Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T21:57:49.3296928Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T21:57:49.3298553Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T21:57:49.3300162Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T21:57:49.3301769Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T21:57:49.3303344Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T21:57:49.3304952Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T21:57:49.3375691Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T21:57:49.3449376Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T21:57:49.3450956Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T21:57:49.3452260Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-20T21:57:49.3453393Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-20T21:57:49.3454561Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-20T21:57:49.3455798Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-20T21:57:49.3457196Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-20T21:57:49.3458762Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-20T21:57:49.3460259Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-20T21:57:49.3461683Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-20T21:57:49.3463145Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-20T21:57:49.3464591Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-20T21:57:49.3466058Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-20T21:57:49.3467519Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-20T21:57:49.3468966Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-20T21:57:49.3485139Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-20T21:57:49.3562489Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-20T21:57:49.3564200Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T21:57:49.3565615Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-20T21:57:49.3566875Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-20T21:57:49.3568055Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-20T21:57:49.3569225Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-20T21:57:49.3570386Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-08-20T21:57:49.3571502Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-20T21:57:49.3572785Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-20T21:57:49.3573853Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-20T21:57:49.3633258Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-20T21:57:49.3634264Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-20T21:57:49.3635224Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-20T21:57:49.3636369Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-20T21:57:49.3643882Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T21:57:49.3645164Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-20T21:57:49.3646480Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T21:57:49.3647897Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T21:57:49.3649231Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T21:57:49.3650802Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T21:57:49.3652324Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T21:57:49.3653738Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T21:57:49.3655120Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T21:57:49.3656726Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T21:57:49.3658330Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T21:57:49.3659928Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T21:57:49.3661543Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T21:57:49.3663163Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T21:57:49.3664821Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T21:57:49.3666237Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T21:57:49.3667660Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T21:57:49.3720877Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T21:57:49.3722852Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T21:57:49.3725062Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T21:57:49.3727255Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T21:57:49.3729273Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T21:57:49.3731182Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T21:57:49.3733122Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T21:57:49.3735066Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T21:57:49.3737006Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T21:57:49.3776324Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T21:57:49.3823813Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T21:57:49.3825754Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T21:57:49.3827550Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T21:57:49.3829305Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T21:57:49.3830950Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T21:57:49.3832673Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T21:57:49.3834352Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T21:57:49.3836038Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T21:57:49.3906344Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T21:57:49.3980217Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T21:57:49.3981883Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T21:57:49.3983293Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-20T21:57:49.3984500Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-20T21:57:49.3985800Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-20T21:57:49.3987143Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-20T21:57:49.3988594Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-20T21:57:49.3990296Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-20T21:57:49.3991880Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-20T21:57:49.3993393Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-20T21:57:49.3994936Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-20T21:57:49.3996686Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-20T21:57:49.3998411Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-20T21:57:49.3999964Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-20T21:57:49.4001522Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-20T21:57:49.4003184Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-20T21:57:49.4147769Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-20T21:57:49.4149278Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-20T21:57:49.4150890Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-20T21:57:49.4152653Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-20T21:57:49.4154348Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-20T21:57:49.4155927Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-20T21:57:49.4157533Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-20T21:57:49.4159179Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-20T21:57:49.4160829Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-20T21:57:49.4162469Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-20T21:57:49.4164107Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-20T21:57:49.4180953Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-20T21:57:49.4247541Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-20T21:57:49.4249292Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T21:57:49.4250813Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-20T21:57:49.4252172Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-20T21:57:49.4253454Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-20T21:57:49.4254668Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-20T21:57:49.4255983Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-08-20T21:57:49.4257163Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-20T21:57:49.4258271Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-20T21:57:49.4259399Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-20T21:57:49.4384056Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-20T21:57:49.4435724Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-20T21:57:49.4436480Z creating: build/lib/ 2024-08-20T21:57:49.4448027Z inflating: build/lib/libpthreadpool.a 2024-08-20T21:57:49.4449109Z inflating: build/lib/libclog.a 2024-08-20T21:57:49.4452451Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-20T21:57:49.4462432Z inflating: build/lib/libcpuinfo.a 2024-08-20T21:57:49.4472096Z inflating: build/lib/libcpuinfo_internals.a 2024-08-20T21:57:49.4579145Z inflating: build/lib/libprotobuf-lite.a 2024-08-20T21:57:49.5125507Z inflating: build/lib/libprotobuf.a 2024-08-20T21:57:49.5204082Z inflating: build/lib/libgtest.a 2024-08-20T21:57:49.5294526Z inflating: build/lib/libbenchmark.a 2024-08-20T21:57:49.5303941Z inflating: build/lib/libittnotify.a 2024-08-20T21:57:49.5381383Z inflating: build/lib/libasmjit.a 2024-08-20T21:57:49.5415064Z inflating: build/lib/libtensorpipe_uv.a 2024-08-20T21:57:49.5568299Z inflating: build/lib/libgloo.a 2024-08-20T21:57:49.5593428Z inflating: build/lib/libfmt.a 2024-08-20T21:57:49.5595778Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-20T21:57:49.5710737Z inflating: build/lib/libc10.so 2024-08-20T21:57:49.5712630Z inflating: build/lib/libtorch_global_deps.so 2024-08-20T21:57:49.6317916Z inflating: build/lib/libprotoc.a 2024-08-20T21:57:49.6342070Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-20T21:57:49.6363773Z inflating: build/lib/libgmock.a 2024-08-20T21:57:49.6364619Z inflating: build/lib/libgtest_main.a 2024-08-20T21:57:49.6365699Z inflating: build/lib/libbenchmark_main.a 2024-08-20T21:57:50.8480187Z inflating: build/lib/libdnnl.a 2024-08-20T21:57:50.8501827Z inflating: build/lib/libnnpack.a 2024-08-20T21:57:50.9188106Z inflating: build/lib/libtensorpipe.a 2024-08-20T21:57:50.9258179Z inflating: build/lib/libc10_cuda.so 2024-08-20T21:57:51.0755492Z inflating: build/lib/libfbgemm.a 2024-08-20T21:57:51.0756307Z inflating: build/lib/libgmock_main.a 2024-08-20T21:57:51.1368114Z inflating: build/lib/libkineto.a 2024-08-20T21:57:51.1674771Z inflating: build/lib/libtensorpipe_cuda.a 2024-08-20T21:57:51.1898123Z inflating: build/lib/libXNNPACK.a 2024-08-20T21:57:51.2359331Z inflating: build/lib/libgloo_cuda.a 2024-08-20T21:57:51.2410175Z inflating: build/lib/libonnx_proto.a 2024-08-20T21:57:51.3241877Z inflating: build/lib/libonnx.a 2024-08-20T21:57:54.2649342Z inflating: build/lib/libtorch_cpu.so 2024-08-20T21:57:54.2654205Z inflating: build/lib/libshm.so 2024-08-20T21:57:54.2660148Z inflating: build/lib/libunbox_lib.a 2024-08-20T21:57:56.6893526Z inflating: build/lib/libtorch_cuda.so 2024-08-20T21:57:56.6895146Z inflating: build/lib/libtorch.so 2024-08-20T21:57:56.6898530Z inflating: build/lib/libc10d_cuda_test.so 2024-08-20T21:57:57.8129013Z inflating: build/lib/libtorch_cuda_linalg.so 2024-08-20T21:57:57.8154666Z inflating: build/lib/libjitbackend_test.so 2024-08-20T21:57:57.8242124Z inflating: build/lib/libtorchbind_test.so 2024-08-20T21:57:57.8274345Z inflating: build/lib/libaoti_custom_ops.so 2024-08-20T21:57:57.8305148Z inflating: build/lib/libbackend_with_compiler.so 2024-08-20T21:57:58.0713714Z inflating: build/lib/libtorch_python.so 2024-08-20T21:57:58.0756255Z inflating: build/lib/libnnapi_backend.so 2024-08-20T21:57:58.0756831Z creating: build/bin/ 2024-08-20T21:57:58.0817603Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-20T21:57:58.0880546Z inflating: build/bin/c10_DeviceGuard_test 2024-08-20T21:57:58.0942982Z inflating: build/bin/c10_Device_test 2024-08-20T21:57:58.1014666Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-20T21:57:58.1074710Z inflating: build/bin/c10_StreamGuard_test 2024-08-20T21:57:58.1139967Z inflating: build/bin/c10_Scalar_test 2024-08-20T21:57:58.1202103Z inflating: build/bin/c10_SymInt_test 2024-08-20T21:57:58.1269043Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-20T21:57:58.1336855Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-20T21:57:58.1405555Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-20T21:57:58.1492466Z inflating: build/bin/c10_cow_test 2024-08-20T21:57:58.1557370Z inflating: build/bin/c10_Bitset_test 2024-08-20T21:57:58.1616812Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-20T21:57:58.1677984Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-20T21:57:58.1739711Z inflating: build/bin/c10_Half_test 2024-08-20T21:57:58.1808198Z inflating: build/bin/c10_LeftRight_test 2024-08-20T21:57:58.1875652Z inflating: build/bin/c10_Metaprogramming_test 2024-08-20T21:57:58.1936382Z inflating: build/bin/c10_Synchronized_test 2024-08-20T21:57:58.2004185Z inflating: build/bin/c10_ThreadLocal_test 2024-08-20T21:57:58.2067162Z inflating: build/bin/c10_TypeIndex_test 2024-08-20T21:57:58.2129169Z inflating: build/bin/c10_TypeList_test 2024-08-20T21:57:58.2189576Z inflating: build/bin/c10_TypeTraits_test 2024-08-20T21:57:58.2252837Z inflating: build/bin/c10_accumulate_test 2024-08-20T21:57:58.2320257Z inflating: build/bin/c10_bfloat16_test 2024-08-20T21:57:58.2381996Z inflating: build/bin/c10_bit_cast_test 2024-08-20T21:57:58.2450868Z inflating: build/bin/c10_complex_math_test 2024-08-20T21:57:58.2514996Z inflating: build/bin/c10_exception_test 2024-08-20T21:57:58.2583153Z inflating: build/bin/c10_complex_test 2024-08-20T21:57:58.2644748Z inflating: build/bin/c10_flags_test 2024-08-20T21:57:58.2705667Z inflating: build/bin/c10_generic_math_test 2024-08-20T21:57:58.2904229Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-20T21:57:58.2966361Z inflating: build/bin/c10_irange_test 2024-08-20T21:57:58.3031456Z inflating: build/bin/c10_lazy_test 2024-08-20T21:57:58.3101345Z inflating: build/bin/c10_logging_test 2024-08-20T21:57:58.3193157Z inflating: build/bin/c10_optional_test 2024-08-20T21:57:58.3269354Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-20T21:57:58.3334981Z inflating: build/bin/c10_registry_test 2024-08-20T21:57:58.3517267Z inflating: build/bin/c10_small_vector_test 2024-08-20T21:57:58.3580364Z inflating: build/bin/c10_ssize_test 2024-08-20T21:57:58.3644095Z inflating: build/bin/c10_string_util_test 2024-08-20T21:57:58.3715451Z inflating: build/bin/c10_string_view_test 2024-08-20T21:57:58.3777179Z inflating: build/bin/c10_tempfile_test 2024-08-20T21:57:58.3845706Z inflating: build/bin/c10_typeid_test 2024-08-20T21:57:58.3904579Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-20T21:57:58.4447284Z inflating: build/bin/protoc-3.13.0.0 2024-08-20T21:57:58.4988787Z inflating: build/bin/protoc 2024-08-20T21:57:58.5053492Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-08-20T21:57:58.5118096Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-08-20T21:57:58.5182271Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-08-20T21:57:58.5245721Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-08-20T21:57:58.5305701Z inflating: build/bin/c10_cuda_CUDATest 2024-08-20T21:57:58.5370366Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-20T21:57:58.5434808Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-20T21:57:58.5499297Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-08-20T21:57:58.5931658Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-20T21:57:58.6345825Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-20T21:57:58.6743550Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-20T21:57:58.6807879Z inflating: build/bin/FileStoreTest 2024-08-20T21:57:58.6876053Z inflating: build/bin/TCPStoreTest 2024-08-20T21:57:58.6941146Z inflating: build/bin/HashStoreTest 2024-08-20T21:57:58.7005512Z inflating: build/bin/BackoffTest 2024-08-20T21:57:58.7021899Z inflating: build/bin/ProcessGroupMPITest 2024-08-20T21:57:58.7027334Z inflating: build/bin/torch_shm_manager 2024-08-20T21:57:58.7093276Z inflating: build/bin/test_edge_op_registration 2024-08-20T21:57:58.7096886Z inflating: build/bin/example_allreduce 2024-08-20T21:57:58.7161046Z inflating: build/bin/Dimname_test 2024-08-20T21:57:58.7251069Z inflating: build/bin/Dict_test 2024-08-20T21:57:58.7329284Z inflating: build/bin/MaybeOwned_test 2024-08-20T21:57:58.7399406Z inflating: build/bin/NamedTensor_test 2024-08-20T21:57:58.7471875Z inflating: build/bin/apply_utils_test 2024-08-20T21:57:58.7543875Z inflating: build/bin/atest 2024-08-20T21:57:58.7620561Z inflating: build/bin/basic 2024-08-20T21:57:58.7687631Z inflating: build/bin/broadcast_test 2024-08-20T21:57:58.7758701Z inflating: build/bin/cpu_generator_test 2024-08-20T21:57:58.7820497Z inflating: build/bin/cpu_allocator_test 2024-08-20T21:57:58.7885972Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-20T21:57:58.7947573Z inflating: build/bin/dlconvertor_test 2024-08-20T21:57:58.8008302Z inflating: build/bin/dispatch_key_set_test 2024-08-20T21:57:58.8121464Z inflating: build/bin/cpu_rng_test 2024-08-20T21:57:58.8192636Z inflating: build/bin/extension_backend_test 2024-08-20T21:57:58.8260122Z inflating: build/bin/half_test 2024-08-20T21:57:58.8376720Z inflating: build/bin/ivalue_test 2024-08-20T21:57:58.8437470Z inflating: build/bin/lazy_tensor_test 2024-08-20T21:57:58.8504191Z inflating: build/bin/math_kernel_test 2024-08-20T21:57:58.8570320Z inflating: build/bin/memory_format_test 2024-08-20T21:57:58.8635096Z inflating: build/bin/memory_overlapping_test 2024-08-20T21:57:58.8699988Z inflating: build/bin/mobile_memory_cleanup 2024-08-20T21:57:58.8768686Z inflating: build/bin/native_test 2024-08-20T21:57:58.8830275Z inflating: build/bin/operator_name_test 2024-08-20T21:57:58.8894039Z inflating: build/bin/packedtensoraccessor_test 2024-08-20T21:57:58.8957058Z inflating: build/bin/operators_test 2024-08-20T21:57:58.9038923Z inflating: build/bin/pow_test 2024-08-20T21:57:58.9101240Z inflating: build/bin/reportMemoryUsage_test 2024-08-20T21:57:58.9170922Z inflating: build/bin/quantized_test 2024-08-20T21:57:58.9231717Z inflating: build/bin/reduce_ops_test 2024-08-20T21:57:58.9300960Z inflating: build/bin/scalar_tensor_test 2024-08-20T21:57:58.9372657Z inflating: build/bin/scalar_test 2024-08-20T21:57:58.9435677Z inflating: build/bin/StorageUtils_test 2024-08-20T21:57:58.9500628Z inflating: build/bin/stride_properties_test 2024-08-20T21:57:58.9596456Z inflating: build/bin/tensor_iterator_test 2024-08-20T21:57:58.9664612Z inflating: build/bin/type_ptr_test 2024-08-20T21:57:58.9730354Z inflating: build/bin/test_parallel 2024-08-20T21:57:58.9734169Z inflating: build/bin/thread_init_test 2024-08-20T21:57:58.9798192Z inflating: build/bin/undefined_tensor_test 2024-08-20T21:57:58.9871326Z inflating: build/bin/type_test 2024-08-20T21:57:58.9873426Z inflating: build/bin/verify_api_visibility 2024-08-20T21:57:58.9936298Z inflating: build/bin/weakref_test 2024-08-20T21:57:59.0021223Z inflating: build/bin/legacy_vmap_test 2024-08-20T21:57:59.0084511Z inflating: build/bin/wrapdim_test 2024-08-20T21:57:59.0158005Z inflating: build/bin/IListRef_test 2024-08-20T21:57:59.0220793Z inflating: build/bin/xla_tensor_test 2024-08-20T21:57:59.0350379Z inflating: build/bin/List_test 2024-08-20T21:57:59.0498247Z inflating: build/bin/kernel_function_legacy_test 2024-08-20T21:57:59.0615775Z inflating: build/bin/kernel_function_test 2024-08-20T21:57:59.0697065Z inflating: build/bin/KernelFunction_test 2024-08-20T21:57:59.0852850Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-20T21:57:59.0978335Z inflating: build/bin/kernel_lambda_test 2024-08-20T21:57:59.1053321Z inflating: build/bin/kernel_stackbased_test 2024-08-20T21:57:59.1115394Z inflating: build/bin/CppSignature_test 2024-08-20T21:57:59.1232823Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-20T21:57:59.1292507Z inflating: build/bin/op_allowlist_test 2024-08-20T21:57:59.1360831Z inflating: build/bin/backend_fallback_test 2024-08-20T21:57:59.1438739Z inflating: build/bin/inline_container_test 2024-08-20T21:57:59.1802036Z inflating: build/bin/op_registration_test 2024-08-20T21:57:59.1866586Z inflating: build/bin/cuda_apply_test 2024-08-20T21:57:59.1929725Z inflating: build/bin/cuda_allocator_test 2024-08-20T21:57:59.2001927Z inflating: build/bin/cuda_atomic_ops_test 2024-08-20T21:57:59.2068906Z inflating: build/bin/cuda_caching_host_allocator_test 2024-08-20T21:57:59.2154650Z inflating: build/bin/cuda_complex_math_test 2024-08-20T21:57:59.2226381Z inflating: build/bin/cuda_complex_test 2024-08-20T21:57:59.2287333Z inflating: build/bin/cuda_device_test 2024-08-20T21:57:59.2358159Z inflating: build/bin/cuda_cub_test 2024-08-20T21:57:59.2419797Z inflating: build/bin/cuda_dlconvertor_test 2024-08-20T21:57:59.2482280Z inflating: build/bin/cuda_integer_divider_test 2024-08-20T21:57:59.2562365Z inflating: build/bin/cuda_distributions_test 2024-08-20T21:57:59.2631312Z inflating: build/bin/cuda_generator_test 2024-08-20T21:57:59.2692686Z inflating: build/bin/cuda_half_test 2024-08-20T21:57:59.2753027Z inflating: build/bin/cuda_optional_test 2024-08-20T21:57:59.2813930Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-08-20T21:57:59.2888026Z inflating: build/bin/cuda_stream_test 2024-08-20T21:57:59.2952473Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-08-20T21:57:59.3015035Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-08-20T21:57:59.3075556Z inflating: build/bin/cuda_cudnn_test 2024-08-20T21:57:59.3139193Z inflating: build/bin/cuda_vectorized_test 2024-08-20T21:57:59.3218825Z inflating: build/bin/ProcessGroupGlooTest 2024-08-20T21:57:59.3236455Z inflating: build/bin/tutorial_tensorexpr 2024-08-20T21:57:59.3306666Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-08-20T21:57:59.3384425Z inflating: build/bin/ProcessGroupNCCLTest 2024-08-20T21:57:59.3448708Z inflating: build/bin/ProcessGroupUCCTest 2024-08-20T21:57:59.3524268Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-08-20T21:57:59.3591603Z inflating: build/bin/test_dist_autograd 2024-08-20T21:57:59.3675812Z inflating: build/bin/test_cpp_rpc 2024-08-20T21:57:59.3679032Z inflating: build/bin/parallel_benchmark 2024-08-20T21:57:59.3690093Z inflating: build/bin/aot_model_compiler_test 2024-08-20T21:57:59.3772004Z inflating: build/bin/test_mobile_nnc 2024-08-20T21:57:59.4189336Z inflating: build/bin/test_lazy 2024-08-20T21:57:59.5190824Z inflating: build/bin/test_tensorexpr 2024-08-20T21:57:59.6619575Z inflating: build/bin/test_api 2024-08-20T21:57:59.7322780Z inflating: build/bin/test_jit 2024-08-20T21:57:59.7323358Z creating: .additional_ci_files/ 2024-08-20T21:57:59.7396732Z inflating: .additional_ci_files/test-times.json 2024-08-20T21:57:59.7684876Z inflating: .additional_ci_files/test-class-times.json 2024-08-20T21:57:59.7726081Z ##[group]Run rm artifacts.zip 2024-08-20T21:57:59.7726573Z rm artifacts.zip 2024-08-20T21:57:59.7734442Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:57:59.7735071Z env: 2024-08-20T21:57:59.7735403Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:57:59.7735949Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:57:59.7736550Z ##[endgroup] 2024-08-20T21:57:59.8438109Z ##[group]Run df -H 2024-08-20T21:57:59.8438507Z df -H 2024-08-20T21:57:59.8445157Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:57:59.8445773Z env: 2024-08-20T21:57:59.8446108Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:57:59.8446673Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:57:59.8447265Z ##[endgroup] 2024-08-20T21:57:59.8494564Z Filesystem Size Used Avail Use% Mounted on 2024-08-20T21:57:59.8495198Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-08-20T21:57:59.8496866Z tmpfs 129G 0 129G 0% /dev/shm 2024-08-20T21:57:59.8497545Z tmpfs 52G 635k 52G 1% /run 2024-08-20T21:57:59.8498551Z /dev/xvda1 161G 47G 115G 30% / 2024-08-20T21:57:59.8499121Z tmpfs 129G 8.2k 129G 1% /tmp 2024-08-20T21:57:59.8499694Z /dev/xvda128 11M 1.4M 9.2M 13% /boot/efi 2024-08-20T21:57:59.8503741Z tmpfs 26G 0 26G 0% /run/user/0 2024-08-20T21:57:59.8555529Z Prepare all required actions 2024-08-20T21:57:59.8556190Z Getting action download info 2024-08-20T21:57:59.9682645Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-20T21:57:59.9683209Z with: 2024-08-20T21:57:59.9683517Z env: 2024-08-20T21:57:59.9683849Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:57:59.9684396Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:57:59.9684995Z ##[endgroup] 2024-08-20T21:57:59.9728548Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T21:57:59.9729111Z with: 2024-08-20T21:57:59.9729434Z name: td_results 2024-08-20T21:57:59.9729805Z s3-bucket: gha-artifacts 2024-08-20T21:57:59.9730236Z region: us-east-1 2024-08-20T21:57:59.9730595Z env: 2024-08-20T21:57:59.9730917Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:57:59.9731476Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:57:59.9732077Z ##[endgroup] 2024-08-20T21:58:00.5871821Z (node:59969) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T21:58:00.5872742Z 2024-08-20T21:58:00.5873087Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T21:58:00.5873988Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T21:58:00.5875108Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T21:58:00.6705014Z Found 1 objects with prefix pytorch/pytorch/10479309237/td_results/ 2024-08-20T21:58:00.6706250Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-08-20T21:58:00.8235981Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-08-20T21:58:00.8246930Z Artifact download has finished successfully 2024-08-20T21:58:00.8456136Z ##[group]Run mkdir -p .additional_ci_files 2024-08-20T21:58:00.8456744Z mkdir -p .additional_ci_files 2024-08-20T21:58:00.8457465Z mv td_results.json .additional_ci_files/td_results.json 2024-08-20T21:58:00.8466127Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:58:00.8466756Z env: 2024-08-20T21:58:00.8467097Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:00.8467641Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:00.8468247Z ##[endgroup] 2024-08-20T21:58:00.8566558Z ##[group]Run .github/scripts/parse_ref.py 2024-08-20T21:58:00.8567201Z .github/scripts/parse_ref.py 2024-08-20T21:58:00.8573993Z shell: /usr/bin/bash -e {0} 2024-08-20T21:58:00.8574434Z env: 2024-08-20T21:58:00.8574767Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:00.8575314Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:00.8575918Z ##[endgroup] 2024-08-20T21:58:00.8933556Z Prepare all required actions 2024-08-20T21:58:00.8986246Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-20T21:58:00.8986824Z with: 2024-08-20T21:58:00.8987546Z github-token: *** 2024-08-20T21:58:00.8987909Z env: 2024-08-20T21:58:00.8988242Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:00.8988804Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:00.8989394Z ##[endgroup] 2024-08-20T21:58:00.9013514Z ##[group]Run set -eux 2024-08-20T21:58:00.9013932Z set -eux 2024-08-20T21:58:00.9014665Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-20T21:58:00.9021471Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:58:00.9022077Z env: 2024-08-20T21:58:00.9022411Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:00.9022977Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:00.9023780Z GITHUB_TOKEN: *** 2024-08-20T21:58:00.9024163Z ##[endgroup] 2024-08-20T21:58:00.9052030Z + python3 .github/scripts/get_workflow_job_id.py 10479309237 i-050c06cb9ff8794b9 2024-08-20T21:58:02.9813587Z setting job-id=29025850949 2024-08-20T21:58:02.9814833Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, amz2023.linux.8xlarge.nvidia.gpu) 2024-08-20T21:58:03.0099653Z Prepare all required actions 2024-08-20T21:58:03.0100253Z Getting action download info 2024-08-20T21:58:03.1221490Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-20T21:58:03.1222070Z with: 2024-08-20T21:58:03.1222658Z github-token: *** 2024-08-20T21:58:03.1224657Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}]} 2024-08-20T21:58:03.1227203Z job-name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, amz2023.linux.8xlarge.nvidia.gpu) 2024-08-20T21:58:03.1228192Z env: 2024-08-20T21:58:03.1228525Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:03.1229072Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:03.1229679Z ##[endgroup] 2024-08-20T21:58:03.1309448Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T21:58:03.1310169Z with: 2024-08-20T21:58:03.1310472Z shell: bash 2024-08-20T21:58:03.1310837Z timeout_minutes: 10 2024-08-20T21:58:03.1311236Z max_attempts: 5 2024-08-20T21:58:03.1311602Z retry_wait_seconds: 30 2024-08-20T21:58:03.1313023Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T21:58:03.1314537Z polling_interval_seconds: 1 2024-08-20T21:58:03.1314987Z warning_on_retry: true 2024-08-20T21:58:03.1315422Z continue_on_error: false 2024-08-20T21:58:03.1315841Z env: 2024-08-20T21:58:03.1316161Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:03.1316719Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:03.1317550Z GITHUB_TOKEN: *** 2024-08-20T21:58:03.1317920Z ##[endgroup] 2024-08-20T21:58:03.2237309Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T21:58:03.5498387Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T21:58:03.7352892Z Collecting requests==2.27.1 2024-08-20T21:58:03.7814716Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2024-08-20T21:58:04.0136211Z Collecting pyyaml==6.0.1 2024-08-20T21:58:04.0179059Z Downloading PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (738 kB) 2024-08-20T21:58:04.0599152Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-08-20T21:58:04.0613367Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-08-20T21:58:04.1172453Z Collecting certifi>=2017.4.17 2024-08-20T21:58:04.1215639Z Downloading certifi-2024.7.4-py3-none-any.whl (162 kB) 2024-08-20T21:58:04.4878839Z Collecting charset-normalizer~=2.0.0 2024-08-20T21:58:04.4921508Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-08-20T21:58:04.6050308Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2024-08-20T21:58:04.7665575Z Successfully installed certifi-2024.7.4 charset-normalizer-2.0.12 pyyaml-6.0.1 requests-2.27.1 2024-08-20T21:58:05.2036269Z Command completed after 1 attempt(s). 2024-08-20T21:58:05.2094756Z ##[group]Run set -x 2024-08-20T21:58:05.2095163Z set -x 2024-08-20T21:58:05.2095521Z  2024-08-20T21:58:05.2096212Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T21:58:05.2097061Z # in runner workspace 2024-08-20T21:58:05.2097725Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-20T21:58:05.2105969Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:58:05.2106586Z env: 2024-08-20T21:58:05.2106931Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:05.2107522Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:05.2108321Z ##[endgroup] 2024-08-20T21:58:05.2140089Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-20T21:58:05.2456408Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T21:58:05.2457079Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T21:58:05.2457653Z echo "Job name: ${JOB_NAME}" 2024-08-20T21:58:05.2458146Z  2024-08-20T21:58:05.2458805Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T21:58:05.2459654Z # in runner workspace 2024-08-20T21:58:05.2460372Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-20T21:58:05.2461184Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-20T21:58:05.2461733Z  --job-name "${JOB_NAME}" \ 2024-08-20T21:58:05.2463898Z  --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}]}" \ 2024-08-20T21:58:05.2466087Z  --selected-test-configs "" \ 2024-08-20T21:58:05.2466655Z  --pr-number "${PR_NUMBER}" \ 2024-08-20T21:58:05.2467154Z  --tag "${TAG}" \ 2024-08-20T21:58:05.2467632Z  --event-name "${EVENT_NAME}" \ 2024-08-20T21:58:05.2468180Z  --schedule "${SCHEDULE}" \ 2024-08-20T21:58:05.2468691Z  --branch "${HEAD_BRANCH}" 2024-08-20T21:58:05.2475492Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:58:05.2476128Z env: 2024-08-20T21:58:05.2476449Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:05.2477008Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:05.2477861Z GITHUB_TOKEN: *** 2024-08-20T21:58:05.2478720Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, amz2023.linux.8xlarge.nvidia.gpu) 2024-08-20T21:58:05.2479694Z PR_NUMBER: 133712 2024-08-20T21:58:05.2480064Z TAG: 2024-08-20T21:58:05.2480386Z EVENT_NAME: pull_request 2024-08-20T21:58:05.2480812Z SCHEDULE: 2024-08-20T21:58:05.2481160Z HEAD_BRANCH: 2024-08-20T21:58:05.2481496Z ##[endgroup] 2024-08-20T21:58:05.2509557Z Workflow: pull 2024-08-20T21:58:05.2510652Z Job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, amz2023.linux.8xlarge.nvidia.gpu) 2024-08-20T21:58:05.5236416Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-20T21:58:05.6962620Z ##[group]Run echo "Filtered matrix:" 2024-08-20T21:58:05.6963175Z echo "Filtered matrix:" 2024-08-20T21:58:05.6965292Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}]}" 2024-08-20T21:58:05.6967348Z  2024-08-20T21:58:05.6967660Z echo 2024-08-20T21:58:05.6968120Z echo "Is the current job unstable? False" 2024-08-20T21:58:05.6968686Z  2024-08-20T21:58:05.6968994Z echo 2024-08-20T21:58:05.6969435Z echo "Is keep-going label set? False" 2024-08-20T21:58:05.6969981Z  2024-08-20T21:58:05.6970284Z echo 2024-08-20T21:58:05.6970666Z echo "Renabled issues? " 2024-08-20T21:58:05.6977522Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:58:05.6978127Z env: 2024-08-20T21:58:05.6978461Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:05.6979211Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:05.6979797Z ##[endgroup] 2024-08-20T21:58:05.7007900Z Filtered matrix: 2024-08-20T21:58:05.7010293Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: amz2023.linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 2, num_shards: 3, runner: amz2023.linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 3, num_shards: 3, runner: amz2023.linux.8xlarge.nvidia.gpu}]} 2024-08-20T21:58:05.7012172Z 2024-08-20T21:58:05.7012354Z Is the current job unstable? False 2024-08-20T21:58:05.7012722Z 2024-08-20T21:58:05.7013040Z Is keep-going label set? False 2024-08-20T21:58:05.7013362Z 2024-08-20T21:58:05.7013526Z Renabled issues? 2024-08-20T21:58:05.7067619Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T21:58:05.7069804Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T21:58:05.7076426Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:58:05.7077048Z env: 2024-08-20T21:58:05.7077385Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:05.7077995Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:05.7078590Z JOB_TIMEOUT: 360 2024-08-20T21:58:05.7078963Z ##[endgroup] 2024-08-20T21:58:05.7174716Z ##[group]Run set -x 2024-08-20T21:58:05.7175242Z set -x 2024-08-20T21:58:05.7175599Z  2024-08-20T21:58:05.7176029Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-20T21:58:05.7176701Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-20T21:58:05.7177407Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-20T21:58:05.7178046Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-20T21:58:05.7178542Z else 2024-08-20T21:58:05.7178962Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T21:58:05.7179489Z fi 2024-08-20T21:58:05.7179801Z  2024-08-20T21:58:05.7180384Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-20T21:58:05.7181360Z # TODO: Stop building test binaries as part of the build phase 2024-08-20T21:58:05.7182198Z # Used for GPU_FLAG since that doesn't play nice 2024-08-20T21:58:05.7182937Z # shellcheck disable=SC2086,SC2090 2024-08-20T21:58:05.7183502Z container_name=$(docker run \ 2024-08-20T21:58:05.7184022Z  ${GPU_FLAG:-} \ 2024-08-20T21:58:05.7184483Z  -e BUILD_ENVIRONMENT \ 2024-08-20T21:58:05.7184963Z  -e PR_NUMBER \ 2024-08-20T21:58:05.7185407Z  -e GITHUB_ACTIONS \ 2024-08-20T21:58:05.7185893Z  -e GITHUB_REPOSITORY \ 2024-08-20T21:58:05.7186375Z  -e GITHUB_WORKFLOW \ 2024-08-20T21:58:05.7186851Z  -e GITHUB_JOB \ 2024-08-20T21:58:05.7187298Z  -e GITHUB_RUN_ID \ 2024-08-20T21:58:05.7187769Z  -e GITHUB_RUN_NUMBER \ 2024-08-20T21:58:05.7188254Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-20T21:58:05.7188746Z  -e JOB_ID \ 2024-08-20T21:58:05.7189169Z  -e JOB_NAME \ 2024-08-20T21:58:05.7189581Z  -e BASE_SHA \ 2024-08-20T21:58:05.7190004Z  -e BRANCH \ 2024-08-20T21:58:05.7190411Z  -e SHA1 \ 2024-08-20T21:58:05.7190820Z  -e AWS_DEFAULT_REGION \ 2024-08-20T21:58:05.7191323Z  -e IN_WHEEL_TEST \ 2024-08-20T21:58:05.7191788Z  -e SHARD_NUMBER \ 2024-08-20T21:58:05.7192227Z  -e TEST_CONFIG \ 2024-08-20T21:58:05.7192683Z  -e NUM_TEST_SHARDS \ 2024-08-20T21:58:05.7193167Z  -e REENABLED_ISSUES \ 2024-08-20T21:58:05.7193660Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-20T21:58:05.7194188Z  -e VERBOSE_TEST_LOGS \ 2024-08-20T21:58:05.7194687Z  -e TEST_SHOWLOCALS \ 2024-08-20T21:58:05.7195154Z  -e NO_TEST_TIMEOUT \ 2024-08-20T21:58:05.7195621Z  -e NO_TD \ 2024-08-20T21:58:05.7196039Z  -e TD_DISTRIBUTED \ 2024-08-20T21:58:05.7196491Z  -e PR_LABELS \ 2024-08-20T21:58:05.7197170Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-20T21:58:05.7197740Z  -e SCCACHE_BUCKET \ 2024-08-20T21:58:05.7198215Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-20T21:58:05.7198722Z  -e XLA_CUDA \ 2024-08-20T21:58:05.7199222Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-20T21:58:05.7199844Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-20T21:58:05.7200509Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-20T21:58:05.7201151Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-20T21:58:05.7201720Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-20T21:58:05.7202284Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-08-20T21:58:05.7202831Z  -e DASHBOARD_TAG \ 2024-08-20T21:58:05.7203412Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-20T21:58:05.7204115Z  --security-opt seccomp=unconfined \ 2024-08-20T21:58:05.7204695Z  --cap-add=SYS_PTRACE \ 2024-08-20T21:58:05.7205192Z  --ipc=host \ 2024-08-20T21:58:05.7205616Z  --shm-size="${SHM_SIZE}" \ 2024-08-20T21:58:05.7206101Z  --tty \ 2024-08-20T21:58:05.7206482Z  --detach \ 2024-08-20T21:58:05.7207027Z  --name="${container_name}" \ 2024-08-20T21:58:05.7207568Z  --user jenkins \ 2024-08-20T21:58:05.7208177Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-20T21:58:05.7208867Z  -w /var/lib/jenkins/workspace \ 2024-08-20T21:58:05.7209406Z  "${DOCKER_IMAGE}" 2024-08-20T21:58:05.7209836Z ) 2024-08-20T21:58:05.7210323Z # Propagate download.pytorch.org IP to container 2024-08-20T21:58:05.7211504Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-20T21:58:05.7212750Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-20T21:58:05.7213899Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-20T21:58:05.7220877Z shell: /usr/bin/bash -e {0} 2024-08-20T21:58:05.7221319Z env: 2024-08-20T21:58:05.7221650Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:05.7222201Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:05.7222931Z BUILD_ENVIRONMENT: linux-focal-cuda11.8-py3.10-gcc9 2024-08-20T21:58:05.7223531Z PR_NUMBER: 133712 2024-08-20T21:58:05.7223935Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-20T21:58:05.7224441Z GITHUB_WORKFLOW: pull 2024-08-20T21:58:05.7224849Z GITHUB_JOB: test 2024-08-20T21:58:05.7225213Z GITHUB_RUN_ID: 10479309237 2024-08-20T21:58:05.7225656Z GITHUB_RUN_NUMBER: 238988 2024-08-20T21:58:05.7226096Z GITHUB_RUN_ATTEMPT: 1 2024-08-20T21:58:05.7226486Z JOB_ID: 29025850949 2024-08-20T21:58:05.7227346Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, amz2023.linux.8xlarge.nvidia.gpu) 2024-08-20T21:58:05.7228326Z BRANCH: pull/133712 2024-08-20T21:58:05.7228778Z SHA1: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:58:05.7229427Z BASE_SHA: 91f3d614142df02f619e44a68e3d9e0dfeba49ec 2024-08-20T21:58:05.7230017Z TEST_CONFIG: distributed 2024-08-20T21:58:05.7230444Z SHARD_NUMBER: 3 2024-08-20T21:58:05.7230804Z NUM_TEST_SHARDS: 3 2024-08-20T21:58:05.7231199Z REENABLED_ISSUES: 2024-08-20T21:58:05.7231594Z CONTINUE_THROUGH_ERROR: False 2024-08-20T21:58:05.7232067Z VERBOSE_TEST_LOGS: False 2024-08-20T21:58:05.7232501Z TEST_SHOWLOCALS: False 2024-08-20T21:58:05.7232907Z NO_TEST_TIMEOUT: False 2024-08-20T21:58:05.7233314Z NO_TD: False 2024-08-20T21:58:05.7233678Z TD_DISTRIBUTED: False 2024-08-20T21:58:05.7234175Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-20T21:58:05.7234784Z SCCACHE_S3_KEY_PREFIX: pull 2024-08-20T21:58:05.7235223Z SHM_SIZE: 2g 2024-08-20T21:58:05.7236433Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:58:05.7238196Z XLA_CUDA: 2024-08-20T21:58:05.7238795Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-20T21:58:05.7239565Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-20T21:58:05.7240088Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-20T21:58:05.7240634Z DASHBOARD_TAG: 2024-08-20T21:58:05.7241021Z HUGGING_FACE_HUB_TOKEN: 2024-08-20T21:58:05.7241448Z SCRIBE_GRAPHQL_ACCESS_TOKEN: 2024-08-20T21:58:05.7241909Z ##[endgroup] 2024-08-20T21:58:05.7269968Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-08-20T21:58:05.7270740Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *onnx* ]] 2024-08-20T21:58:05.7271370Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T21:58:05.7279249Z +++ nproc --ignore=2 2024-08-20T21:58:05.7316644Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10479309237 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:58:16.8812575Z + container_name=42ff1b32f81951577dea4e347041b56c1d24119a3992eaf37c41c02aecf51e0b 2024-08-20T21:58:16.8817193Z + grep download.pytorch.org /etc/hosts 2024-08-20T21:58:16.8819389Z + docker exec -i 42ff1b32f81951577dea4e347041b56c1d24119a3992eaf37c41c02aecf51e0b sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-20T21:58:17.0578682Z + echo DOCKER_CONTAINER_ID=42ff1b32f81951577dea4e347041b56c1d24119a3992eaf37c41c02aecf51e0b 2024-08-20T21:58:17.0583493Z ++ echo dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T21:58:17.0586138Z + docker exec -t 42ff1b32f81951577dea4e347041b56c1d24119a3992eaf37c41c02aecf51e0b sh -c 'pip install dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-20T21:58:17.5679471Z Processing ./dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+git40ec5f6) 2024-08-20T21:58:18.6979686Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.13.1) 2024-08-20T21:58:18.6984052Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (4.12.2) 2024-08-20T21:58:18.6988948Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.8.8) 2024-08-20T21:58:18.6994079Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.1.4) 2024-08-20T21:58:18.6999397Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2024.6.1) 2024-08-20T21:58:18.7007298Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.13.1) 2024-08-20T21:58:18.7031221Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.3.0) 2024-08-20T21:58:18.7052383Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.3.0) 2024-08-20T21:58:18.7078207Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.22.4) 2024-08-20T21:58:18.7610794Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.1.5) 2024-08-20T21:58:19.2001963Z Installing collected packages: torch 2024-08-20T21:58:31.9601191Z Successfully installed torch-2.5.0a0+git40ec5f6 2024-08-20T21:58:32.0365883Z + export TERM=vt100 2024-08-20T21:58:32.0368049Z + TERM=vt100 2024-08-20T21:58:32.0369652Z ++ dirname .ci/pytorch/test.sh 2024-08-20T21:58:32.0378288Z + source .ci/pytorch/common.sh 2024-08-20T21:58:32.0382113Z +++ dirname .ci/pytorch/common.sh 2024-08-20T21:58:32.0390615Z ++ source .ci/pytorch/common_utils.sh 2024-08-20T21:58:32.0392824Z +++ declare -f -t trap_add 2024-08-20T21:58:32.0399837Z ++ set -ex 2024-08-20T21:58:32.0400408Z ++ [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-08-20T21:58:32.0401012Z ++ BUILD_TEST_LIBTORCH=0 2024-08-20T21:58:32.0402022Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-08-20T21:58:32.0405138Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-20T21:58:32.0421619Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-20T21:58:32.0422147Z + trap_add cleanup_workspace EXIT 2024-08-20T21:58:32.0423874Z + trap_add_cmd=cleanup_workspace 2024-08-20T21:58:32.0424384Z + shift 2024-08-20T21:58:32.0424729Z + for trap_add_name in "$@" 2024-08-20T21:58:32.0430735Z +++ trap -p EXIT 2024-08-20T21:58:32.0434053Z ++ eval 'extract_trap_cmd ' 2024-08-20T21:58:32.0434536Z +++ extract_trap_cmd 2024-08-20T21:58:32.0434961Z +++ printf '%s\n' '' 2024-08-20T21:58:32.0435458Z ++ printf '%s\n' cleanup_workspace 2024-08-20T21:58:32.0436826Z + trap -- ' 2024-08-20T21:58:32.0437523Z cleanup_workspace' EXIT 2024-08-20T21:58:32.0438151Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-20T21:58:32.6486807Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-20T21:58:32.6506753Z + echo 'Environment variables:' 2024-08-20T21:58:32.6507257Z Environment variables: 2024-08-20T21:58:32.6507658Z + env 2024-08-20T21:58:32.6516936Z INSTALLED_DB=yes 2024-08-20T21:58:32.6517424Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-08-20T21:58:32.6517925Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T21:58:32.6518412Z NV_NVML_DEV_VERSION=11.8.86-1 2024-08-20T21:58:32.6519184Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T21:58:32.6519888Z CONTINUE_THROUGH_ERROR=False 2024-08-20T21:58:32.6520565Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-08-20T21:58:32.6521962Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-08-20T21:58:32.6522782Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-08-20T21:58:32.6523370Z HOSTNAME=42ff1b32f819 2024-08-20T21:58:32.6524511Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_05c306e3-14e3-4b40-ae29-337f45730baa 2024-08-20T21:58:32.6525574Z GITHUB_ACTION=__self 2024-08-20T21:58:32.6525976Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T21:58:32.6528726Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-08-20T21:58:32.6531739Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-08-20T21:58:32.6532722Z NV_NVTX_VERSION=11.8.86-1 2024-08-20T21:58:32.6533145Z GITHUB_RUN_NUMBER=238988 2024-08-20T21:58:32.6533539Z TEST_CONFIG=distributed 2024-08-20T21:58:32.6534025Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T21:58:32.6534636Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T21:58:32.6535214Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-08-20T21:58:32.6535785Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-08-20T21:58:32.6536334Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T21:58:32.6536875Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-08-20T21:58:32.6537804Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2024-08-20T21:58:32.6538446Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T21:58:32.6538989Z GITHUB_REF_TYPE=branch 2024-08-20T21:58:32.6539458Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T21:58:32.6539951Z NCCL_VERSION=2.16.2-1 2024-08-20T21:58:32.6540427Z BASE_SHA=91f3d614142df02f619e44a68e3d9e0dfeba49ec 2024-08-20T21:58:32.6540971Z XLA_CUDA= 2024-08-20T21:58:32.6541316Z HUGGING_FACE_HUB_TOKEN= 2024-08-20T21:58:32.6543808Z *** 2024-08-20T21:58:32.6544176Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T21:58:32.6544672Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T21:58:32.6545105Z GITHUB_ACTIONS=true 2024-08-20T21:58:32.6545718Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:32.6546424Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-08-20T21:58:32.6547103Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-08-20T21:58:32.6547696Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:58:32.6548357Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T21:58:32.6548953Z GITHUB_SHA=f2fb9405c2fa9f9502a76363091cce6fd8179736 2024-08-20T21:58:32.6549909Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/133712/merge 2024-08-20T21:58:32.6550714Z UCC_HOME=/usr 2024-08-20T21:58:32.6551158Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-08-20T21:58:32.6551681Z VERBOSE_TEST_LOGS=False 2024-08-20T21:58:32.6552127Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T21:58:32.6552771Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-08-20T21:58:32.6553380Z GITHUB_REF=refs/pull/133712/merge 2024-08-20T21:58:32.6553952Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-08-20T21:58:32.6554446Z SHARD_NUMBER=3 2024-08-20T21:58:32.6554834Z GITHUB_REF_PROTECTED=false 2024-08-20T21:58:32.6555290Z HOME=/var/lib/jenkins 2024-08-20T21:58:32.6555768Z GITHUB_API_URL=https://api.github.com 2024-08-20T21:58:32.6556334Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T21:58:32.6556963Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T21:58:32.6557603Z SCCACHE_S3_KEY_PREFIX=pull 2024-08-20T21:58:32.6558047Z CUDA_VERSION=11.8.0 2024-08-20T21:58:32.6558631Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-08-20T21:58:32.6559221Z NUM_TEST_SHARDS=3 2024-08-20T21:58:32.6559593Z UCX_HOME=/usr 2024-08-20T21:58:32.6560303Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-08-20T21:58:32.6561786Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_05c306e3-14e3-4b40-ae29-337f45730baa 2024-08-20T21:58:32.6563532Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, amz2023.linux.8xlarge.nvidia.gpu) 2024-08-20T21:58:32.6565266Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_05c306e3-14e3-4b40-ae29-337f45730baa 2024-08-20T21:58:32.6566823Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T21:58:32.6567739Z GITHUB_EVENT_NAME=pull_request 2024-08-20T21:58:32.6568179Z DASHBOARD_TAG= 2024-08-20T21:58:32.6568566Z GITHUB_RUN_ID=10479309237 2024-08-20T21:58:32.6569200Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-08-20T21:58:32.6569921Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-08-20T21:58:32.6571308Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_05c306e3-14e3-4b40-ae29-337f45730baa 2024-08-20T21:58:32.6572495Z GITHUB_ACTOR=pytorchmergebot 2024-08-20T21:58:32.6573028Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-08-20T21:58:32.6573707Z PR_NUMBER=133712 2024-08-20T21:58:32.6574098Z GITHUB_RUN_ATTEMPT=1 2024-08-20T21:58:32.6574516Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T21:58:32.6575103Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T21:58:32.6575720Z TERM=vt100 2024-08-20T21:58:32.6576155Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-08-20T21:58:32.6576696Z INSTALLED_VISION=yes 2024-08-20T21:58:32.6577113Z BRANCH=pull/133712 2024-08-20T21:58:32.6577518Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T21:58:32.6578040Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T21:58:32.6578594Z CUDA_PATH=/usr/local/cuda 2024-08-20T21:58:32.6579621Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T21:58:32.6580636Z GITHUB_SERVER_URL=https://github.com 2024-08-20T21:58:32.6581271Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T21:58:32.6581868Z REENABLED_ISSUES= 2024-08-20T21:58:32.6582247Z SHLVL=1 2024-08-20T21:58:32.6582582Z MAX_JOBS=30 2024-08-20T21:58:32.6582985Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-08-20T21:58:32.6583450Z NVARCH=x86_64 2024-08-20T21:58:32.6583827Z GITHUB_ACTOR_ID=97764156 2024-08-20T21:58:32.6584416Z GITHUB_WORKFLOW_SHA=f2fb9405c2fa9f9502a76363091cce6fd8179736 2024-08-20T21:58:32.6585202Z GITHUB_REF_NAME=133712/merge 2024-08-20T21:58:32.6585823Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-08-20T21:58:32.6586764Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T21:58:32.6587498Z GITHUB_JOB=test 2024-08-20T21:58:32.6588016Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-08-20T21:58:32.6588698Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T21:58:32.6589335Z NO_TEST_TIMEOUT=False 2024-08-20T21:58:32.6589738Z TD_DISTRIBUTED=False 2024-08-20T21:58:32.6590206Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-08-20T21:58:32.6590758Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T21:58:32.6591294Z NV_NVPROF_VERSION=11.8.87-1 2024-08-20T21:58:32.6591737Z GITHUB_RETENTION_DAYS=90 2024-08-20T21:58:32.6592161Z OPENSSL_DIR=/opt/openssl 2024-08-20T21:58:32.6592587Z GITHUB_ACTION_REPOSITORY= 2024-08-20T21:58:32.6593905Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T21:58:32.6595318Z GITHUB_BASE_REF=gh/XuehaiPan/146/base 2024-08-20T21:58:32.6595841Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T21:58:32.6596273Z CI=true 2024-08-20T21:58:32.6596679Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-08-20T21:58:32.6597199Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T21:58:32.6597640Z JOB_ID=29025850949 2024-08-20T21:58:32.6598020Z INSTALLED_PROTOBUF=yes 2024-08-20T21:58:32.6598455Z GITHUB_HEAD_REF=gh/XuehaiPan/146/head 2024-08-20T21:58:32.6598931Z GITHUB_ACTION_REF= 2024-08-20T21:58:32.6599497Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T21:58:32.6600086Z TEST_SHOWLOCALS=False 2024-08-20T21:58:32.6600471Z GITHUB_WORKFLOW=pull 2024-08-20T21:58:32.6600893Z DEBIAN_FRONTEND=noninteractive 2024-08-20T21:58:32.6602089Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_05c306e3-14e3-4b40-ae29-337f45730baa 2024-08-20T21:58:32.6603107Z NO_TD=False 2024-08-20T21:58:32.6603476Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T21:58:32.6603926Z _=/usr/bin/env 2024-08-20T21:58:32.6604533Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-20T21:58:32.6817786Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-08-20T21:58:32.6818938Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T21:58:32.6820160Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-08-20T21:58:32.6821297Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-08-20T21:58:32.6822069Z + BUILD_DIR=build 2024-08-20T21:58:32.6822467Z + BUILD_RENAMED_DIR=build_renamed 2024-08-20T21:58:32.6822931Z + BUILD_BIN_DIR=build/bin 2024-08-20T21:58:32.6823558Z + SHARD_NUMBER=3 2024-08-20T21:58:32.6823928Z + NUM_TEST_SHARDS=3 2024-08-20T21:58:32.6824329Z + export VALGRIND=ON 2024-08-20T21:58:32.6824693Z + VALGRIND=ON 2024-08-20T21:58:32.6825295Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *clang9* ]] 2024-08-20T21:58:32.6825847Z + [[ 0 == \1 ]] 2024-08-20T21:58:32.6826207Z + [[ False == \1 ]] 2024-08-20T21:58:32.6826776Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *bazel* ]] 2024-08-20T21:58:32.6827385Z ++ realpath build/custom_test_artifacts 2024-08-20T21:58:32.6836445Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-20T21:58:32.6837578Z + [[ -n '' ]] 2024-08-20T21:58:32.6838016Z + echo 'Environment variables' 2024-08-20T21:58:32.6838489Z Environment variables 2024-08-20T21:58:32.6838873Z + env 2024-08-20T21:58:32.6845816Z INSTALLED_DB=yes 2024-08-20T21:58:32.6846297Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-08-20T21:58:32.6846813Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T21:58:32.6847302Z NV_NVML_DEV_VERSION=11.8.86-1 2024-08-20T21:58:32.6848047Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T21:58:32.6848777Z CONTINUE_THROUGH_ERROR=False 2024-08-20T21:58:32.6849406Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-08-20T21:58:32.6850261Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-08-20T21:58:32.6851022Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-08-20T21:58:32.6851568Z HOSTNAME=42ff1b32f819 2024-08-20T21:58:32.6852670Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_05c306e3-14e3-4b40-ae29-337f45730baa 2024-08-20T21:58:32.6853738Z GITHUB_ACTION=__self 2024-08-20T21:58:32.6854141Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T21:58:32.6856864Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-08-20T21:58:32.6859790Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-08-20T21:58:32.6860434Z NV_NVTX_VERSION=11.8.86-1 2024-08-20T21:58:32.6860852Z GITHUB_RUN_NUMBER=238988 2024-08-20T21:58:32.6861278Z TEST_CONFIG=distributed 2024-08-20T21:58:32.6861719Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T21:58:32.6862312Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T21:58:32.6862898Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-08-20T21:58:32.6863472Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-08-20T21:58:32.6863986Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T21:58:32.6864518Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-08-20T21:58:32.6865049Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2024-08-20T21:58:32.6865690Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T21:58:32.6866309Z GITHUB_REF_TYPE=branch 2024-08-20T21:58:32.6866805Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T21:58:32.6867327Z NCCL_VERSION=2.16.2-1 2024-08-20T21:58:32.6867872Z BASE_SHA=91f3d614142df02f619e44a68e3d9e0dfeba49ec 2024-08-20T21:58:32.6868408Z XLA_CUDA= 2024-08-20T21:58:32.6868762Z HUGGING_FACE_HUB_TOKEN= 2024-08-20T21:58:32.6869282Z *** 2024-08-20T21:58:32.6869629Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T21:58:32.6870180Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T21:58:32.6870614Z GITHUB_ACTIONS=true 2024-08-20T21:58:32.6871045Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:32.6871722Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-08-20T21:58:32.6872436Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-08-20T21:58:32.6873059Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:58:32.6873810Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T21:58:32.6874410Z GITHUB_SHA=f2fb9405c2fa9f9502a76363091cce6fd8179736 2024-08-20T21:58:32.6875348Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/133712/merge 2024-08-20T21:58:32.6876248Z UCC_HOME=/usr 2024-08-20T21:58:32.6876822Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-08-20T21:58:32.6877391Z VERBOSE_TEST_LOGS=False 2024-08-20T21:58:32.6877792Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T21:58:32.6878423Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-08-20T21:58:32.6879048Z GITHUB_REF=refs/pull/133712/merge 2024-08-20T21:58:32.6879670Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-08-20T21:58:32.6880115Z SHARD_NUMBER=3 2024-08-20T21:58:32.6880492Z GITHUB_REF_PROTECTED=false 2024-08-20T21:58:32.6880994Z HOME=/var/lib/jenkins 2024-08-20T21:58:32.6881418Z GITHUB_API_URL=https://api.github.com 2024-08-20T21:58:32.6881982Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T21:58:32.6882611Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T21:58:32.6883286Z SCCACHE_S3_KEY_PREFIX=pull 2024-08-20T21:58:32.6883674Z CUDA_VERSION=11.8.0 2024-08-20T21:58:32.6884262Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-08-20T21:58:32.6884846Z NUM_TEST_SHARDS=3 2024-08-20T21:58:32.6885214Z UCX_HOME=/usr 2024-08-20T21:58:32.6885929Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-08-20T21:58:32.6887423Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_05c306e3-14e3-4b40-ae29-337f45730baa 2024-08-20T21:58:32.6889262Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, amz2023.linux.8xlarge.nvidia.gpu) 2024-08-20T21:58:32.6890980Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_05c306e3-14e3-4b40-ae29-337f45730baa 2024-08-20T21:58:32.6892533Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T21:58:32.6893416Z GITHUB_EVENT_NAME=pull_request 2024-08-20T21:58:32.6893924Z DASHBOARD_TAG= 2024-08-20T21:58:32.6894277Z GITHUB_RUN_ID=10479309237 2024-08-20T21:58:32.6894892Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-08-20T21:58:32.6895631Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-08-20T21:58:32.6897002Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_05c306e3-14e3-4b40-ae29-337f45730baa 2024-08-20T21:58:32.6898176Z GITHUB_ACTOR=pytorchmergebot 2024-08-20T21:58:32.6898725Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-08-20T21:58:32.6899225Z PR_NUMBER=133712 2024-08-20T21:58:32.6899610Z GITHUB_RUN_ATTEMPT=1 2024-08-20T21:58:32.6900018Z VALGRIND=ON 2024-08-20T21:58:32.6900397Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T21:58:32.6900969Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T21:58:32.6901574Z TERM=vt100 2024-08-20T21:58:32.6902024Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-08-20T21:58:32.6902552Z INSTALLED_VISION=yes 2024-08-20T21:58:32.6902969Z BRANCH=pull/133712 2024-08-20T21:58:32.6903389Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T21:58:32.6903895Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T21:58:32.6904450Z CUDA_PATH=/usr/local/cuda 2024-08-20T21:58:32.6905495Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T21:58:32.6906506Z GITHUB_SERVER_URL=https://github.com 2024-08-20T21:58:32.6907148Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T21:58:32.6907764Z REENABLED_ISSUES= 2024-08-20T21:58:32.6908123Z SHLVL=1 2024-08-20T21:58:32.6908454Z MAX_JOBS=30 2024-08-20T21:58:32.6908886Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-08-20T21:58:32.6909339Z NVARCH=x86_64 2024-08-20T21:58:32.6909722Z GITHUB_ACTOR_ID=97764156 2024-08-20T21:58:32.6910330Z GITHUB_WORKFLOW_SHA=f2fb9405c2fa9f9502a76363091cce6fd8179736 2024-08-20T21:58:32.6910993Z GITHUB_REF_NAME=133712/merge 2024-08-20T21:58:32.6911593Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-08-20T21:58:32.6912501Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T21:58:32.6913245Z GITHUB_JOB=test 2024-08-20T21:58:32.6913790Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-08-20T21:58:32.6914533Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T21:58:32.6915201Z NO_TEST_TIMEOUT=False 2024-08-20T21:58:32.6915621Z TD_DISTRIBUTED=False 2024-08-20T21:58:32.6942059Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-08-20T21:58:32.6942622Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T21:58:32.6943144Z NV_NVPROF_VERSION=11.8.87-1 2024-08-20T21:58:32.6943589Z GITHUB_RETENTION_DAYS=90 2024-08-20T21:58:32.6944046Z OPENSSL_DIR=/opt/openssl 2024-08-20T21:58:32.6944430Z GITHUB_ACTION_REPOSITORY= 2024-08-20T21:58:32.6945762Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T21:58:32.6947174Z GITHUB_BASE_REF=gh/XuehaiPan/146/base 2024-08-20T21:58:32.6947689Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T21:58:32.6948105Z CI=true 2024-08-20T21:58:32.6948506Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-08-20T21:58:32.6949024Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T21:58:32.6949466Z JOB_ID=29025850949 2024-08-20T21:58:32.6949833Z INSTALLED_PROTOBUF=yes 2024-08-20T21:58:32.6950243Z GITHUB_HEAD_REF=gh/XuehaiPan/146/head 2024-08-20T21:58:32.6950743Z GITHUB_ACTION_REF= 2024-08-20T21:58:32.6951289Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T21:58:32.6951858Z TEST_SHOWLOCALS=False 2024-08-20T21:58:32.6952257Z GITHUB_WORKFLOW=pull 2024-08-20T21:58:32.6952647Z DEBIAN_FRONTEND=noninteractive 2024-08-20T21:58:32.6954004Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_05c306e3-14e3-4b40-ae29-337f45730baa 2024-08-20T21:58:32.6955052Z NO_TD=False 2024-08-20T21:58:32.6955388Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T21:58:32.6955840Z _=/usr/bin/env 2024-08-20T21:58:32.6956274Z + echo 'Testing pytorch' 2024-08-20T21:58:32.6956666Z Testing pytorch 2024-08-20T21:58:32.6957039Z + export LANG=C.UTF-8 2024-08-20T21:58:32.6957465Z + LANG=C.UTF-8 2024-08-20T21:58:32.6957810Z + PR_NUMBER=133712 2024-08-20T21:58:32.6958207Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-08-20T21:58:32.6958769Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-20T21:58:32.6959497Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-08-20T21:58:32.6960077Z + [[ distributed == \s\l\o\w ]] 2024-08-20T21:58:32.6960772Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *slow-gradcheck* ]] 2024-08-20T21:58:32.6961564Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-08-20T21:58:32.6962193Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T21:58:32.6962746Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T21:58:32.6963242Z + [[ distributed == *crossref* ]] 2024-08-20T21:58:32.6963869Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-08-20T21:58:32.6964592Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *xpu* ]] 2024-08-20T21:58:32.6965328Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-08-20T21:58:32.6965987Z + pip_install --user ninja==1.10.2 2024-08-20T21:58:32.6966654Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-20T21:58:33.2664842Z Collecting ninja==1.10.2 2024-08-20T21:58:33.2859747Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-20T21:58:33.2989611Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-20T21:58:33.7292493Z Installing collected packages: ninja 2024-08-20T21:58:33.7396804Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-20T21:58:33.7398525Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-20T21:58:33.7456341Z Successfully installed ninja-1.10.2 2024-08-20T21:58:33.8147920Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T21:58:33.8150715Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T21:58:33.8152904Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-08-20T21:58:33.8153486Z + install_tlparse 2024-08-20T21:58:33.8154008Z + pip_install --user tlparse==0.3.25 2024-08-20T21:58:33.8154719Z + pip install --progress-bar off --user tlparse==0.3.25 2024-08-20T21:58:34.3111623Z Collecting tlparse==0.3.25 2024-08-20T21:58:34.3291176Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-08-20T21:58:34.3427188Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-20T21:58:34.8117493Z Installing collected packages: tlparse 2024-08-20T21:58:34.8538344Z Successfully installed tlparse-0.3.25 2024-08-20T21:58:34.9230058Z ++ python -m site --user-base 2024-08-20T21:58:34.9548960Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T21:58:34.9552172Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *asan* ]] 2024-08-20T21:58:34.9552957Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-debug* ]] 2024-08-20T21:58:34.9554176Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-08-20T21:58:34.9556182Z + echo 'We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass' 2024-08-20T21:58:34.9557660Z We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass 2024-08-20T21:58:34.9558531Z + cd test 2024-08-20T21:58:34.9559214Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-20T21:58:37.0810716Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-20T21:58:37.0811483Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-20T21:58:37.0812991Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-20T21:58:37.0813901Z + [[ distributed == *pr_time_benchmarks* ]] 2024-08-20T21:58:37.0814726Z + [[ distributed == *dynamo_eager* ]] 2024-08-20T21:58:37.0815281Z + [[ distributed == *aot_eager* ]] 2024-08-20T21:58:37.0815830Z + [[ distributed == *aot_inductor* ]] 2024-08-20T21:58:37.0816350Z + [[ distributed == *inductor* ]] 2024-08-20T21:58:37.0816859Z + [[ distributed == *dynamic* ]] 2024-08-20T21:58:37.0817323Z + [[ distributed == *cpu* ]] 2024-08-20T21:58:37.0818115Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-20T21:58:37.0856882Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-08-20T21:58:37.0857707Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-bazel-* ]] 2024-08-20T21:58:37.0859593Z + cd test 2024-08-20T21:58:37.0860357Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-20T21:58:38.9571418Z PyTorch built with: 2024-08-20T21:58:38.9572663Z - GCC 9.4 2024-08-20T21:58:38.9573151Z - C++ Version: 201703 2024-08-20T21:58:38.9574303Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T21:58:38.9575723Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T21:58:38.9576616Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T21:58:38.9577271Z - LAPACK is enabled (usually provided by MKL) 2024-08-20T21:58:38.9577897Z - NNPACK is enabled 2024-08-20T21:58:38.9578394Z - CPU capability usage: AVX2 2024-08-20T21:58:38.9578922Z - CUDA Runtime 11.8 2024-08-20T21:58:38.9579608Z - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52 2024-08-20T21:58:38.9580317Z - CuDNN 90.1 2024-08-20T21:58:38.9580714Z - Magma 2.6.1 2024-08-20T21:58:38.9589906Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-20T21:58:38.9598644Z 2024-08-20T21:58:39.2241179Z + cd test 2024-08-20T21:58:39.2242499Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-20T21:58:41.0701927Z ATen/Parallel: 2024-08-20T21:58:41.0702857Z at::get_num_threads() : 16 2024-08-20T21:58:41.0703875Z at::get_num_interop_threads() : 16 2024-08-20T21:58:41.0704907Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T21:58:41.0705899Z omp_get_max_threads() : 16 2024-08-20T21:58:41.0708828Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T21:58:41.0710927Z mkl_get_max_threads() : 16 2024-08-20T21:58:41.0712582Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T21:58:41.0714209Z std::thread::hardware_concurrency() : 32 2024-08-20T21:58:41.0715290Z Environment variables: 2024-08-20T21:58:41.0716166Z OMP_NUM_THREADS : [not set] 2024-08-20T21:58:41.0717096Z MKL_NUM_THREADS : [not set] 2024-08-20T21:58:41.0718026Z ATen parallel backend: OpenMP 2024-08-20T21:58:41.0718705Z 2024-08-20T21:58:41.3194125Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-08-20T21:58:41.3194861Z + [[ distributed == *backward* ]] 2024-08-20T21:58:41.3195368Z + [[ distributed == *xla* ]] 2024-08-20T21:58:41.3195816Z + [[ distributed == *executorch* ]] 2024-08-20T21:58:41.3196360Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-20T21:58:41.3197161Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-08-20T21:58:41.3197752Z + [[ distributed == distributed ]] 2024-08-20T21:58:41.3198304Z + test_distributed 2024-08-20T21:58:41.3198773Z + echo 'Testing distributed python tests' 2024-08-20T21:58:41.3199317Z Testing distributed python tests 2024-08-20T21:58:41.3200114Z + python test/run_test.py --distributed-tests --shard 3 3 --verbose 2024-08-20T21:58:41.4540072Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T21:58:41.4541593Z import pkg_resources 2024-08-20T21:58:46.0570864Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T21:58:46.1147204Z Ignoring disabled issues: [''] 2024-08-20T21:58:46.1289960Z Found test times from artifacts 2024-08-20T21:58:46.1840514Z Found test times from artifacts 2024-08-20T21:58:46.1857793Z Running 25% of tests based on TD 2024-08-20T21:58:46.1978835Z Running parallel tests on 2 processes 2024-08-20T21:58:46.1980268Z Name: tests to run (est. time: 9.6min) 2024-08-20T21:58:46.1981317Z Serial tests (19): 2024-08-20T21:58:46.1982171Z distributed/_tensor/test_dtensor_compile 1/1 2024-08-20T21:58:46.1982812Z distributed/_tensor/debug/test_op_coverage 1/1 2024-08-20T21:58:46.1983463Z distributed/test_c10d_functional_native 1/1 2024-08-20T21:58:46.1984063Z distributed/pipelining/test_pipe 1/1 2024-08-20T21:58:46.1985233Z distributed/tensor/parallel/test_micro_pipeline_tp 1/1 2024-08-20T21:58:46.1985995Z distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 2024-08-20T21:58:46.1987411Z distributed/pipelining/test_microbatch 1/1 2024-08-20T21:58:46.1988343Z distributed/_composable/test_contract 1/1 2024-08-20T21:58:46.1989257Z distributed/_tensor/test_common_rules 1/1 2024-08-20T21:58:46.1989966Z distributed/_tensor/experimental/test_register_sharding 1/1 2024-08-20T21:58:46.1990670Z distributed/optim/test_named_optimizer 1/1 2024-08-20T21:58:46.1991256Z distributed/_tensor/test_api 1/1 2024-08-20T21:58:46.1991785Z distributed/test_device_mesh 1/1 2024-08-20T21:58:46.1992350Z distributed/checkpoint/test_save_load_api 1/1 2024-08-20T21:58:46.1993001Z distributed/checkpoint/test_dedup_tensors 1/1 2024-08-20T21:58:46.1993645Z distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 2024-08-20T21:58:46.1994228Z distributed/test_c10d_spawn_gloo 1/1 2024-08-20T21:58:46.1994820Z distributed/tensor/parallel/test_tp_style 1/1 2024-08-20T21:58:46.1995428Z distributed/_tensor/test_attention 1/1 2024-08-20T21:58:46.1995949Z Parallel tests (0): 2024-08-20T21:58:46.1996416Z Name: excluded (est. time: 76.52min) 2024-08-20T21:58:46.1996915Z Serial tests (53): 2024-08-20T21:58:46.1997360Z distributed/test_distributed_spawn 3/9 2024-08-20T21:58:46.1997930Z distributed/test_distributed_spawn 6/9 2024-08-20T21:58:46.1998500Z distributed/test_distributed_spawn 9/9 2024-08-20T21:58:46.1999062Z distributed/fsdp/test_fsdp_core 1/2 2024-08-20T21:58:46.1999753Z distributed/fsdp/test_fsdp_core 2/2 2024-08-20T21:58:46.2000359Z distributed/fsdp/test_fsdp_optim_state 1/1 2024-08-20T21:58:46.2000995Z distributed/fsdp/test_fsdp_mixed_precision 1/1 2024-08-20T21:58:46.2001646Z distributed/fsdp/test_fsdp_freezing_weights 1/1 2024-08-20T21:58:46.2002292Z distributed/fsdp/test_fsdp_comm_hooks 1/1 2024-08-20T21:58:46.2003065Z distributed/_composable/test_composability/test_2d_composability 1/1 2024-08-20T21:58:46.2003826Z distributed/fsdp/test_fsdp_checkpoint 1/1 2024-08-20T21:58:46.2004477Z distributed/fsdp/test_fsdp_dtensor_state_dict 1/1 2024-08-20T21:58:46.2005152Z distributed/fsdp/test_fsdp_ignored_modules 1/1 2024-08-20T21:58:46.2005751Z distributed/_tensor/test_math_ops 1/1 2024-08-20T21:58:46.2006408Z distributed/checkpoint/e2e/test_e2e_save_and_load 1/1 2024-08-20T21:58:46.2007098Z distributed/fsdp/test_fsdp_flatten_params 1/1 2024-08-20T21:58:46.2007682Z distributed/test_c10d_spawn_nccl 1/1 2024-08-20T21:58:46.2008275Z distributed/test_c10d_object_collectives 1/1 2024-08-20T21:58:46.2008884Z distributed/fsdp/test_fsdp_grad_acc 1/1 2024-08-20T21:58:46.2009545Z distributed/tensor/parallel/test_parallelize_api 1/1 2024-08-20T21:58:46.2010174Z distributed/fsdp/test_fsdp_meta 1/1 2024-08-20T21:58:46.2010763Z distributed/fsdp/test_fsdp_clip_grad_norm 1/1 2024-08-20T21:58:46.2011397Z distributed/checkpoint/test_checkpoint 1/1 2024-08-20T21:58:46.2012035Z distributed/fsdp/test_hsdp_dtensor_state_dict 1/1 2024-08-20T21:58:46.2012652Z distributed/_tensor/test_init 1/1 2024-08-20T21:58:46.2013343Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 2024-08-20T21:58:46.2014157Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 2024-08-20T21:58:46.2014859Z distributed/_tensor/test_embedding_ops 1/1 2024-08-20T21:58:46.2015549Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 2024-08-20T21:58:46.2016205Z distributed/fsdp/test_fsdp_memory 1/1 2024-08-20T21:58:46.2016925Z distributed/_composable/fully_shard/test_fully_shard_runtime 1/1 2024-08-20T21:58:46.2017686Z distributed/checkpoint/test_format_utils 1/1 2024-08-20T21:58:46.2018397Z distributed/_composable/fsdp/test_fully_shard_extensions 1/1 2024-08-20T21:58:46.2019104Z distributed/checkpoint/test_fsspec 1/1 2024-08-20T21:58:46.2019704Z distributed/fsdp/test_fsdp_pure_fp16 1/1 2024-08-20T21:58:46.2020258Z distributed/test_c10d_ops_nccl 1/1 2024-08-20T21:58:46.2020957Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 1/1 2024-08-20T21:58:46.2021782Z distributed/_composable/fsdp/test_fully_shard_memory 1/1 2024-08-20T21:58:46.2022494Z distributed/fsdp/test_fsdp_backward_prefetch 1/1 2024-08-20T21:58:46.2023277Z distributed/elastic/utils/distributed_test 1/1 2024-08-20T21:58:46.2023968Z distributed/fsdp/test_fsdp_multiple_wrapping 1/1 2024-08-20T21:58:46.2024629Z distributed/fsdp/test_fsdp_multiple_forward 1/1 2024-08-20T21:58:46.2025280Z distributed/fsdp/test_fsdp_traversal 1/1 2024-08-20T21:58:46.2025916Z distributed/checkpoint/e2e/test_fine_tuning 1/1 2024-08-20T21:58:46.2026564Z distributed/_tensor/test_pointwise_ops 1/1 2024-08-20T21:58:46.2027149Z distributed/elastic/utils/util_test 1/1 2024-08-20T21:58:46.2027736Z distributed/checkpoint/test_planner 1/1 2024-08-20T21:58:46.2028300Z distributed/fsdp/test_utils 1/1 2024-08-20T21:58:46.2028840Z distributed/test_control_collectives 1/1 2024-08-20T21:58:46.2029444Z distributed/nn/jit/test_instantiator 1/1 2024-08-20T21:58:46.2030030Z distributed/checkpoint/test_utils 1/1 2024-08-20T21:58:46.2030574Z distributed/test_collective_utils 1/1 2024-08-20T21:58:46.2031132Z distributed/rpc/test_faulty_agent 1/1 2024-08-20T21:58:46.2031663Z Parallel tests (0): 2024-08-20T21:58:46.2057404Z Running distributed/_tensor/test_dtensor_compile 1/1 ... [2024-08-20 21:58:46.205372] 2024-08-20T21:58:46.2058293Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T21:58:46.2062636Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 21:58:46.205795] 2024-08-20T21:59:18.8363376Z 2024-08-20T21:59:18.8365594Z distributed/_tensor/test_dtensor_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_compile_1.1_9874889ba8acc94c_.log 2024-08-20T21:59:18.8384510Z Running 26 items in this shard: test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_device_mesh_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_attribute_access_on_intermediate, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_dynamo_disable, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_graph_break, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_contiguous_dtensor_noncontiguous_local_as_tangent, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_different_gradient_placement, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dont_recompile_on_same_placement_devicemesh, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamo_device_mesh_attrs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_noncontiguous_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_graph_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_redistribute_unbalanced_correct_strides, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_recompile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs_forward_hook, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_fakify_dtensor, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_graph_input_is_async, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_placement_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_tp_compile_comm_reordering, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_dtensor_redistribute_backward, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_True 2024-08-20T21:59:18.8402740Z 2024-08-20T21:59:18.8403406Z Running distributed/_tensor/debug/test_op_coverage 1/1 ... [2024-08-20 21:59:18.836767] 2024-08-20T21:59:18.8404257Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T21:59:18.8406545Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/debug/test_op_coverage.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 21:59:18.837216] 2024-08-20T21:59:22.9095309Z 2024-08-20T21:59:22.9097814Z distributed/_tensor/debug/test_op_coverage 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.debug.test_op_coverage_1.1_8fd81d213d1f04be_.log 2024-08-20T21:59:22.9100129Z Running 1 items in this shard: test/distributed/_tensor/debug/test_op_coverage.py::TestOpCoverage::test_trace_with_inductor_decomp 2024-08-20T21:59:22.9101114Z 2024-08-20T21:59:22.9101744Z Running distributed/test_c10d_functional_native 1/1 ... [2024-08-20 21:59:22.909837] 2024-08-20T21:59:22.9102552Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T21:59:22.9107520Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_functional_native.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 21:59:22.910321] 2024-08-20T22:00:21.8400733Z 2024-08-20T22:00:21.8403926Z distributed/test_c10d_functional_native 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_functional_native_1.1_f71255279909e400_.log 2024-08-20T22:00:21.8420790Z Running 25 items in this shard: test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_to_all_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_broadcast, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_inductor_dtypeview_memory_leak, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_py_work, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_threading, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_unwaited, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_to_all_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_broadcast, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_inplace_op_on_view, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reuse_buffer_after_inplace_collective, test/distributed/test_c10d_functional_native.py::CompileTest::test_ranks_and_tag 2024-08-20T22:00:21.8436010Z 2024-08-20T22:00:21.8436614Z Running distributed/pipelining/test_pipe 1/1 ... [2024-08-20 22:00:21.840378] 2024-08-20T22:00:21.8438013Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:00:21.8440278Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_pipe.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:00:21.840814] 2024-08-20T22:00:26.7147825Z 2024-08-20T22:00:26.7150113Z distributed/pipelining/test_pipe 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_pipe_1.1_3facb31e2463593c_.log 2024-08-20T22:00:26.7153470Z Running 3 items in this shard: test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass0, test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass1, test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass2 2024-08-20T22:00:26.7155524Z 2024-08-20T22:00:26.7156237Z Running distributed/tensor/parallel/test_micro_pipeline_tp 1/1 ... [2024-08-20 22:00:26.715003] 2024-08-20T22:00:26.7157172Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:00:26.7159562Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_micro_pipeline_tp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:00:26.715442] 2024-08-20T22:00:31.5895009Z 2024-08-20T22:00:31.5897310Z distributed/tensor/parallel/test_micro_pipeline_tp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_micro_pipeline_tp_1.1_dc94f97915815674_.log 2024-08-20T22:00:31.5923667Z Running 29 items in this shard: test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_all_gather_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_reduce_scatter_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_get_unexposed_collectives 2024-08-20T22:00:31.5949673Z 2024-08-20T22:00:31.5950387Z Running distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 ... [2024-08-20 22:00:31.589812] 2024-08-20T22:00:31.5951270Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:00:31.5953594Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_sharded_grad_scaler.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:00:31.590243] 2024-08-20T22:02:25.7198417Z 2024-08-20T22:02:25.7200596Z distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_sharded_grad_scaler_1.1_dc8f8946aea44b6f_.log 2024-08-20T22:02:25.7222582Z Running 20 items in this shard: test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_grad_scaling, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_inf_gradients_skip_optim_step, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_scaling_unscaling_sparse, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_sharded_grad_scaler_found_inf 2024-08-20T22:02:25.7243816Z 2024-08-20T22:02:25.7244494Z Running distributed/pipelining/test_microbatch 1/1 ... [2024-08-20 22:02:25.720224] 2024-08-20T22:02:25.7245356Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:02:25.7247657Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_microbatch.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:02:25.720710] 2024-08-20T22:02:29.9933856Z 2024-08-20T22:02:29.9935934Z distributed/pipelining/test_microbatch 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_microbatch_1.1_6054e40f52a72274_.log 2024-08-20T22:02:29.9939575Z Running 2 items in this shard: test/distributed/pipelining/test_microbatch.py::MicrobatchTests::test_chunk_spec, test/distributed/pipelining/test_microbatch.py::MicrobatchTests::test_split_and_merge 2024-08-20T22:02:29.9941019Z 2024-08-20T22:02:29.9941619Z Running distributed/_composable/test_contract 1/1 ... [2024-08-20 22:02:29.993688] 2024-08-20T22:02:29.9942438Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:02:29.9946364Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_contract.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:02:29.994201] 2024-08-20T22:02:33.8661289Z 2024-08-20T22:02:33.8663451Z distributed/_composable/test_contract 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_contract_1.1_f6c14164118116e3_.log 2024-08-20T22:02:33.8670095Z Running 5 items in this shard: test/distributed/_composable/test_contract.py::TestContract::test_add_hooks, test/distributed/_composable/test_contract.py::TestContract::test_modify_fqn, test/distributed/_composable/test_contract.py::TestContract::test_multi_module_api, test/distributed/_composable/test_contract.py::TestContract::test_registry, test/distributed/_composable/test_contract.py::TestContract::test_state 2024-08-20T22:02:33.8673342Z 2024-08-20T22:02:33.8673965Z Running distributed/_tensor/test_common_rules 1/1 ... [2024-08-20 22:02:33.866413] 2024-08-20T22:02:33.8674785Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:02:33.8677020Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_common_rules.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:02:33.866873] 2024-08-20T22:03:15.5677641Z 2024-08-20T22:03:15.5679922Z distributed/_tensor/test_common_rules 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_common_rules_1.1_fae0db91b9ef821f_.log 2024-08-20T22:03:15.5689837Z Running 10 items in this shard: test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_basic_propagation, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_errors, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_linearity, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_merge_sharding, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_pointwise_propagation, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_enforce_sharding_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_rules_broadcasting, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_rules_suggestion 2024-08-20T22:03:15.5696167Z 2024-08-20T22:03:15.5696961Z Running distributed/_tensor/experimental/test_register_sharding 1/1 ... [2024-08-20 22:03:15.567893] 2024-08-20T22:03:15.5697914Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:03:15.5700279Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/experimental/test_register_sharding.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:03:15.568339] 2024-08-20T22:03:28.6079990Z 2024-08-20T22:03:28.6082384Z distributed/_tensor/experimental/test_register_sharding 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.experimental.test_register_sharding_1.1_92abc9a2fb1112ea_.log 2024-08-20T22:03:28.6085521Z Running 2 items in this shard: test/distributed/_tensor/experimental/test_register_sharding.py::TestRegisterSharding::test_argmax, test/distributed/_tensor/experimental/test_register_sharding.py::TestRegisterSharding::test_softmax_fwd 2024-08-20T22:03:28.6087175Z 2024-08-20T22:03:28.6087805Z Running distributed/optim/test_named_optimizer 1/1 ... [2024-08-20 22:03:28.608246] 2024-08-20T22:03:28.6088599Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:03:28.6090858Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/optim/test_named_optimizer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:03:28.608677] 2024-08-20T22:03:32.0034776Z 2024-08-20T22:03:32.0036832Z distributed/optim/test_named_optimizer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_named_optimizer_1.1_78b5b2ec144aa314_.log 2024-08-20T22:03:32.0038815Z 2024-08-20T22:03:32.0040914Z Running distributed/_tensor/test_api 1/1 ... [2024-08-20 22:03:32.003757] 2024-08-20T22:03:32.0041642Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:03:32.0046017Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:03:32.004194] 2024-08-20T22:04:06.4410348Z 2024-08-20T22:04:06.4412152Z distributed/_tensor/test_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_api_1.1_7169204f220c5176_.log 2024-08-20T22:04:06.4418684Z Running 8 items in this shard: test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_casting, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_input_fn_output_fn, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_input_fn_output_fn_warning, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_meta, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor_errors, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor_uneven_sharding 2024-08-20T22:04:06.4423359Z 2024-08-20T22:04:06.4423914Z Running distributed/test_device_mesh 1/1 ... [2024-08-20 22:04:06.441371] 2024-08-20T22:04:06.4424663Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:04:06.4426846Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_device_mesh.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:04:06.441855] 2024-08-20T22:06:46.9176594Z 2024-08-20T22:06:46.9180542Z distributed/test_device_mesh 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_device_mesh_1.1_19030377338943fc_.log 2024-08-20T22:06:46.9204752Z Running 41 items in this shard: test/distributed/test_device_mesh.py::DeviceMeshTestGlooBackend::test_device_mesh_reuse_default_group, test/distributed/test_device_mesh.py::DeviceMeshTest::test_assert_invalid_mesh_tensor, test/distributed/test_device_mesh.py::DeviceMeshTest::test_device_mesh_2d, test/distributed/test_device_mesh.py::DeviceMeshTest::test_device_mesh_init_backend, test/distributed/test_device_mesh.py::DeviceMeshTest::test_fake_pg_device_mesh, test/distributed/test_device_mesh.py::DeviceMeshTest::test_from_group_with_global_pg, test/distributed/test_device_mesh.py::DeviceMeshTest::test_from_group_with_invalid_mesh, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_group_and_get_all_groups, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_local_rank, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_local_rank_raises_exception, test/distributed/test_device_mesh.py::DeviceMeshTest::test_init_process_group, test/distributed/test_device_mesh.py::DeviceMeshTest::test_raises_invalid_device_type, test/distributed/test_device_mesh.py::DeviceMeshTest::test_set_mesh_dim_group_options, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_hash, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_nd, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_parent_child_hash, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_from_group_with_mesh_shape, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_get_local_rank_3d, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_init_device_mesh, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_raises_duplicate_mesh_dim_names, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_raises_mesh_shape_mesh_dim_names_mismatch, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_cache_and_reuse_submesh_slice_result, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_flatten_mesh, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_1d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_2d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_3d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_3d_noncontiguous_slicing, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_invalid_mesh_dim_name, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_no_mesh_dim_found, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_mesh_dim_by_name, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_root_mesh, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_root_mesh_dim_exist, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_root_mesh_dim_not_exist, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_all_gather_uneven, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_1d, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_nd, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_reduce_scatter_contiguous, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_reduce_scatter_uneven, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_1d, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_nd, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_uneven 2024-08-20T22:06:46.9227660Z 2024-08-20T22:06:46.9228329Z Running distributed/checkpoint/test_save_load_api 1/1 ... [2024-08-20 22:06:46.917969] 2024-08-20T22:06:46.9229218Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:06:46.9231505Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_save_load_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:06:46.918439] 2024-08-20T22:06:54.1965106Z 2024-08-20T22:06:54.1967389Z distributed/checkpoint/test_save_load_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_save_load_api_1.1_4c62f319a2161602_.log 2024-08-20T22:06:54.1969588Z Running 1 items in this shard: test/distributed/checkpoint/test_save_load_api.py::TestSaveAndLoadAPI::test_auto_detect 2024-08-20T22:06:54.1970501Z 2024-08-20T22:06:54.1971128Z Running distributed/checkpoint/test_dedup_tensors 1/1 ... [2024-08-20 22:06:54.196784] 2024-08-20T22:06:54.1972023Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:06:54.1976473Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_dedup_tensors.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:06:54.197221] 2024-08-20T22:06:58.0188956Z 2024-08-20T22:06:58.0191260Z distributed/checkpoint/test_dedup_tensors 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dedup_tensors_1.1_30a7dfb6dcfa3a03_.log 2024-08-20T22:06:58.0193466Z Running 1 items in this shard: test/distributed/checkpoint/test_dedup_tensors.py::TestDedupTensor::test_dedup_shards 2024-08-20T22:06:58.0194365Z 2024-08-20T22:06:58.0195005Z Running distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 ... [2024-08-20 22:06:58.019138] 2024-08-20T22:06:58.0195855Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:06:58.0200071Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/fsdp/test_fsdp_dsd.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:06:58.019565] 2024-08-20T22:07:30.7949019Z 2024-08-20T22:07:30.7951127Z distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.fsdp.test_fsdp_dsd_1.1_d1812b43a49eea07_.log 2024-08-20T22:07:30.7958940Z Running 6 items in this shard: test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_1d_fsdp_cpu_offload_full_model_state_dict, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_1d_fsdp_get_model_state_dict, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_save_with_fsdp1_and_load_with_fsdp2, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_save_with_fsdp1_and_load_with_fsdp2_tp, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_save_with_fsdp2_tp_and_load_with_tp, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_save_with_tp_and_load_with_fsdp2_tp 2024-08-20T22:07:30.7964643Z 2024-08-20T22:07:30.7965274Z Running distributed/test_c10d_spawn_gloo 1/1 ... [2024-08-20 22:07:30.795201] 2024-08-20T22:07:30.7966072Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:07:30.7968366Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_spawn_gloo.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:07:30.795652] 2024-08-20T22:08:51.8796158Z 2024-08-20T22:08:51.8799052Z distributed/test_c10d_spawn_gloo 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_spawn_gloo_1.1_04ae7895f08c6965_.log 2024-08-20T22:08:51.8807076Z Running 11 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cpu, test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cuda, test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_rnn, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_gather, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all_single, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_allreduce, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_broadcast, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_gather, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_reduce, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_scatter 2024-08-20T22:08:51.8814988Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cpu 2024-08-20T22:08:51.8816736Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cuda 2024-08-20T22:08:51.8818481Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_rnn 2024-08-20T22:08:51.8820138Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_gather 2024-08-20T22:08:51.8821768Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all 2024-08-20T22:08:51.8823435Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all_single 2024-08-20T22:08:51.8825543Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_allreduce 2024-08-20T22:08:51.8827141Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_broadcast 2024-08-20T22:08:51.8828745Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_gather 2024-08-20T22:08:51.8830323Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_reduce 2024-08-20T22:08:51.8831911Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_scatter 2024-08-20T22:08:51.8832805Z 2024-08-20T22:08:51.8833465Z Running distributed/tensor/parallel/test_tp_style 1/1 ... [2024-08-20 22:08:51.880346] 2024-08-20T22:08:51.8834340Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:08:51.8836842Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_style.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:08:51.880824] 2024-08-20T22:09:37.3854886Z 2024-08-20T22:09:37.3857200Z distributed/tensor/parallel/test_tp_style 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_style_1.1_fa7ae7fef5383a81_.log 2024-08-20T22:09:37.3865214Z Running 9 items in this shard: test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_colwise_parallel_embedding, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_colwise_parallel_style, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_input, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_input_multiple_inputs, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_kwargs_input, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_output, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_rowwise_parallel_embedding, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_rowwise_parallel_style, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_sequence_parallel_style 2024-08-20T22:09:37.3871923Z 2024-08-20T22:09:37.3872491Z Running distributed/_tensor/test_attention 1/1 ... [2024-08-20 22:09:37.385712] 2024-08-20T22:09:37.3873292Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:09:37.3875523Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_attention.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:09:37.386135] 2024-08-20T22:10:05.1564441Z 2024-08-20T22:10:05.1566772Z distributed/_tensor/test_attention 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_attention_1.1_7d913c226703ab6a_.log 2024-08-20T22:10:05.1574509Z Running 8 items in this shard: test/distributed/_tensor/test_attention.py::RingAttentionTest::test_is_causal_behavior, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_custom_transformer, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_native_transformer_is_causal_False, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_native_transformer_is_causal_True, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa_is_causal_False_compiled_False_backend0, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa_is_causal_False_compiled_True_backend0, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa_is_causal_True_compiled_False_backend0, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa_is_causal_True_compiled_True_backend0 2024-08-20T22:10:05.1580856Z 2024-08-20T22:10:05.6588285Z Running test batch 'tests to run' cost 679.46 seconds 2024-08-20T22:10:06.0787638Z 2024-08-20T22:10:06.0788437Z real 11m24.759s 2024-08-20T22:10:06.0788908Z user 33m42.047s 2024-08-20T22:10:06.0789251Z sys 10m30.657s 2024-08-20T22:10:06.0789625Z + assert_git_not_dirty 2024-08-20T22:10:06.0790468Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-08-20T22:10:06.0791214Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *xla* ]] 2024-08-20T22:10:06.0794750Z ++ git status --porcelain 2024-08-20T22:10:06.0795584Z ++ grep -v '?? third_party' 2024-08-20T22:10:09.0662260Z ++ true 2024-08-20T22:10:09.0663195Z + git_status= 2024-08-20T22:10:09.0663814Z + [[ -n '' ]] 2024-08-20T22:10:09.0664423Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-08-20T22:10:09.0665013Z + [[ 3 == 1 ]] 2024-08-20T22:10:09.0665361Z + [[ 3 == 1 ]] 2024-08-20T22:10:09.0665723Z + cleanup_workspace 2024-08-20T22:10:09.0667027Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-20T22:10:09.0668464Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-20T22:10:09.0669667Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-20T22:10:09.0670480Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-20T22:10:09.0671535Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-20T22:10:09.0672670Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-20T22:10:09.0673572Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-20T22:10:09.6664303Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-20T22:10:09.6665011Z cat test/**/*_toprint.log || true 2024-08-20T22:10:09.6675791Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:09.6676422Z env: 2024-08-20T22:10:09.6676743Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:09.6677307Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:09.6678238Z DOCKER_CONTAINER_ID: 42ff1b32f81951577dea4e347041b56c1d24119a3992eaf37c41c02aecf51e0b 2024-08-20T22:10:09.6679027Z ##[endgroup] 2024-08-20T22:10:09.6770721Z cat: 'test/**/*_toprint.log': No such file or directory 2024-08-20T22:10:09.6810868Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-20T22:10:09.6811758Z kill "$MONITOR_SCRIPT_PID" 2024-08-20T22:10:09.6818753Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:09.6819363Z env: 2024-08-20T22:10:09.6819703Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:09.6820266Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:09.6821196Z DOCKER_CONTAINER_ID: 42ff1b32f81951577dea4e347041b56c1d24119a3992eaf37c41c02aecf51e0b 2024-08-20T22:10:09.6822028Z MONITOR_SCRIPT_PID: 59905 2024-08-20T22:10:09.6822462Z ##[endgroup] 2024-08-20T22:10:09.7070329Z Prepare all required actions 2024-08-20T22:10:09.7070926Z Getting action download info 2024-08-20T22:10:09.9274778Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-20T22:10:10.1032346Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-20T22:10:10.1032917Z with: 2024-08-20T22:10:10.1033579Z file-suffix: test-distributed-3-3-amz2023.linux.8xlarge.nvidia.gpu_29025850949 2024-08-20T22:10:10.1034418Z s3-bucket: gha-artifacts 2024-08-20T22:10:10.1034814Z env: 2024-08-20T22:10:10.1035143Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:10.1035696Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:10.1036598Z DOCKER_CONTAINER_ID: 42ff1b32f81951577dea4e347041b56c1d24119a3992eaf37c41c02aecf51e0b 2024-08-20T22:10:10.1038241Z ##[endgroup] 2024-08-20T22:10:10.1073530Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-20T22:10:10.1074370Z # Remove any previous test jsons if they exist 2024-08-20T22:10:10.1074991Z rm -f test-jsons-*.zip 2024-08-20T22:10:10.1075642Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-20T22:10:10.1082508Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:10.1083123Z env: 2024-08-20T22:10:10.1083462Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:10.1084030Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:10.1084955Z DOCKER_CONTAINER_ID: 42ff1b32f81951577dea4e347041b56c1d24119a3992eaf37c41c02aecf51e0b 2024-08-20T22:10:10.1086074Z FILE_SUFFIX: test-distributed-3-3-amz2023.linux.8xlarge.nvidia.gpu_29025850949 2024-08-20T22:10:10.1086908Z ##[endgroup] 2024-08-20T22:10:10.1444623Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-20T22:10:10.1490381Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-20T22:10:10.1491233Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-20T22:10:10.1500363Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-20T22:10:10.1507986Z adding: test/slow_tests.json (deflated 82%) 2024-08-20T22:10:10.1510882Z adding: test/test-reports/td_exclusions-9d559bcb2fb0d23ca246.json (deflated 86%) 2024-08-20T22:10:10.1528375Z adding: test/.pytorch-disabled-tests.json (deflated 88%) 2024-08-20T22:10:10.1565148Z ##[group]Run # Remove any previous test reports if they exist 2024-08-20T22:10:10.1566302Z # Remove any previous test reports if they exist 2024-08-20T22:10:10.1566980Z rm -f test-reports-*.zip 2024-08-20T22:10:10.1567718Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-20T22:10:10.1574761Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:10.1575384Z env: 2024-08-20T22:10:10.1575728Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:10.1576270Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:10.1577201Z DOCKER_CONTAINER_ID: 42ff1b32f81951577dea4e347041b56c1d24119a3992eaf37c41c02aecf51e0b 2024-08-20T22:10:10.1578334Z FILE_SUFFIX: test-distributed-3-3-amz2023.linux.8xlarge.nvidia.gpu_29025850949 2024-08-20T22:10:10.1579123Z ##[endgroup] 2024-08-20T22:10:10.1808591Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor_compile/distributed._tensor.test_dtensor_compile-377c81dfc601eaaf.xml (deflated 85%) 2024-08-20T22:10:10.1810995Z adding: test/test-reports/python-pytest/distributed._tensor.debug.test_op_coverage/distributed._tensor.debug.test_op_coverage-966443b9de9d8d95.xml (deflated 35%) 2024-08-20T22:10:10.1813307Z adding: test/test-reports/python-pytest/distributed.test_c10d_functional_native/distributed.test_c10d_functional_native-036396088dc4c26a.xml (deflated 90%) 2024-08-20T22:10:10.1815521Z adding: test/test-reports/python-pytest/distributed.pipelining.test_pipe/distributed.pipelining.test_pipe-74cff0c3d02b153f.xml (deflated 65%) 2024-08-20T22:10:10.1818493Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_micro_pipeline_tp/distributed.tensor.parallel.test_micro_pipeline_tp-12f8f20f2d0c953a.xml (deflated 95%) 2024-08-20T22:10:10.1821130Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_sharded_grad_scaler/distributed.fsdp.test_fsdp_sharded_grad_scaler-52d0100583f23402.xml (deflated 88%) 2024-08-20T22:10:10.1823511Z adding: test/test-reports/python-pytest/distributed.pipelining.test_microbatch/distributed.pipelining.test_microbatch-9646021d296f5786.xml (deflated 54%) 2024-08-20T22:10:10.1826020Z adding: test/test-reports/python-pytest/distributed._composable.test_contract/distributed._composable.test_contract-a05b5dc7e172b402.xml (deflated 69%) 2024-08-20T22:10:10.1828258Z adding: test/test-reports/python-pytest/distributed._tensor.test_common_rules/distributed._tensor.test_common_rules-e82cad4aa32bcb83.xml (deflated 77%) 2024-08-20T22:10:10.1831227Z adding: test/test-reports/python-pytest/distributed._tensor.experimental.test_register_sharding/distributed._tensor.experimental.test_register_sharding-766311a43c6b0af3.xml (deflated 49%) 2024-08-20T22:10:10.1833562Z adding: test/test-reports/python-pytest/distributed._tensor.test_api/distributed._tensor.test_api-a79968a0781dbbb3.xml (deflated 75%) 2024-08-20T22:10:10.1835547Z adding: test/test-reports/python-pytest/distributed.test_device_mesh/distributed.test_device_mesh-c384395b596a2f9b.xml (deflated 85%) 2024-08-20T22:10:10.1838008Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_save_load_api/distributed.checkpoint.test_save_load_api-e562ed1dbc238bf7.xml (deflated 50%) 2024-08-20T22:10:10.1840439Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_dedup_tensors/distributed.checkpoint.test_dedup_tensors-d081f2481c586c97.xml (deflated 37%) 2024-08-20T22:10:10.1842860Z adding: test/test-reports/python-pytest/distributed.checkpoint.fsdp.test_fsdp_dsd/distributed.checkpoint.fsdp.test_fsdp_dsd-d707daca36dd17bb.xml (deflated 76%) 2024-08-20T22:10:10.1845122Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-ee63082e012e8d00.xml (deflated 36%) 2024-08-20T22:10:10.1847208Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-4c7f2b5749796c7a.xml (deflated 36%) 2024-08-20T22:10:10.1849281Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-85511185082dfd61.xml (deflated 35%) 2024-08-20T22:10:10.1851357Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-f15f8e06dbd42190.xml (deflated 36%) 2024-08-20T22:10:10.1853414Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-083738ad9257cc11.xml (deflated 36%) 2024-08-20T22:10:10.1855495Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-384107522b5d65e0.xml (deflated 35%) 2024-08-20T22:10:10.1857578Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-c3a96e991b5d0ca3.xml (deflated 36%) 2024-08-20T22:10:10.1859711Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-3a42eab74a42faa0.xml (deflated 35%) 2024-08-20T22:10:10.1861774Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-48bc375abd8f619d.xml (deflated 36%) 2024-08-20T22:10:10.1863852Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-0404fd5005bccbd1.xml (deflated 35%) 2024-08-20T22:10:10.1865934Z adding: test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-8c07d4d1f4c069b7.xml (deflated 35%) 2024-08-20T22:10:10.1868212Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_style/distributed.tensor.parallel.test_tp_style-ec3b5938fb162ee7.xml (deflated 77%) 2024-08-20T22:10:10.1870649Z adding: test/test-reports/python-pytest/distributed._tensor.test_attention/distributed._tensor.test_attention-f9aa24e364b971d0.xml (deflated 80%) 2024-08-20T22:10:10.1903684Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-20T22:10:10.1904473Z # Remove any previous usage logs if they exist 2024-08-20T22:10:10.1905095Z rm -f logs-*.zip 2024-08-20T22:10:10.1905927Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-20T22:10:10.1906921Z # so check to see if the file exists first 2024-08-20T22:10:10.1907535Z if [ -f 'usage_log.txt' ]; then 2024-08-20T22:10:10.1908190Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-20T22:10:10.1908791Z fi 2024-08-20T22:10:10.1909390Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-20T22:10:10.1910106Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-20T22:10:10.1910731Z fi 2024-08-20T22:10:10.1917071Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:10.1917696Z env: 2024-08-20T22:10:10.1918028Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:10.1918571Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:10.1919492Z DOCKER_CONTAINER_ID: 42ff1b32f81951577dea4e347041b56c1d24119a3992eaf37c41c02aecf51e0b 2024-08-20T22:10:10.1920622Z FILE_SUFFIX: test-distributed-3-3-amz2023.linux.8xlarge.nvidia.gpu_29025850949 2024-08-20T22:10:10.1921415Z ##[endgroup] 2024-08-20T22:10:10.1977967Z adding: usage_log.txt (deflated 92%) 2024-08-20T22:10:10.2215885Z adding: test/test-reports/distributed._tensor.test_dtensor_compile_1.1_9874889ba8acc94c_.log (deflated 84%) 2024-08-20T22:10:10.2217503Z adding: test/test-reports/distributed._tensor.debug.test_op_coverage_1.1_8fd81d213d1f04be_.log (deflated 54%) 2024-08-20T22:10:10.2219130Z adding: test/test-reports/distributed.test_c10d_functional_native_1.1_f71255279909e400_.log (deflated 87%) 2024-08-20T22:10:10.2220671Z adding: test/test-reports/distributed.pipelining.test_pipe_1.1_3facb31e2463593c_.log (deflated 62%) 2024-08-20T22:10:10.2222290Z adding: test/test-reports/distributed.tensor.parallel.test_micro_pipeline_tp_1.1_dc94f97915815674_.log (deflated 90%) 2024-08-20T22:10:10.2224569Z adding: test/test-reports/distributed.fsdp.test_fsdp_sharded_grad_scaler_1.1_dc8f8946aea44b6f_.log (deflated 93%) 2024-08-20T22:10:10.2226202Z adding: test/test-reports/distributed.pipelining.test_microbatch_1.1_6054e40f52a72274_.log (deflated 58%) 2024-08-20T22:10:10.2227777Z adding: test/test-reports/distributed._composable.test_contract_1.1_f6c14164118116e3_.log (deflated 66%) 2024-08-20T22:10:10.2229304Z adding: test/test-reports/distributed._tensor.test_common_rules_1.1_fae0db91b9ef821f_.log (deflated 76%) 2024-08-20T22:10:10.2230982Z adding: test/test-reports/distributed._tensor.experimental.test_register_sharding_1.1_92abc9a2fb1112ea_.log (deflated 60%) 2024-08-20T22:10:10.2232677Z adding: test/test-reports/distributed.optim.test_named_optimizer_1.1_78b5b2ec144aa314_.log (deflated 35%) 2024-08-20T22:10:10.2234159Z adding: test/test-reports/distributed._tensor.test_api_1.1_7169204f220c5176_.log (deflated 89%) 2024-08-20T22:10:10.2235541Z adding: test/test-reports/distributed.test_device_mesh_1.1_19030377338943fc_.log (deflated 83%) 2024-08-20T22:10:10.2237038Z adding: test/test-reports/distributed.checkpoint.test_save_load_api_1.1_4c62f319a2161602_.log (deflated 50%) 2024-08-20T22:10:10.2238938Z adding: test/test-reports/distributed.checkpoint.test_dedup_tensors_1.1_30a7dfb6dcfa3a03_.log (deflated 52%) 2024-08-20T22:10:10.2240584Z adding: test/test-reports/distributed.checkpoint.fsdp.test_fsdp_dsd_1.1_d1812b43a49eea07_.log (deflated 84%) 2024-08-20T22:10:10.2242114Z adding: test/test-reports/distributed.test_c10d_spawn_gloo_1.1_04ae7895f08c6965_.log (deflated 93%) 2024-08-20T22:10:10.2244099Z adding: test/test-reports/distributed.tensor.parallel.test_tp_style_1.1_fa7ae7fef5383a81_.log (deflated 76%) 2024-08-20T22:10:10.2245731Z adding: test/test-reports/distributed._tensor.test_attention_1.1_7d913c226703ab6a_.log (deflated 75%) 2024-08-20T22:10:10.2278158Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-20T22:10:10.2279401Z # Remove any previous debugging artifacts if they exist 2024-08-20T22:10:10.2280092Z rm -f debug-*.zip 2024-08-20T22:10:10.2280557Z if [ -d 'test/debug' ]; then 2024-08-20T22:10:10.2281163Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-20T22:10:10.2281751Z fi 2024-08-20T22:10:10.2288071Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:10.2288672Z env: 2024-08-20T22:10:10.2289003Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:10.2289728Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:10.2290632Z DOCKER_CONTAINER_ID: 42ff1b32f81951577dea4e347041b56c1d24119a3992eaf37c41c02aecf51e0b 2024-08-20T22:10:10.2291766Z FILE_SUFFIX: test-distributed-3-3-amz2023.linux.8xlarge.nvidia.gpu_29025850949 2024-08-20T22:10:10.2292574Z ##[endgroup] 2024-08-20T22:10:10.2398943Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:10:10.2399477Z with: 2024-08-20T22:10:10.2399817Z s3-bucket: gha-artifacts 2024-08-20T22:10:10.2400355Z s3-prefix: pytorch/pytorch/10479309237/1/artifact 2024-08-20T22:10:10.2400923Z retention-days: 14 2024-08-20T22:10:10.2401331Z if-no-files-found: warn 2024-08-20T22:10:10.2401765Z path: test-jsons-*.zip 2024-08-20T22:10:10.2402160Z name: artifact 2024-08-20T22:10:10.2402527Z region: us-east-1 2024-08-20T22:10:10.2402888Z env: 2024-08-20T22:10:10.2403195Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:10.2403749Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:10.2404680Z DOCKER_CONTAINER_ID: 42ff1b32f81951577dea4e347041b56c1d24119a3992eaf37c41c02aecf51e0b 2024-08-20T22:10:10.2405464Z ##[endgroup] 2024-08-20T22:10:10.6855722Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T22:10:10.6856747Z With the provided path, there will be 1 file uploaded 2024-08-20T22:10:10.6857911Z Uploading to s3 prefix: pytorch/pytorch/10479309237/1/artifact 2024-08-20T22:10:10.8743588Z Starting upload of test-jsons-test-distributed-3-3-amz2023.linux.8xlarge.nvidia.gpu_29025850949.zip 2024-08-20T22:10:11.0380313Z Finished upload of test-jsons-test-distributed-3-3-amz2023.linux.8xlarge.nvidia.gpu_29025850949.zip 2024-08-20T22:10:11.0576533Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:10:11.0577072Z with: 2024-08-20T22:10:11.0577412Z s3-bucket: gha-artifacts 2024-08-20T22:10:11.0577953Z s3-prefix: pytorch/pytorch/10479309237/1/artifact 2024-08-20T22:10:11.0578541Z retention-days: 14 2024-08-20T22:10:11.0578934Z if-no-files-found: error 2024-08-20T22:10:11.0579404Z path: test-reports-*.zip 2024-08-20T22:10:11.0579828Z name: artifact 2024-08-20T22:10:11.0580176Z region: us-east-1 2024-08-20T22:10:11.0580929Z env: 2024-08-20T22:10:11.0581265Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:11.0581831Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:11.0582754Z DOCKER_CONTAINER_ID: 42ff1b32f81951577dea4e347041b56c1d24119a3992eaf37c41c02aecf51e0b 2024-08-20T22:10:11.0583557Z ##[endgroup] 2024-08-20T22:10:11.4996625Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T22:10:11.4997616Z With the provided path, there will be 1 file uploaded 2024-08-20T22:10:11.4998386Z Uploading to s3 prefix: pytorch/pytorch/10479309237/1/artifact 2024-08-20T22:10:11.5050043Z Starting upload of test-reports-test-distributed-3-3-amz2023.linux.8xlarge.nvidia.gpu_29025850949.zip 2024-08-20T22:10:11.7110372Z Finished upload of test-reports-test-distributed-3-3-amz2023.linux.8xlarge.nvidia.gpu_29025850949.zip 2024-08-20T22:10:11.7311289Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:10:11.7312052Z with: 2024-08-20T22:10:11.7312397Z s3-bucket: gha-artifacts 2024-08-20T22:10:11.7313166Z s3-prefix: pytorch/pytorch/10479309237/1/artifact 2024-08-20T22:10:11.7313776Z retention-days: 14 2024-08-20T22:10:11.7314191Z if-no-files-found: ignore 2024-08-20T22:10:11.7314632Z path: logs-*.zip 2024-08-20T22:10:11.7314989Z name: artifact 2024-08-20T22:10:11.7315355Z region: us-east-1 2024-08-20T22:10:11.7315719Z env: 2024-08-20T22:10:11.7316033Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:11.7316596Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:11.7317520Z DOCKER_CONTAINER_ID: 42ff1b32f81951577dea4e347041b56c1d24119a3992eaf37c41c02aecf51e0b 2024-08-20T22:10:11.7318309Z ##[endgroup] 2024-08-20T22:10:12.1797536Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T22:10:12.1798500Z With the provided path, there will be 1 file uploaded 2024-08-20T22:10:12.1799253Z Uploading to s3 prefix: pytorch/pytorch/10479309237/1/artifact 2024-08-20T22:10:12.1852003Z Starting upload of logs-test-distributed-3-3-amz2023.linux.8xlarge.nvidia.gpu_29025850949.zip 2024-08-20T22:10:12.2803179Z Finished upload of logs-test-distributed-3-3-amz2023.linux.8xlarge.nvidia.gpu_29025850949.zip 2024-08-20T22:10:12.2997676Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:10:12.2998212Z with: 2024-08-20T22:10:12.2998557Z s3-bucket: gha-artifacts 2024-08-20T22:10:12.2999097Z s3-prefix: pytorch/pytorch/10479309237/1/artifact 2024-08-20T22:10:12.2999671Z retention-days: 14 2024-08-20T22:10:12.3000081Z if-no-files-found: ignore 2024-08-20T22:10:12.3000522Z path: debug-*.zip 2024-08-20T22:10:12.3000887Z name: artifact 2024-08-20T22:10:12.3001255Z region: us-east-1 2024-08-20T22:10:12.3001619Z env: 2024-08-20T22:10:12.3001929Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:12.3002484Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:12.3003405Z DOCKER_CONTAINER_ID: 42ff1b32f81951577dea4e347041b56c1d24119a3992eaf37c41c02aecf51e0b 2024-08-20T22:10:12.3004208Z ##[endgroup] 2024-08-20T22:10:12.7334936Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-20T22:10:12.7525193Z ##[group]Run # shellcheck disable=SC2156 2024-08-20T22:10:12.7525825Z # shellcheck disable=SC2156 2024-08-20T22:10:12.7526832Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-20T22:10:12.7534696Z shell: /usr/bin/bash -e {0} 2024-08-20T22:10:12.7535145Z env: 2024-08-20T22:10:12.7535483Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:12.7536032Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:12.7536965Z DOCKER_CONTAINER_ID: 42ff1b32f81951577dea4e347041b56c1d24119a3992eaf37c41c02aecf51e0b 2024-08-20T22:10:12.7538250Z ##[endgroup] 2024-08-20T22:10:13.0382415Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-20T22:10:13.0383139Z with: 2024-08-20T22:10:13.0383462Z env: 2024-08-20T22:10:13.0383787Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:13.0384332Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:13.0385250Z DOCKER_CONTAINER_ID: 42ff1b32f81951577dea4e347041b56c1d24119a3992eaf37c41c02aecf51e0b 2024-08-20T22:10:13.0386048Z ##[endgroup] 2024-08-20T22:10:13.0413076Z ##[group]Run set -eou pipefail 2024-08-20T22:10:13.0413580Z set -eou pipefail 2024-08-20T22:10:13.0414007Z  2024-08-20T22:10:13.0414636Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-20T22:10:13.0415425Z for _ in $(seq 1440); do 2024-08-20T22:10:13.0416007Z  # Break if no ssh session exists anymore 2024-08-20T22:10:13.0416625Z  if [ "$(who)" = "" ]; then 2024-08-20T22:10:13.0417104Z  break 2024-08-20T22:10:13.0417537Z  fi 2024-08-20T22:10:13.0417890Z  echo "." 2024-08-20T22:10:13.0418287Z  sleep 5 2024-08-20T22:10:13.0418674Z done 2024-08-20T22:10:13.0425547Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:13.0426171Z env: 2024-08-20T22:10:13.0426504Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:13.0427047Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:13.0427971Z DOCKER_CONTAINER_ID: 42ff1b32f81951577dea4e347041b56c1d24119a3992eaf37c41c02aecf51e0b 2024-08-20T22:10:13.0428779Z ##[endgroup] 2024-08-20T22:10:13.0459066Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-20T22:10:13.0519668Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:10:13.0520643Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:10:13.0521384Z # shellcheck disable=SC2046 2024-08-20T22:10:13.0521963Z docker stop $(docker ps -q) || true 2024-08-20T22:10:13.0522543Z # Prune all of the docker images 2024-08-20T22:10:13.0523106Z docker system prune -af 2024-08-20T22:10:13.0529880Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:13.0530504Z env: 2024-08-20T22:10:13.0530836Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:13.0531378Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:10:13.0532296Z DOCKER_CONTAINER_ID: 42ff1b32f81951577dea4e347041b56c1d24119a3992eaf37c41c02aecf51e0b 2024-08-20T22:10:13.0533096Z ##[endgroup] 2024-08-20T22:10:13.7752686Z 42ff1b32f819 2024-08-20T22:10:14.2678688Z Deleted Containers: 2024-08-20T22:10:14.2679355Z 42ff1b32f81951577dea4e347041b56c1d24119a3992eaf37c41c02aecf51e0b 2024-08-20T22:10:14.2679918Z 2024-08-20T22:10:20.3369557Z Deleted Images: 2024-08-20T22:10:20.3371373Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:10:20.3373965Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9@sha256:0b3aa0836d4f18656c98dcf355b46cbae2123243424ebbed0929206c3d798126 2024-08-20T22:10:20.3375791Z deleted: sha256:35f4945dfeab4592c8657de7107f96a9fd9a51999f582a0064937246789c0825 2024-08-20T22:10:20.3376865Z deleted: sha256:5aeec7923eaf143c8c860346742d9e9c0a627a72c06c4a8574f03d7359497d8a 2024-08-20T22:10:20.3377925Z deleted: sha256:8c2e3c359e6f8f863100522d9503272ad2a22af4d3245cb32988cf7f75dff9bf 2024-08-20T22:10:20.3378986Z deleted: sha256:b9c8bef934615ea09715897fa2252d4ff9a33ae4b0b99d65f4314edb98a08508 2024-08-20T22:10:20.3380015Z deleted: sha256:7325e907409be83e224427131a7e02928ab8ca8880e53b975b309a17c9887b3d 2024-08-20T22:10:20.3381076Z deleted: sha256:93c3c0295ffeb06e6b45fd31a03c7b9de49d6a6404a889643dcb02a29956a46d 2024-08-20T22:10:20.3382143Z deleted: sha256:75434bb2c96c820ddef9e3cd15729787375a35257f82c07e9a2e49a63ccd258b 2024-08-20T22:10:20.3383528Z deleted: sha256:2b55c8d634372339b274a7068c1e8058ff9ca9b828c020d97462adddfbbab68e 2024-08-20T22:10:20.3384630Z deleted: sha256:f4242713ef5734c93522c188d00d66c11c2ea30e1edc576f7557feec4a71fb44 2024-08-20T22:10:20.3385690Z deleted: sha256:e3f0028f9a27ad44915c7ce616833089a5ac41163b1f3fd441c29448c9967bd6 2024-08-20T22:10:20.3386720Z deleted: sha256:157d6992f3a76ca9422961a443176f6c953d6d2609d8f26abc95e283177d8838 2024-08-20T22:10:20.3387776Z deleted: sha256:80725860ab404ed4a3e2d4f1c9179efb9628ddaaead438f0e2dfd45860821659 2024-08-20T22:10:20.3388861Z deleted: sha256:12612899d16f7cab2f6bcce2c8c2ce9a6aabe0f8c57d6a9ec3df95aebc24eb66 2024-08-20T22:10:20.3389943Z deleted: sha256:f892c2afba839178fe4f4480ffa99124c5788c76959cef3c751d31ffbc56532e 2024-08-20T22:10:20.3390984Z deleted: sha256:3783df37fa2f17df24597d0c9b48925712f7ec5cdb75993e04a750249242d945 2024-08-20T22:10:20.3392039Z deleted: sha256:03700104a8d899077ca0cf2dcca4902d103f736f5605fc2dcb1228b4fbb01cf8 2024-08-20T22:10:20.3393089Z deleted: sha256:c22e6b22d7d1206421c03a90e2a0d202a5b0293700ea39051d3062ab4e02212f 2024-08-20T22:10:20.3394128Z deleted: sha256:6a5fd513d78286d2a97e6118bdfaf09d345513de1afe00f7214930aab4e182ce 2024-08-20T22:10:20.3395247Z deleted: sha256:12d6046ca585372ab7221c75bf1ff9fbad344c584bfac3bf48490124569b09ce 2024-08-20T22:10:20.3396320Z deleted: sha256:11a342b5bdf2902b76dc6b741fa09aeada64dc62a4634c30e0b92dd49ecaa3e6 2024-08-20T22:10:20.3397388Z deleted: sha256:9c19056af41bd6f7d0316312696e9fb6ec9e16b9b2916bebb487ed1302846fca 2024-08-20T22:10:20.3398437Z deleted: sha256:dff9ad772a710e9d7a74de2fb2e60c8d87630855c5a75c7348b20d817de188c9 2024-08-20T22:10:20.3399503Z deleted: sha256:21911a7fa8e0ff3a30aa732134f3d24cb0e339bc36725f9980a7687e520e356a 2024-08-20T22:10:20.3400567Z deleted: sha256:f40a21fc1dad1aa6a566142c92a08afe87864068e6c360b6ade83ec521803286 2024-08-20T22:10:20.3401801Z deleted: sha256:617a6a309e07100390ac08e5ea2680a47e1f79fa72907e8b08ff46309a46f1ba 2024-08-20T22:10:20.3403759Z deleted: sha256:8d26fabf19fef49bc3a331f43fdaa661678b87d53109fc7540bfb9b44ce810bf 2024-08-20T22:10:20.3405749Z deleted: sha256:a2d242ed225d8f09315e3ee4597c3feb02f088337f59c95ddf440784d2be4f52 2024-08-20T22:10:20.3407169Z deleted: sha256:e10875c7907612ac7bd831be770f133f6eccaa347118b765c5d9ec776512f0b2 2024-08-20T22:10:20.3408225Z deleted: sha256:b5fa471f84592d75a2cf8625dcf9a285b9d76a7e7edccff81d385b66a46f9532 2024-08-20T22:10:20.3409301Z deleted: sha256:e42f988c8d3d938d4a1e3975407f64e4f962f3cabc1122a8fe98828492e34657 2024-08-20T22:10:20.3410358Z deleted: sha256:9507391f7f3611eb7c6f95061219d63ee3ecfd4cc81b3de44126c6d3c4667551 2024-08-20T22:10:20.3411383Z deleted: sha256:5e72605653a900048cf413ca242bb695bc9932595763521ed44f8fbe7166166a 2024-08-20T22:10:20.3412428Z deleted: sha256:435005331c11e4210ce1b81221a7054aee7ccaeb202bb2e09b45f2510f6ab254 2024-08-20T22:10:20.3413489Z deleted: sha256:a2f04d898ab3e490e7247db9696a9726e1f1f85012b46115a89aedab38930710 2024-08-20T22:10:20.3414517Z deleted: sha256:07b864569ca9924353696ae1560e08d337c4867cb398ce0e552e04c37d4cf6e5 2024-08-20T22:10:20.3415572Z deleted: sha256:907fa71a16fcb35456269eb13d0128ce7763243eaf3449c44cdd36b5b49312e1 2024-08-20T22:10:20.3416657Z deleted: sha256:f79790dfbd9f1c7e3fc5d45ffd290c41176ec11dffaf7a8c63c2c58b797fcdb6 2024-08-20T22:10:20.3417740Z deleted: sha256:406d461e2aa914e38d6eef5723bb633966284960223c93a97cd00be8a1cc0faf 2024-08-20T22:10:20.3418784Z deleted: sha256:119a4673c9c39860d87ebbc4b72e29b12aa2f3242fabcf48cb24a5c8cfcd1704 2024-08-20T22:10:20.3419848Z deleted: sha256:624417752cf64b2297d09ab2e429cc9c87774134cc1b34bf2e5c2b79cd34ecc4 2024-08-20T22:10:20.3420911Z deleted: sha256:2bcfc72dea6a6bc48e2290c98fa34228cfe20302176ff01ce6b84971d811079a 2024-08-20T22:10:20.3421980Z deleted: sha256:d4aee3900a65d7e4f9a427d9bae22e836280b85f289fd5b80adc4ed2cd3aff53 2024-08-20T22:10:20.3423041Z deleted: sha256:2a255798af4e419932e17c11a82527d26470216c9bc57ab8807a735dbd2c93ae 2024-08-20T22:10:20.3424083Z deleted: sha256:fa19e724b62cc0bbe219419c371baf2471725f88a55292672c1aa24358db4a9b 2024-08-20T22:10:20.3425272Z deleted: sha256:9c5a64873ca39cb9da2e93ad3ec1eb31211c8c088c3a90be1541ec5ac891ac4e 2024-08-20T22:10:20.3426359Z deleted: sha256:8421b1b55b2087ef547bb0133e57f5859efb3c9429d011de45772b63c923dd90 2024-08-20T22:10:20.3427421Z deleted: sha256:66066e4af5674b8211099bfebb5a5ab99b3d9181fcfc1dc67b54a0eff5ded07a 2024-08-20T22:10:20.3428498Z deleted: sha256:dad58a2cd6e9c8234f343c2e5ca4cd2a317b729f8048049d34151b15c83b0dae 2024-08-20T22:10:20.3429537Z deleted: sha256:3bc136283d22fe55b48989019c8484e37c075a7a98fae0a5b7f827623cb2fd5b 2024-08-20T22:10:20.3430611Z deleted: sha256:2543cfb072205917a252ea669db6e7fe93ac6bf424262d6d67b6bf24c1edfe55 2024-08-20T22:10:20.3431672Z deleted: sha256:a0abc6809e0930ad53392ef3443966cb3f6bc7645d3e1968f2d18cb0648d1240 2024-08-20T22:10:20.3432739Z deleted: sha256:0068bc6311b72b957fcc8bec156923eef76c142d408a73e201ff797dcf126f5a 2024-08-20T22:10:20.3433814Z deleted: sha256:3bb2c93e3ca7da8a708900f677778ff0abe6bcb9666d9480bdad78acd6793a59 2024-08-20T22:10:20.3434929Z deleted: sha256:d3a2bdf80e0add5b55bffcb4cf9bd55c993adad4fb66a4ec8864e0d8a118bb0a 2024-08-20T22:10:20.3435998Z deleted: sha256:c021994940e99e271b90c5bf290f821f73a14768aa34bfae3755c04685c88251 2024-08-20T22:10:20.3437520Z deleted: sha256:688ff5904c74620c1b2bd34c392425856fe1f759f94bf0419d051997c4fb23ad 2024-08-20T22:10:20.3438641Z deleted: sha256:d397bdc2172f15f11a5f57d5f57064aee1b038d71c3ca9e967ebf47dcbd5dec0 2024-08-20T22:10:20.3439729Z deleted: sha256:a70ac2bc18340667dfc9895cf91a48cc3c2b5a6c2acdc898bae02810b111c7d7 2024-08-20T22:10:20.3440796Z deleted: sha256:0fbec293f1fa0d21f414351f21bd86a114facc2f3bc74715393f4f33c9b2c89e 2024-08-20T22:10:20.3441874Z deleted: sha256:4a26099a629bbf706fb951dcb1381af8361bf5f3e736e82ea839a4a27788d0b3 2024-08-20T22:10:20.3442937Z deleted: sha256:b6284de21252ed91fd7dea71b02196645a8e239d6188ecd9f080d9b4a2026b77 2024-08-20T22:10:20.3443980Z deleted: sha256:adca9f30c5b25b7904016386497a34e3699d12b61be1cabda8703a4f6f51adaf 2024-08-20T22:10:20.3445027Z deleted: sha256:b1007278b864200bed03e3738e6a554a7913b41e9246c86925126b02f19d2512 2024-08-20T22:10:20.3446091Z deleted: sha256:ac9c37335cbc79df52a9de95491fe9ffa89b9fe730c29153ae1d8c93a046bda9 2024-08-20T22:10:20.3447160Z deleted: sha256:d0e212b9d05b634be5d1abfbfcc4fd090277954332f2b8e519c695a979bfd0dd 2024-08-20T22:10:20.3448436Z deleted: sha256:abcbe99d7f9c9df7e3c594587238e23d28f96a5fc92c2a9dd1d183ff7af4800c 2024-08-20T22:10:20.3449501Z deleted: sha256:7740288981d55c53dc1d781eb5a229ccbd5c81c56584c4d4217d7fbb512de04b 2024-08-20T22:10:20.3450555Z deleted: sha256:020ed2c4861bb111c010888e7c4ca3b2c1125b9d7b8a29148de859f786ecea66 2024-08-20T22:10:20.3451582Z deleted: sha256:89edf8c906678c217b9a88160e1561991f53909c3fd27a4a5e1f0ca0f7ff7888 2024-08-20T22:10:20.3452659Z deleted: sha256:d5ba67f8f9b0c2b6da06d2d5d9d5fedc9f17cbe1a5e13a1a4033a4619f4528ba 2024-08-20T22:10:20.3453726Z deleted: sha256:5d61311e1a293c0c9642e34871719525dfeec7028e4b45e08344a469e3fe24bf 2024-08-20T22:10:20.3454764Z deleted: sha256:b89bb70f745a94c0edd5e974aee294e160735f76fd666a1a06a0b9b1be3caf63 2024-08-20T22:10:20.3455845Z deleted: sha256:9e5477f1f21a38c3dfe21a56a7d77d1b1a42453aed3e11729afcbb9d5731770b 2024-08-20T22:10:20.3456968Z deleted: sha256:b76f4af91cffe2bb7b3f09960f5bdbb26f741419defda65c451c4407fc5b0a92 2024-08-20T22:10:20.3458086Z deleted: sha256:3f8748a00ebd394c0e9d9d481972d764dff58896841ef0150b3f7efaac0f4a4a 2024-08-20T22:10:20.3459138Z deleted: sha256:13989c771e3ba4d40f618344e55e5cb7062a35041b5efd69e2f740ba819f46ee 2024-08-20T22:10:20.3460208Z deleted: sha256:9ebb27837f6c67ad8c17e9af4e63c008faae03fa226c3641dd8bbbfeefb106c4 2024-08-20T22:10:20.3461269Z deleted: sha256:8c16907d735950f8093582bc3cf392b70e002456bf9c4224de5a5b86b78b3ea2 2024-08-20T22:10:20.3462297Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-08-20T22:10:20.3462959Z 2024-08-20T22:10:20.3463133Z Total reclaimed space: 34.72GB 2024-08-20T22:10:20.3558754Z Post job cleanup. 2024-08-20T22:10:20.3719330Z Post job cleanup. 2024-08-20T22:10:20.4886845Z [command]/usr/bin/git version 2024-08-20T22:10:20.4946788Z git version 2.40.1 2024-08-20T22:10:20.5007025Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/27a66fec-652d-43ef-9dcf-184c1d586229' before making global git config changes 2024-08-20T22:10:20.5008660Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T22:10:20.5014307Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:20.5060725Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T22:10:20.5099904Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T22:10:20.5450291Z Entering 'android/libs/fbjni' 2024-08-20T22:10:20.5513218Z Entering 'third_party/FP16' 2024-08-20T22:10:20.5576758Z Entering 'third_party/FXdiv' 2024-08-20T22:10:20.5640706Z Entering 'third_party/NNPACK' 2024-08-20T22:10:20.5703862Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:10:20.5767955Z Entering 'third_party/XNNPACK' 2024-08-20T22:10:20.5851086Z Entering 'third_party/benchmark' 2024-08-20T22:10:20.5914767Z Entering 'third_party/cpp-httplib' 2024-08-20T22:10:20.5978334Z Entering 'third_party/cpuinfo' 2024-08-20T22:10:20.6042420Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:10:20.6104678Z Entering 'third_party/cutlass' 2024-08-20T22:10:20.6179041Z Entering 'third_party/eigen' 2024-08-20T22:10:20.6244650Z Entering 'third_party/fbgemm' 2024-08-20T22:10:20.6308103Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:10:20.6370885Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:10:20.6435169Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:10:20.6506808Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:10:20.6569123Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:10:20.6632838Z Entering 'third_party/flatbuffers' 2024-08-20T22:10:20.6699783Z Entering 'third_party/fmt' 2024-08-20T22:10:20.6763517Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:10:20.6826255Z Entering 'third_party/gloo' 2024-08-20T22:10:20.6890328Z Entering 'third_party/googletest' 2024-08-20T22:10:20.6953889Z Entering 'third_party/ideep' 2024-08-20T22:10:20.7014489Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:10:20.7084302Z Entering 'third_party/ittapi' 2024-08-20T22:10:20.7147242Z Entering 'third_party/kineto' 2024-08-20T22:10:20.7209553Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:10:20.7273457Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:10:20.7337821Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:10:20.7400691Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:10:20.7465424Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:10:20.7526004Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:10:20.7592103Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:10:20.7655494Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:10:20.7717895Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:10:20.7782381Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:10:20.7846398Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:10:20.7909347Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:10:20.7974058Z Entering 'third_party/mimalloc' 2024-08-20T22:10:20.8035945Z Entering 'third_party/nccl/nccl' 2024-08-20T22:10:20.8099709Z Entering 'third_party/nlohmann' 2024-08-20T22:10:20.8163806Z Entering 'third_party/onnx' 2024-08-20T22:10:20.8244891Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:10:20.8307342Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:10:20.8373624Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:10:20.8437586Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:10:20.8499036Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:10:20.8561177Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:10:20.8622409Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:10:20.8686806Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:10:20.8748259Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:10:20.8809902Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:10:20.8871632Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:10:20.8935402Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:10:20.8998702Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:10:20.9083777Z Entering 'third_party/pocketfft' 2024-08-20T22:10:20.9146048Z Entering 'third_party/protobuf' 2024-08-20T22:10:20.9212697Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:10:20.9277645Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:10:20.9341640Z Entering 'third_party/psimd' 2024-08-20T22:10:20.9404213Z Entering 'third_party/pthreadpool' 2024-08-20T22:10:20.9466682Z Entering 'third_party/pybind11' 2024-08-20T22:10:20.9529010Z Entering 'third_party/python-peachpy' 2024-08-20T22:10:20.9592085Z Entering 'third_party/sleef' 2024-08-20T22:10:20.9654978Z Entering 'third_party/tensorpipe' 2024-08-20T22:10:20.9716798Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:10:20.9779133Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:10:20.9841969Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:10:20.9904229Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:10:20.9965636Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:10:21.0050366Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T22:10:21.0080280Z http.https://github.com/.extraheader 2024-08-20T22:10:21.0091989Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-20T22:10:21.0133331Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T22:10:21.0474111Z Entering 'android/libs/fbjni' 2024-08-20T22:10:21.0516256Z http.https://github.com/.extraheader 2024-08-20T22:10:21.0555199Z Entering 'third_party/FP16' 2024-08-20T22:10:21.0597895Z http.https://github.com/.extraheader 2024-08-20T22:10:21.0636638Z Entering 'third_party/FXdiv' 2024-08-20T22:10:21.0680158Z http.https://github.com/.extraheader 2024-08-20T22:10:21.0717700Z Entering 'third_party/NNPACK' 2024-08-20T22:10:21.0761069Z http.https://github.com/.extraheader 2024-08-20T22:10:21.0799319Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:10:21.0842495Z http.https://github.com/.extraheader 2024-08-20T22:10:21.0880625Z Entering 'third_party/XNNPACK' 2024-08-20T22:10:21.0923628Z http.https://github.com/.extraheader 2024-08-20T22:10:21.0981440Z Entering 'third_party/benchmark' 2024-08-20T22:10:21.1023617Z http.https://github.com/.extraheader 2024-08-20T22:10:21.1061923Z Entering 'third_party/cpp-httplib' 2024-08-20T22:10:21.1103824Z http.https://github.com/.extraheader 2024-08-20T22:10:21.1145339Z Entering 'third_party/cpuinfo' 2024-08-20T22:10:21.1187534Z http.https://github.com/.extraheader 2024-08-20T22:10:21.1226079Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:10:21.1270325Z http.https://github.com/.extraheader 2024-08-20T22:10:21.1308367Z Entering 'third_party/cutlass' 2024-08-20T22:10:21.1350764Z http.https://github.com/.extraheader 2024-08-20T22:10:21.1398969Z Entering 'third_party/eigen' 2024-08-20T22:10:21.1443014Z http.https://github.com/.extraheader 2024-08-20T22:10:21.1483535Z Entering 'third_party/fbgemm' 2024-08-20T22:10:21.1525950Z http.https://github.com/.extraheader 2024-08-20T22:10:21.1565376Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:10:21.1607287Z http.https://github.com/.extraheader 2024-08-20T22:10:21.1646772Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:10:21.1688172Z http.https://github.com/.extraheader 2024-08-20T22:10:21.1726685Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:10:21.1769240Z http.https://github.com/.extraheader 2024-08-20T22:10:21.1814861Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:10:21.1857379Z http.https://github.com/.extraheader 2024-08-20T22:10:21.1895279Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:10:21.1937241Z http.https://github.com/.extraheader 2024-08-20T22:10:21.1976845Z Entering 'third_party/flatbuffers' 2024-08-20T22:10:21.2020039Z http.https://github.com/.extraheader 2024-08-20T22:10:21.2061696Z Entering 'third_party/fmt' 2024-08-20T22:10:21.2103527Z http.https://github.com/.extraheader 2024-08-20T22:10:21.2142460Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:10:21.2184674Z http.https://github.com/.extraheader 2024-08-20T22:10:21.2222658Z Entering 'third_party/gloo' 2024-08-20T22:10:21.2265470Z http.https://github.com/.extraheader 2024-08-20T22:10:21.2305260Z Entering 'third_party/googletest' 2024-08-20T22:10:21.2348060Z http.https://github.com/.extraheader 2024-08-20T22:10:21.2386430Z Entering 'third_party/ideep' 2024-08-20T22:10:21.2429254Z http.https://github.com/.extraheader 2024-08-20T22:10:21.2466346Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:10:21.2507381Z http.https://github.com/.extraheader 2024-08-20T22:10:21.2555160Z Entering 'third_party/ittapi' 2024-08-20T22:10:21.2597226Z http.https://github.com/.extraheader 2024-08-20T22:10:21.2634744Z Entering 'third_party/kineto' 2024-08-20T22:10:21.2678155Z http.https://github.com/.extraheader 2024-08-20T22:10:21.2715917Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:10:21.2759982Z http.https://github.com/.extraheader 2024-08-20T22:10:21.2798223Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:10:21.2841607Z http.https://github.com/.extraheader 2024-08-20T22:10:21.2882023Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:10:21.2923763Z http.https://github.com/.extraheader 2024-08-20T22:10:21.2964932Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:10:21.3007191Z http.https://github.com/.extraheader 2024-08-20T22:10:21.3047231Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:10:21.3089678Z http.https://github.com/.extraheader 2024-08-20T22:10:21.3127761Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:10:21.3172604Z http.https://github.com/.extraheader 2024-08-20T22:10:21.3213386Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:10:21.3255880Z http.https://github.com/.extraheader 2024-08-20T22:10:21.3294969Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:10:21.3336631Z http.https://github.com/.extraheader 2024-08-20T22:10:21.3377056Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:10:21.3418532Z http.https://github.com/.extraheader 2024-08-20T22:10:21.3459421Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:10:21.3501626Z http.https://github.com/.extraheader 2024-08-20T22:10:21.3543168Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:10:21.3585148Z http.https://github.com/.extraheader 2024-08-20T22:10:21.3622646Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:10:21.3665239Z http.https://github.com/.extraheader 2024-08-20T22:10:21.3704830Z Entering 'third_party/mimalloc' 2024-08-20T22:10:21.3747802Z http.https://github.com/.extraheader 2024-08-20T22:10:21.3786195Z Entering 'third_party/nccl/nccl' 2024-08-20T22:10:21.3829090Z http.https://github.com/.extraheader 2024-08-20T22:10:21.3868942Z Entering 'third_party/nlohmann' 2024-08-20T22:10:21.3910883Z http.https://github.com/.extraheader 2024-08-20T22:10:21.3950762Z Entering 'third_party/onnx' 2024-08-20T22:10:21.3993412Z http.https://github.com/.extraheader 2024-08-20T22:10:21.4049193Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:10:21.4090944Z http.https://github.com/.extraheader 2024-08-20T22:10:21.4128786Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:10:21.4172046Z http.https://github.com/.extraheader 2024-08-20T22:10:21.4212219Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:10:21.4255375Z http.https://github.com/.extraheader 2024-08-20T22:10:21.4295376Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:10:21.4336962Z http.https://github.com/.extraheader 2024-08-20T22:10:21.4375560Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:10:21.4417550Z http.https://github.com/.extraheader 2024-08-20T22:10:21.4455913Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:10:21.4497213Z http.https://github.com/.extraheader 2024-08-20T22:10:21.4535159Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:10:21.4578084Z http.https://github.com/.extraheader 2024-08-20T22:10:21.4617051Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:10:21.4660242Z http.https://github.com/.extraheader 2024-08-20T22:10:21.4697950Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:10:21.4741107Z http.https://github.com/.extraheader 2024-08-20T22:10:21.4778961Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:10:21.4819931Z http.https://github.com/.extraheader 2024-08-20T22:10:21.4859194Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:10:21.4900447Z http.https://github.com/.extraheader 2024-08-20T22:10:21.4941333Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:10:21.4983280Z http.https://github.com/.extraheader 2024-08-20T22:10:21.5023198Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:10:21.5065808Z http.https://github.com/.extraheader 2024-08-20T22:10:21.5127226Z Entering 'third_party/pocketfft' 2024-08-20T22:10:21.5172076Z http.https://github.com/.extraheader 2024-08-20T22:10:21.5209706Z Entering 'third_party/protobuf' 2024-08-20T22:10:21.5254023Z http.https://github.com/.extraheader 2024-08-20T22:10:21.5295602Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:10:21.5337397Z http.https://github.com/.extraheader 2024-08-20T22:10:21.5376101Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:10:21.5417876Z http.https://github.com/.extraheader 2024-08-20T22:10:21.5457946Z Entering 'third_party/psimd' 2024-08-20T22:10:21.5499605Z http.https://github.com/.extraheader 2024-08-20T22:10:21.5537356Z Entering 'third_party/pthreadpool' 2024-08-20T22:10:21.5579846Z http.https://github.com/.extraheader 2024-08-20T22:10:21.5617924Z Entering 'third_party/pybind11' 2024-08-20T22:10:21.5661528Z http.https://github.com/.extraheader 2024-08-20T22:10:21.5699349Z Entering 'third_party/python-peachpy' 2024-08-20T22:10:21.5742597Z http.https://github.com/.extraheader 2024-08-20T22:10:21.5779794Z Entering 'third_party/sleef' 2024-08-20T22:10:21.5821735Z http.https://github.com/.extraheader 2024-08-20T22:10:21.5861289Z Entering 'third_party/tensorpipe' 2024-08-20T22:10:21.5903131Z http.https://github.com/.extraheader 2024-08-20T22:10:21.5941882Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:10:21.5983882Z http.https://github.com/.extraheader 2024-08-20T22:10:21.6021380Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:10:21.6064318Z http.https://github.com/.extraheader 2024-08-20T22:10:21.6101805Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:10:21.6143315Z http.https://github.com/.extraheader 2024-08-20T22:10:21.6181264Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:10:21.6222951Z http.https://github.com/.extraheader 2024-08-20T22:10:21.6260421Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:10:21.6302540Z http.https://github.com/.extraheader 2024-08-20T22:10:21.6446265Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-20T22:10:21.6473174Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-20T22:10:21.6479851Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:21.6480508Z ##[endgroup] 2024-08-20T22:10:29.7751632Z Cleaning up orphan processes