2024-08-06T20:30:58.9627535Z Current runner version: '2.318.0' 2024-08-06T20:30:58.9637565Z Runner name: 'i-06f045a861c998624' 2024-08-06T20:30:58.9639017Z Runner group name: 'Default' 2024-08-06T20:30:58.9640480Z Machine name: 'ip-10-0-56-92' 2024-08-06T20:30:58.9647148Z ##[group]GITHUB_TOKEN Permissions 2024-08-06T20:30:58.9650546Z Actions: read 2024-08-06T20:30:58.9651690Z Attestations: read 2024-08-06T20:30:58.9652656Z Checks: read 2024-08-06T20:30:58.9653618Z Contents: read 2024-08-06T20:30:58.9654649Z Deployments: read 2024-08-06T20:30:58.9655597Z Discussions: read 2024-08-06T20:30:58.9656548Z Issues: read 2024-08-06T20:30:58.9657571Z Metadata: read 2024-08-06T20:30:58.9659094Z Packages: read 2024-08-06T20:30:58.9660090Z Pages: read 2024-08-06T20:30:58.9661127Z PullRequests: read 2024-08-06T20:30:58.9662091Z RepositoryProjects: read 2024-08-06T20:30:58.9663133Z SecurityEvents: read 2024-08-06T20:30:58.9664213Z Statuses: read 2024-08-06T20:30:58.9665212Z ##[endgroup] 2024-08-06T20:30:58.9670151Z Secret source: Actions 2024-08-06T20:30:58.9671556Z Prepare workflow directory 2024-08-06T20:30:59.3737638Z Prepare all required actions 2024-08-06T20:30:59.3799054Z Getting action download info 2024-08-06T20:30:59.5965785Z Download action repository 'pytorch/test-infra@main' (SHA:a1f5a89251fc4258ab59806094fe3108f7d6741a) 2024-08-06T20:31:00.0120488Z Download action repository 'pytorch/pytorch@main' (SHA:de00c7958301ce81b9716bdef5731ed40d4d14ca) 2024-08-06T20:31:03.6046368Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-06T20:31:03.7799567Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-06T20:31:04.0066036Z Getting action download info 2024-08-06T20:31:04.1345016Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-06T20:31:04.4449103Z Getting action download info 2024-08-06T20:31:04.5790012Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-06T20:31:04.7320405Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/132710/merge (bf5bb5a1585a03379137fab341e87c02c77e76cd) 2024-08-06T20:31:04.7323406Z ##[group] Inputs 2024-08-06T20:31:04.7323937Z build-environment: linux-focal-cuda11.8-py3.10-gcc9 2024-08-06T20:31:04.7325740Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]} 2024-08-06T20:31:04.7327824Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-06T20:31:04.7328942Z sync-tag: 2024-08-06T20:31:04.7330341Z timeout-minutes: 360 2024-08-06T20:31:04.7330802Z use-gha: 2024-08-06T20:31:04.7331298Z dashboard-tag: 2024-08-06T20:31:04.7331768Z s3-bucket: gha-artifacts 2024-08-06T20:31:04.7332199Z aws-role-to-assume: 2024-08-06T20:31:04.7332692Z ##[endgroup] 2024-08-06T20:31:04.7333479Z Complete job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-08-06T20:31:04.8106829Z A job started hook has been configured by the self-hosted runner administrator 2024-08-06T20:31:04.8262551Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-06T20:31:04.8274647Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:31:04.8275658Z ##[endgroup] 2024-08-06T20:31:06.4305185Z Runner Type: linux.8xlarge.nvidia.gpu 2024-08-06T20:31:06.4305861Z Instance Type: g3.8xlarge 2024-08-06T20:31:06.4306277Z AMI Name: amzn2-ami-hvm-2.0.20240306.2-x86_64-ebs 2024-08-06T20:31:06.4306753Z AMI ID: ami-0ce0c36d7a00b20e2 2024-08-06T20:31:11.5186556Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-06T20:31:11.5187381Z with: 2024-08-06T20:31:11.5188301Z github-secret: *** 2024-08-06T20:31:11.5189225Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-06T20:31:11.5190152Z activate-with-label: false 2024-08-06T20:31:11.5190527Z label: with-ssh 2024-08-06T20:31:11.5190860Z remove-existing-keys: true 2024-08-06T20:31:11.5191211Z fail-silently: true 2024-08-06T20:31:11.5191572Z env: 2024-08-06T20:31:11.5191862Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:31:11.5192199Z ##[endgroup] 2024-08-06T20:31:11.6306892Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-06T20:31:11.9318414Z Grabbing public ssh keys from https://github.com/drisspg.keys 2024-08-06T20:31:12.0092356Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-06T20:31:12.0114604Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-06T20:31:12.0155673Z Login using: ssh ec2-user@ec2-54-90-212-63.compute-1.amazonaws.com 2024-08-06T20:31:12.0156402Z All testing is done inside the container, to start an interactive session run: 2024-08-06T20:31:12.0157088Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-06T20:31:12.0292563Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-06T20:31:12.0293165Z with: 2024-08-06T20:31:12.0293456Z submodules: recursive 2024-08-06T20:31:12.0293820Z fetch-depth: 0 2024-08-06T20:31:12.0294133Z env: 2024-08-06T20:31:12.0294410Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:31:12.0294764Z ##[endgroup] 2024-08-06T20:31:12.0401113Z ##[group]Run retry () { 2024-08-06T20:31:12.0401514Z retry () { 2024-08-06T20:31:12.0401949Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-06T20:31:12.0402447Z } 2024-08-06T20:31:12.0402769Z echo "${GITHUB_WORKSPACE}" 2024-08-06T20:31:12.0403188Z if [ -z "${NO_SUDO}" ]; then 2024-08-06T20:31:12.0403635Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-06T20:31:12.0404067Z else 2024-08-06T20:31:12.0404394Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-06T20:31:12.0404810Z fi 2024-08-06T20:31:12.0405189Z mkdir "${GITHUB_WORKSPACE}" 2024-08-06T20:31:12.0413488Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:31:12.0413992Z env: 2024-08-06T20:31:12.0414274Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:31:12.0414627Z NO_SUDO: 2024-08-06T20:31:12.0414918Z ##[endgroup] 2024-08-06T20:31:12.0441371Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-06T20:31:15.3397978Z ##[group]Run malfet/checkout@silent-checkout 2024-08-06T20:31:15.3398470Z with: 2024-08-06T20:31:15.3398795Z ref: b9d86fa89636e301796d4201f36d86c73f6e49bc 2024-08-06T20:31:15.3399232Z fetch-depth: 0 2024-08-06T20:31:15.3399562Z submodules: recursive 2024-08-06T20:31:15.3399917Z quiet-checkout: true 2024-08-06T20:31:15.3400281Z repository: pytorch/pytorch 2024-08-06T20:31:15.3400794Z token: *** 2024-08-06T20:31:15.3401100Z ssh-strict: true 2024-08-06T20:31:15.3401420Z persist-credentials: true 2024-08-06T20:31:15.3401785Z clean: true 2024-08-06T20:31:15.3402112Z sparse-checkout-cone-mode: true 2024-08-06T20:31:15.3402500Z lfs: false 2024-08-06T20:31:15.3402811Z set-safe-directory: true 2024-08-06T20:31:15.3403142Z env: 2024-08-06T20:31:15.3403432Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:31:15.3403780Z ##[endgroup] 2024-08-06T20:31:15.4692190Z Syncing repository: pytorch/pytorch 2024-08-06T20:31:15.4694047Z ##[group]Getting Git version info 2024-08-06T20:31:15.4694686Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-06T20:31:15.4695552Z [command]/usr/bin/git version 2024-08-06T20:31:15.4698428Z git version 2.40.1 2024-08-06T20:31:15.4733468Z ##[endgroup] 2024-08-06T20:31:15.4756362Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/4e35dc62-50ba-466c-9b1a-76b6e47ba09c' before making global git config changes 2024-08-06T20:31:15.4757919Z Adding repository directory to the temporary git global config as a safe directory 2024-08-06T20:31:15.4763130Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-06T20:31:15.4806705Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-06T20:31:15.4812888Z ##[group]Initializing the repository 2024-08-06T20:31:15.4816831Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-06T20:31:15.4848001Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-06T20:31:15.4848780Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-06T20:31:15.4849498Z hint: of your new repositories, which will suppress this warning, call: 2024-08-06T20:31:15.4850036Z hint: 2024-08-06T20:31:15.4850438Z hint: git config --global init.defaultBranch 2024-08-06T20:31:15.4850913Z hint: 2024-08-06T20:31:15.4851359Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-06T20:31:15.4852063Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-06T20:31:15.4852847Z hint: 2024-08-06T20:31:15.4853190Z hint: git branch -m 2024-08-06T20:31:15.4853830Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-06T20:31:15.4861523Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-06T20:31:15.4893588Z ##[endgroup] 2024-08-06T20:31:15.4894159Z ##[group]Disabling automatic garbage collection 2024-08-06T20:31:15.4897986Z [command]/usr/bin/git config --local gc.auto 0 2024-08-06T20:31:15.4930809Z ##[endgroup] 2024-08-06T20:31:15.4931323Z ##[group]Setting up auth 2024-08-06T20:31:15.4941278Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-06T20:31:15.4974052Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-06T20:31:15.5263978Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-06T20:31:15.5295300Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-06T20:31:15.5584832Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-06T20:31:15.5633988Z ##[endgroup] 2024-08-06T20:31:15.5634714Z ##[group]Fetching the repository 2024-08-06T20:31:15.5642116Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-06T20:31:17.7834200Z remote: Enumerating objects: 1008383 2024-08-06T20:31:17.7834883Z remote: Enumerating objects: 1008693, done. 2024-08-06T20:31:17.7836345Z remote: Counting objects: 0% (1/310) 2024-08-06T20:31:17.7837328Z remote: Counting objects: 1% (4/310) 2024-08-06T20:31:17.7838509Z remote: Counting objects: 2% (7/310) 2024-08-06T20:31:17.7839653Z remote: Counting objects: 3% (10/310) 2024-08-06T20:31:17.7840551Z remote: Counting objects: 4% (13/310) 2024-08-06T20:31:17.7841108Z remote: Counting objects: 5% (16/310) 2024-08-06T20:31:17.7841600Z remote: Counting objects: 6% (19/310) 2024-08-06T20:31:17.7842073Z remote: Counting objects: 7% (22/310) 2024-08-06T20:31:17.7842646Z remote: Counting objects: 8% (25/310) 2024-08-06T20:31:17.7843140Z remote: Counting objects: 9% (28/310) 2024-08-06T20:31:17.7843616Z remote: Counting objects: 10% (31/310) 2024-08-06T20:31:17.7844624Z remote: Counting objects: 11% (35/310) 2024-08-06T20:31:17.7845103Z remote: Counting objects: 12% (38/310) 2024-08-06T20:31:17.7845675Z remote: Counting objects: 13% (41/310) 2024-08-06T20:31:17.7846150Z remote: Counting objects: 14% (44/310) 2024-08-06T20:31:17.7846638Z remote: Counting objects: 15% (47/310) 2024-08-06T20:31:17.7847183Z remote: Counting objects: 16% (50/310) 2024-08-06T20:31:17.7847830Z remote: Counting objects: 17% (53/310) 2024-08-06T20:31:17.7848289Z remote: Counting objects: 18% (56/310) 2024-08-06T20:31:17.7848857Z remote: Counting objects: 19% (59/310) 2024-08-06T20:31:17.7849329Z remote: Counting objects: 20% (62/310) 2024-08-06T20:31:17.7849788Z remote: Counting objects: 21% (66/310) 2024-08-06T20:31:17.7850328Z remote: Counting objects: 22% (69/310) 2024-08-06T20:31:17.7850814Z remote: Counting objects: 23% (72/310) 2024-08-06T20:31:17.7851269Z remote: Counting objects: 24% (75/310) 2024-08-06T20:31:17.7851843Z remote: Counting objects: 25% (78/310) 2024-08-06T20:31:17.7852330Z remote: Counting objects: 26% (81/310) 2024-08-06T20:31:17.7852876Z remote: Counting objects: 27% (84/310) 2024-08-06T20:31:17.7853655Z remote: Counting objects: 28% (87/310) 2024-08-06T20:31:17.7854180Z remote: Counting objects: 29% (90/310) 2024-08-06T20:31:17.7854635Z remote: Counting objects: 30% (93/310) 2024-08-06T20:31:17.7855213Z remote: Counting objects: 31% (97/310) 2024-08-06T20:31:17.7855689Z remote: Counting objects: 32% (100/310) 2024-08-06T20:31:17.7856154Z remote: Counting objects: 33% (103/310) 2024-08-06T20:31:17.7856765Z remote: Counting objects: 34% (106/310) 2024-08-06T20:31:17.7857249Z remote: Counting objects: 35% (109/310) 2024-08-06T20:31:17.7857711Z remote: Counting objects: 36% (112/310) 2024-08-06T20:31:17.7858974Z remote: Counting objects: 37% (115/310) 2024-08-06T20:31:17.7859602Z remote: Counting objects: 38% (118/310) 2024-08-06T20:31:17.7860086Z remote: Counting objects: 39% (121/310) 2024-08-06T20:31:17.7860562Z remote: Counting objects: 40% (124/310) 2024-08-06T20:31:17.7861143Z remote: Counting objects: 41% (128/310) 2024-08-06T20:31:17.7861612Z remote: Counting objects: 42% (131/310) 2024-08-06T20:31:17.7862087Z remote: Counting objects: 43% (134/310) 2024-08-06T20:31:17.7862689Z remote: Counting objects: 44% (137/310) 2024-08-06T20:31:17.7863165Z remote: Counting objects: 45% (140/310) 2024-08-06T20:31:17.7863645Z remote: Counting objects: 46% (143/310) 2024-08-06T20:31:17.7864230Z remote: Counting objects: 47% (146/310) 2024-08-06T20:31:17.7864699Z remote: Counting objects: 48% (149/310) 2024-08-06T20:31:17.7865185Z remote: Counting objects: 49% (152/310) 2024-08-06T20:31:17.7865791Z remote: Counting objects: 50% (155/310) 2024-08-06T20:31:17.7866284Z remote: Counting objects: 51% (159/310) 2024-08-06T20:31:17.7866761Z remote: Counting objects: 52% (162/310) 2024-08-06T20:31:17.7867384Z remote: Counting objects: 53% (165/310) 2024-08-06T20:31:17.7867947Z remote: Counting objects: 54% (168/310) 2024-08-06T20:31:17.7868439Z remote: Counting objects: 55% (171/310) 2024-08-06T20:31:17.7868920Z remote: Counting objects: 56% (174/310) 2024-08-06T20:31:17.7869501Z remote: Counting objects: 57% (177/310) 2024-08-06T20:31:17.7869982Z remote: Counting objects: 58% (180/310) 2024-08-06T20:31:17.7870452Z remote: Counting objects: 59% (183/310) 2024-08-06T20:31:17.7870990Z remote: Counting objects: 60% (186/310) 2024-08-06T20:31:17.7871472Z remote: Counting objects: 61% (190/310) 2024-08-06T20:31:17.7871944Z remote: Counting objects: 62% (193/310) 2024-08-06T20:31:17.7872558Z remote: Counting objects: 63% (196/310) 2024-08-06T20:31:17.7873240Z remote: Counting objects: 64% (199/310) 2024-08-06T20:31:17.7873718Z remote: Counting objects: 65% (202/310) 2024-08-06T20:31:17.7874333Z remote: Counting objects: 66% (205/310) 2024-08-06T20:31:17.7874790Z remote: Counting objects: 67% (208/310) 2024-08-06T20:31:17.7875278Z remote: Counting objects: 68% (211/310) 2024-08-06T20:31:17.7875890Z remote: Counting objects: 69% (214/310) 2024-08-06T20:31:17.7876345Z remote: Counting objects: 70% (217/310) 2024-08-06T20:31:17.7876894Z remote: Counting objects: 71% (221/310) 2024-08-06T20:31:17.7877409Z remote: Counting objects: 72% (224/310) 2024-08-06T20:31:17.7877863Z remote: Counting objects: 73% (227/310) 2024-08-06T20:31:17.7878362Z remote: Counting objects: 74% (230/310) 2024-08-06T20:31:17.7878970Z remote: Counting objects: 75% (233/310) 2024-08-06T20:31:17.7879428Z remote: Counting objects: 76% (236/310) 2024-08-06T20:31:17.7880039Z remote: Counting objects: 77% (239/310) 2024-08-06T20:31:17.7880541Z remote: Counting objects: 78% (242/310) 2024-08-06T20:31:17.7880996Z remote: Counting objects: 79% (245/310) 2024-08-06T20:31:17.7881468Z remote: Counting objects: 80% (248/310) 2024-08-06T20:31:17.7882239Z remote: Counting objects: 81% (252/310) 2024-08-06T20:31:17.7882726Z remote: Counting objects: 82% (255/310) 2024-08-06T20:31:17.7883349Z remote: Counting objects: 83% (258/310) 2024-08-06T20:31:17.7883828Z remote: Counting objects: 84% (261/310) 2024-08-06T20:31:17.7884283Z remote: Counting objects: 85% (264/310) 2024-08-06T20:31:17.7884898Z remote: Counting objects: 86% (267/310) 2024-08-06T20:31:17.7885380Z remote: Counting objects: 87% (270/310) 2024-08-06T20:31:17.7885837Z remote: Counting objects: 88% (273/310) 2024-08-06T20:31:17.7886429Z remote: Counting objects: 89% (276/310) 2024-08-06T20:31:17.7887011Z remote: Counting objects: 90% (279/310) 2024-08-06T20:31:17.7887471Z remote: Counting objects: 91% (283/310) 2024-08-06T20:31:17.7888089Z remote: Counting objects: 92% (286/310) 2024-08-06T20:31:17.7888571Z remote: Counting objects: 93% (289/310) 2024-08-06T20:31:17.7889118Z remote: Counting objects: 94% (292/310) 2024-08-06T20:31:17.7890211Z remote: Counting objects: 95% (295/310) 2024-08-06T20:31:17.7891193Z remote: Counting objects: 96% (298/310) 2024-08-06T20:31:17.7891665Z remote: Counting objects: 97% (301/310) 2024-08-06T20:31:17.7892159Z remote: Counting objects: 98% (304/310) 2024-08-06T20:31:17.7892788Z remote: Counting objects: 99% (307/310) 2024-08-06T20:31:17.7893250Z remote: Counting objects: 100% (310/310) 2024-08-06T20:31:17.7893758Z remote: Counting objects: 100% (310/310), done. 2024-08-06T20:31:17.7894420Z remote: Compressing objects: 0% (1/177) 2024-08-06T20:31:17.7894949Z remote: Compressing objects: 1% (2/177) 2024-08-06T20:31:17.7912018Z remote: Compressing objects: 2% (4/177) 2024-08-06T20:31:17.7935440Z remote: Compressing objects: 3% (6/177) 2024-08-06T20:31:17.8001660Z remote: Compressing objects: 4% (8/177) 2024-08-06T20:31:17.8096351Z remote: Compressing objects: 5% (9/177) 2024-08-06T20:31:17.8150309Z remote: Compressing objects: 6% (11/177) 2024-08-06T20:31:17.8171939Z remote: Compressing objects: 7% (13/177) 2024-08-06T20:31:17.8196512Z remote: Compressing objects: 8% (15/177) 2024-08-06T20:31:17.8228204Z remote: Compressing objects: 9% (16/177) 2024-08-06T20:31:17.8347018Z remote: Compressing objects: 10% (18/177) 2024-08-06T20:31:17.8348176Z remote: Compressing objects: 11% (20/177) 2024-08-06T20:31:17.8349296Z remote: Compressing objects: 12% (22/177) 2024-08-06T20:31:17.8350444Z remote: Compressing objects: 13% (24/177) 2024-08-06T20:31:17.8351504Z remote: Compressing objects: 14% (25/177) 2024-08-06T20:31:17.8352688Z remote: Compressing objects: 15% (27/177) 2024-08-06T20:31:17.8353387Z remote: Compressing objects: 16% (29/177) 2024-08-06T20:31:17.8354197Z remote: Compressing objects: 17% (31/177) 2024-08-06T20:31:17.8355209Z remote: Compressing objects: 18% (32/177) 2024-08-06T20:31:17.8355909Z remote: Compressing objects: 19% (34/177) 2024-08-06T20:31:17.8356494Z remote: Compressing objects: 20% (36/177) 2024-08-06T20:31:17.8356966Z remote: Compressing objects: 21% (38/177) 2024-08-06T20:31:17.8357987Z remote: Compressing objects: 22% (39/177) 2024-08-06T20:31:17.8358871Z remote: Compressing objects: 23% (41/177) 2024-08-06T20:31:17.8359443Z remote: Compressing objects: 24% (43/177) 2024-08-06T20:31:17.8359953Z remote: Compressing objects: 25% (45/177) 2024-08-06T20:31:17.8360440Z remote: Compressing objects: 26% (47/177) 2024-08-06T20:31:17.8361129Z remote: Compressing objects: 27% (48/177) 2024-08-06T20:31:17.8362411Z remote: Compressing objects: 28% (50/177) 2024-08-06T20:31:17.8363017Z remote: Compressing objects: 29% (52/177) 2024-08-06T20:31:17.8363489Z remote: Compressing objects: 30% (54/177) 2024-08-06T20:31:17.8364152Z remote: Compressing objects: 31% (55/177) 2024-08-06T20:31:17.8364944Z remote: Compressing objects: 32% (57/177) 2024-08-06T20:31:17.8366247Z remote: Compressing objects: 33% (59/177) 2024-08-06T20:31:17.8366831Z remote: Compressing objects: 34% (61/177) 2024-08-06T20:31:17.8367363Z remote: Compressing objects: 35% (62/177) 2024-08-06T20:31:17.8368237Z remote: Compressing objects: 36% (64/177) 2024-08-06T20:31:17.8369450Z remote: Compressing objects: 37% (66/177) 2024-08-06T20:31:17.8371197Z remote: Compressing objects: 38% (68/177) 2024-08-06T20:31:17.8372430Z remote: Compressing objects: 39% (70/177) 2024-08-06T20:31:17.8373766Z remote: Compressing objects: 40% (71/177) 2024-08-06T20:31:17.8375378Z remote: Compressing objects: 41% (73/177) 2024-08-06T20:31:17.8376527Z remote: Compressing objects: 42% (75/177) 2024-08-06T20:31:17.8377711Z remote: Compressing objects: 43% (77/177) 2024-08-06T20:31:17.8378722Z remote: Compressing objects: 44% (78/177) 2024-08-06T20:31:17.8380101Z remote: Compressing objects: 45% (80/177) 2024-08-06T20:31:17.8381416Z remote: Compressing objects: 46% (82/177) 2024-08-06T20:31:17.8382542Z remote: Compressing objects: 47% (84/177) 2024-08-06T20:31:17.8383644Z remote: Compressing objects: 48% (85/177) 2024-08-06T20:31:17.8384180Z remote: Compressing objects: 49% (87/177) 2024-08-06T20:31:17.8384998Z remote: Compressing objects: 50% (89/177) 2024-08-06T20:31:17.8386033Z remote: Compressing objects: 51% (91/177) 2024-08-06T20:31:17.8387205Z remote: Compressing objects: 52% (93/177) 2024-08-06T20:31:17.8388096Z remote: Compressing objects: 53% (94/177) 2024-08-06T20:31:17.8389162Z remote: Compressing objects: 54% (96/177) 2024-08-06T20:31:17.8390344Z remote: Compressing objects: 55% (98/177) 2024-08-06T20:31:17.8391372Z remote: Compressing objects: 56% (100/177) 2024-08-06T20:31:17.8392336Z remote: Compressing objects: 57% (101/177) 2024-08-06T20:31:17.8393477Z remote: Compressing objects: 58% (103/177) 2024-08-06T20:31:17.8394600Z remote: Compressing objects: 59% (105/177) 2024-08-06T20:31:17.8395662Z remote: Compressing objects: 60% (107/177) 2024-08-06T20:31:17.8396553Z remote: Compressing objects: 61% (108/177) 2024-08-06T20:31:17.8397185Z remote: Compressing objects: 62% (110/177) 2024-08-06T20:31:17.8397723Z remote: Compressing objects: 63% (112/177) 2024-08-06T20:31:17.8398198Z remote: Compressing objects: 64% (114/177) 2024-08-06T20:31:17.8399044Z remote: Compressing objects: 65% (116/177) 2024-08-06T20:31:17.8399575Z remote: Compressing objects: 66% (117/177) 2024-08-06T20:31:17.8400208Z remote: Compressing objects: 67% (119/177) 2024-08-06T20:31:17.8400701Z remote: Compressing objects: 68% (121/177) 2024-08-06T20:31:17.8401200Z remote: Compressing objects: 69% (123/177) 2024-08-06T20:31:17.8401674Z remote: Compressing objects: 70% (124/177) 2024-08-06T20:31:17.8402259Z remote: Compressing objects: 71% (126/177) 2024-08-06T20:31:17.8402761Z remote: Compressing objects: 72% (128/177) 2024-08-06T20:31:17.8403350Z remote: Compressing objects: 73% (130/177) 2024-08-06T20:31:17.8403841Z remote: Compressing objects: 74% (131/177) 2024-08-06T20:31:17.8404335Z remote: Compressing objects: 75% (133/177) 2024-08-06T20:31:17.8404812Z remote: Compressing objects: 76% (135/177) 2024-08-06T20:31:17.8405451Z remote: Compressing objects: 77% (137/177) 2024-08-06T20:31:17.8406052Z remote: Compressing objects: 78% (139/177) 2024-08-06T20:31:17.8406542Z remote: Compressing objects: 79% (140/177) 2024-08-06T20:31:17.8407037Z remote: Compressing objects: 80% (142/177) 2024-08-06T20:31:17.8407677Z remote: Compressing objects: 81% (144/177) 2024-08-06T20:31:17.8408287Z remote: Compressing objects: 82% (146/177) 2024-08-06T20:31:17.8408816Z remote: Compressing objects: 83% (147/177) 2024-08-06T20:31:17.8409429Z remote: Compressing objects: 84% (149/177) 2024-08-06T20:31:17.8409906Z remote: Compressing objects: 85% (151/177) 2024-08-06T20:31:17.8410401Z remote: Compressing objects: 86% (153/177) 2024-08-06T20:31:17.8410891Z remote: Compressing objects: 87% (154/177) 2024-08-06T20:31:17.8411506Z remote: Compressing objects: 88% (156/177) 2024-08-06T20:31:17.8412182Z remote: Compressing objects: 89% (158/177) 2024-08-06T20:31:17.8412680Z remote: Compressing objects: 90% (160/177) 2024-08-06T20:31:17.8413188Z remote: Compressing objects: 91% (162/177) 2024-08-06T20:31:17.8413687Z remote: Compressing objects: 92% (163/177) 2024-08-06T20:31:17.8414168Z remote: Compressing objects: 93% (165/177) 2024-08-06T20:31:17.8414807Z remote: Compressing objects: 94% (167/177) 2024-08-06T20:31:17.8415407Z remote: Compressing objects: 95% (169/177) 2024-08-06T20:31:17.8415962Z remote: Compressing objects: 96% (170/177) 2024-08-06T20:31:17.8416458Z remote: Compressing objects: 97% (172/177) 2024-08-06T20:31:17.8416952Z remote: Compressing objects: 98% (174/177) 2024-08-06T20:31:17.8417426Z remote: Compressing objects: 99% (176/177) 2024-08-06T20:31:17.8418024Z remote: Compressing objects: 100% (177/177) 2024-08-06T20:31:17.8418941Z remote: Compressing objects: 100% (177/177), done. 2024-08-06T20:31:44.4282188Z remote: Total 1008693 (delta 148), reused 259 (delta 131), pack-reused 1008383 2024-08-06T20:32:17.4257434Z [command]/usr/bin/git rev-parse --verify --quiet b9d86fa89636e301796d4201f36d86c73f6e49bc^{object} 2024-08-06T20:32:17.4286924Z b9d86fa89636e301796d4201f36d86c73f6e49bc 2024-08-06T20:32:17.4292412Z ##[endgroup] 2024-08-06T20:32:17.4293040Z ##[group]Determining the checkout info 2024-08-06T20:32:17.4294362Z ##[endgroup] 2024-08-06T20:32:17.4294883Z ##[group]Checking out the ref 2024-08-06T20:32:17.4299303Z [command]/usr/bin/git checkout --quiet --force b9d86fa89636e301796d4201f36d86c73f6e49bc 2024-08-06T20:32:19.2914086Z ##[endgroup] 2024-08-06T20:32:19.2915002Z ##[group]Setting up auth for fetching submodules 2024-08-06T20:32:19.2920106Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-06T20:32:19.2977262Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-06T20:32:19.3008633Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-06T20:32:19.3042065Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-06T20:32:19.3072251Z ##[endgroup] 2024-08-06T20:32:19.3073281Z ##[group]Fetching submodules 2024-08-06T20:32:19.3143711Z [command]/usr/bin/git submodule sync --recursive 2024-08-06T20:32:19.3406483Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-06T20:32:19.3713507Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-06T20:32:19.3716156Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-06T20:32:19.3719247Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-06T20:32:19.3722335Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-06T20:32:19.3726105Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-06T20:32:19.3729648Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-06T20:32:19.3732862Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-06T20:32:19.3736629Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-06T20:32:19.3741484Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-06T20:32:19.3745661Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-06T20:32:19.3749855Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-06T20:32:19.3753979Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-06T20:32:19.3758522Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-06T20:32:19.3763146Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-06T20:32:19.3767485Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-06T20:32:19.3772391Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-08-06T20:32:19.3777675Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-06T20:32:19.3783194Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-06T20:32:19.3788295Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-06T20:32:19.3793681Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-06T20:32:19.3799066Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-06T20:32:19.3804755Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-06T20:32:19.3810491Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-06T20:32:19.3816362Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-06T20:32:19.3823093Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-06T20:32:19.3829028Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-06T20:32:19.3835594Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-06T20:32:19.3841777Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-06T20:32:19.3848278Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-06T20:32:19.3854899Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-06T20:32:19.3862388Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-06T20:32:19.3869106Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-06T20:32:19.3876241Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-06T20:32:19.3883675Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-06T20:32:19.3890670Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-06T20:32:19.3918992Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-06T20:32:19.6478905Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-06T20:32:19.8115056Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-06T20:32:19.9856885Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-06T20:32:20.2123684Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-06T20:32:22.3297255Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-06T20:32:34.0338722Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-06T20:32:34.4543518Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-06T20:32:34.9252360Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-06T20:32:35.5906093Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-06T20:32:36.9827346Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-06T20:32:39.3382068Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-06T20:32:46.4959183Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-06T20:32:47.8933041Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-06T20:32:49.7761597Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-06T20:32:51.0663486Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-08-06T20:32:51.2235483Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-06T20:32:51.6804051Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-06T20:32:52.0357653Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-06T20:32:53.6894471Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-06T20:32:54.0369517Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-06T20:32:54.2733243Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-06T20:32:56.1272076Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-06T20:32:56.9060129Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-06T20:32:57.6244067Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-06T20:33:04.5288165Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-06T20:33:06.6784293Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-06T20:33:10.3511005Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-06T20:33:10.5637660Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-06T20:33:20.2603465Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-06T20:33:20.4242084Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-06T20:33:20.6119203Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-06T20:33:21.5551467Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-06T20:33:21.8346461Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-06T20:33:22.5056951Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-06T20:33:22.9372257Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-06T20:33:22.9497401Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-06T20:33:22.9593518Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-06T20:33:22.9874132Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-06T20:33:23.0319322Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-06T20:33:24.1948745Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-06T20:33:24.2211883Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-06T20:33:24.2751298Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-06T20:33:24.3941069Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-06T20:33:24.4333401Z Submodule path 'third_party/cudnn_frontend': checked out '98ca4e1941fe3263f128f74f10063a3ea35c7019' 2024-08-06T20:33:25.0981569Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-06T20:33:25.4083577Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-06T20:33:25.5051645Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-06T20:33:25.5069331Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-06T20:33:25.5072237Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-06T20:33:25.5075397Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-06T20:33:25.5078700Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-06T20:33:25.5082368Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-06T20:33:25.5109189Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-06T20:33:26.6267021Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-06T20:33:27.2544407Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-06T20:33:29.5466869Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-06T20:33:30.7250857Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-06T20:33:31.0357300Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-06T20:33:31.1555536Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-06T20:33:31.6819151Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-06T20:33:31.7576265Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-06T20:33:31.7711320Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-06T20:33:31.9104261Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-06T20:33:31.9582943Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-06T20:33:31.9685287Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-08-06T20:33:32.0147586Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-06T20:33:32.0440496Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-06T20:33:32.0979446Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-06T20:33:32.1118616Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-06T20:33:32.1135635Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-06T20:33:32.1163423Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-06T20:33:47.0439025Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-06T20:33:47.0641399Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-06T20:33:47.1778872Z Submodule path 'third_party/kineto': checked out 'da2f2682cabaf95d601fa2a9b7e0979f84fe7667' 2024-08-06T20:33:47.1797576Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-06T20:33:47.1800982Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-06T20:33:47.1804826Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-06T20:33:47.1831339Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-06T20:33:47.8710029Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-06T20:33:49.1234687Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-06T20:33:50.3929313Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-06T20:33:50.3946531Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-06T20:33:50.3949540Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-06T20:33:50.3952790Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-06T20:33:50.3956212Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-06T20:33:50.3959736Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-06T20:33:50.3963435Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-06T20:33:50.3967255Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-06T20:33:50.3971312Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-06T20:33:50.3999001Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-06T20:33:51.3760171Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-06T20:33:51.7684969Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-06T20:33:53.0488603Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-06T20:33:53.4427548Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-06T20:33:53.9993131Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-06T20:33:56.3867278Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-06T20:34:04.4656810Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-06T20:34:04.8842544Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-06T20:34:04.9049940Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-06T20:34:04.9475106Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-06T20:34:04.9616153Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-06T20:34:04.9633585Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-06T20:34:04.9659470Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-06T20:34:05.2700703Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-06T20:34:05.2909137Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-06T20:34:05.3395862Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-06T20:34:05.4614434Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-06T20:34:05.4799036Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-06T20:34:05.5250638Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-06T20:34:05.5933426Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-06T20:34:05.6362946Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-06T20:34:05.6634752Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-06T20:34:05.7949275Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-06T20:34:06.2349085Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-06T20:34:06.2386305Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-06T20:34:06.2389508Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-06T20:34:06.2416724Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-06T20:34:06.6850694Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-06T20:34:07.7089326Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-06T20:34:07.7492045Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-06T20:34:07.8293089Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-06T20:34:07.8304317Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-06T20:34:07.8307183Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-06T20:34:07.8310473Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-06T20:34:07.8313938Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-06T20:34:07.8317732Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-06T20:34:07.8321423Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-06T20:34:07.8325260Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-06T20:34:07.8329084Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-06T20:34:07.8355258Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-06T20:34:08.2547455Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-06T20:34:09.4000107Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-06T20:34:09.7679914Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-06T20:34:16.6566191Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-06T20:34:16.9355748Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-06T20:34:17.1220263Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-06T20:34:17.4798648Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-06T20:34:25.8801905Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-06T20:34:25.9277451Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-06T20:34:25.9452869Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-06T20:34:26.0705148Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-06T20:34:26.0842869Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-06T20:34:26.1000987Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-06T20:34:26.1170297Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-06T20:34:26.1186948Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-06T20:34:26.1190098Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-06T20:34:26.1215236Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-06T20:34:28.2067025Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-06T20:34:29.6458124Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-06T20:34:29.6999360Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-06T20:34:30.2662889Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-06T20:34:30.2789487Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-06T20:34:30.6087353Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-06T20:34:30.6110719Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-06T20:34:30.6114442Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-06T20:34:30.6141808Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-06T20:34:31.0217724Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-06T20:34:32.2382894Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-06T20:34:32.3239941Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-06T20:34:32.3332856Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-06T20:34:32.3470021Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-06T20:34:32.3899226Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-06T20:34:32.4227104Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-06T20:34:32.4723540Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-06T20:34:32.5020380Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-06T20:34:32.5037774Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-06T20:34:32.5041158Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-06T20:34:32.5044171Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-06T20:34:32.5047621Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-06T20:34:32.5073576Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-06T20:34:33.6864041Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-06T20:34:33.9163828Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-06T20:34:35.2265824Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-06T20:34:36.2409764Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-06T20:34:36.2577465Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-06T20:34:36.3406737Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-06T20:34:36.3728874Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-06T20:34:36.3745229Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-06T20:34:36.3770799Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-06T20:34:36.5615309Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-06T20:34:36.5650724Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-06T20:34:36.5967882Z Entering 'android/libs/fbjni' 2024-08-06T20:34:36.6009780Z Entering 'third_party/FP16' 2024-08-06T20:34:36.6051887Z Entering 'third_party/FXdiv' 2024-08-06T20:34:36.6093670Z Entering 'third_party/NNPACK' 2024-08-06T20:34:36.6135330Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-06T20:34:36.6177116Z Entering 'third_party/XNNPACK' 2024-08-06T20:34:36.6236458Z Entering 'third_party/benchmark' 2024-08-06T20:34:36.6278061Z Entering 'third_party/cpp-httplib' 2024-08-06T20:34:36.6319485Z Entering 'third_party/cpuinfo' 2024-08-06T20:34:36.6361239Z Entering 'third_party/cudnn_frontend' 2024-08-06T20:34:36.6402289Z Entering 'third_party/cutlass' 2024-08-06T20:34:36.6453530Z Entering 'third_party/eigen' 2024-08-06T20:34:36.6497346Z Entering 'third_party/fbgemm' 2024-08-06T20:34:36.6538647Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-06T20:34:36.6580691Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-06T20:34:36.6623070Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-06T20:34:36.6670663Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-06T20:34:36.6711021Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-06T20:34:36.6752827Z Entering 'third_party/flatbuffers' 2024-08-06T20:34:36.6797050Z Entering 'third_party/fmt' 2024-08-06T20:34:36.6837271Z Entering 'third_party/foxi' 2024-08-06T20:34:36.6879504Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-06T20:34:36.6920453Z Entering 'third_party/gloo' 2024-08-06T20:34:36.6961546Z Entering 'third_party/googletest' 2024-08-06T20:34:36.7002960Z Entering 'third_party/ideep' 2024-08-06T20:34:36.7043055Z Entering 'third_party/ideep/mkl-dnn' 2024-08-06T20:34:36.7092100Z Entering 'third_party/ittapi' 2024-08-06T20:34:36.7132812Z Entering 'third_party/kineto' 2024-08-06T20:34:36.7174099Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-06T20:34:36.7215353Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-06T20:34:36.7257734Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-06T20:34:36.7298672Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-06T20:34:36.7339033Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-06T20:34:36.7378111Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-06T20:34:36.7420710Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-06T20:34:36.7460782Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-06T20:34:36.7501771Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-06T20:34:36.7545332Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-06T20:34:36.7586898Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-06T20:34:36.7627511Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-06T20:34:36.7669510Z Entering 'third_party/mimalloc' 2024-08-06T20:34:36.7711119Z Entering 'third_party/nccl/nccl' 2024-08-06T20:34:36.7752242Z Entering 'third_party/nlohmann' 2024-08-06T20:34:36.7794904Z Entering 'third_party/onnx' 2024-08-06T20:34:36.7852499Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-06T20:34:36.7893495Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-06T20:34:36.7937491Z Entering 'third_party/opentelemetry-cpp' 2024-08-06T20:34:36.7981546Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-06T20:34:36.8023436Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-06T20:34:36.8063873Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-06T20:34:36.8104566Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-06T20:34:36.8146997Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-06T20:34:36.8187784Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-06T20:34:36.8228017Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-06T20:34:36.8267990Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-06T20:34:36.8311138Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-06T20:34:36.8353567Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-06T20:34:36.8415806Z Entering 'third_party/pocketfft' 2024-08-06T20:34:36.8456418Z Entering 'third_party/protobuf' 2024-08-06T20:34:36.8502144Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-06T20:34:36.8543180Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-06T20:34:36.8585424Z Entering 'third_party/psimd' 2024-08-06T20:34:36.8626516Z Entering 'third_party/pthreadpool' 2024-08-06T20:34:36.8668009Z Entering 'third_party/pybind11' 2024-08-06T20:34:36.8709109Z Entering 'third_party/python-peachpy' 2024-08-06T20:34:36.8750592Z Entering 'third_party/sleef' 2024-08-06T20:34:36.8792226Z Entering 'third_party/tensorpipe' 2024-08-06T20:34:36.8832960Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-06T20:34:36.8873120Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-06T20:34:36.8912758Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-06T20:34:36.8952844Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-06T20:34:36.8992676Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-06T20:34:36.9047412Z ##[endgroup] 2024-08-06T20:34:36.9050720Z ##[group]Persisting credentials for submodules 2024-08-06T20:34:36.9055575Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-06T20:34:36.9367554Z Entering 'android/libs/fbjni' 2024-08-06T20:34:36.9421453Z Entering 'third_party/FP16' 2024-08-06T20:34:36.9471767Z Entering 'third_party/FXdiv' 2024-08-06T20:34:36.9522304Z Entering 'third_party/NNPACK' 2024-08-06T20:34:36.9574555Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-06T20:34:36.9626586Z Entering 'third_party/XNNPACK' 2024-08-06T20:34:36.9695191Z Entering 'third_party/benchmark' 2024-08-06T20:34:36.9747480Z Entering 'third_party/cpp-httplib' 2024-08-06T20:34:36.9798830Z Entering 'third_party/cpuinfo' 2024-08-06T20:34:36.9850064Z Entering 'third_party/cudnn_frontend' 2024-08-06T20:34:36.9902597Z Entering 'third_party/cutlass' 2024-08-06T20:34:36.9962543Z Entering 'third_party/eigen' 2024-08-06T20:34:37.0015331Z Entering 'third_party/fbgemm' 2024-08-06T20:34:37.0067595Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-06T20:34:37.0116878Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-06T20:34:37.0167755Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-06T20:34:37.0224975Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-06T20:34:37.0274107Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-06T20:34:37.0324811Z Entering 'third_party/flatbuffers' 2024-08-06T20:34:37.0377714Z Entering 'third_party/fmt' 2024-08-06T20:34:37.0429920Z Entering 'third_party/foxi' 2024-08-06T20:34:37.0480050Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-06T20:34:37.0530099Z Entering 'third_party/gloo' 2024-08-06T20:34:37.0581522Z Entering 'third_party/googletest' 2024-08-06T20:34:37.0631566Z Entering 'third_party/ideep' 2024-08-06T20:34:37.0680674Z Entering 'third_party/ideep/mkl-dnn' 2024-08-06T20:34:37.0737644Z Entering 'third_party/ittapi' 2024-08-06T20:34:37.0790530Z Entering 'third_party/kineto' 2024-08-06T20:34:37.0840890Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-06T20:34:37.0892198Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-06T20:34:37.0945163Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-06T20:34:37.0995440Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-06T20:34:37.1047452Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-06T20:34:37.1096742Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-06T20:34:37.1149825Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-06T20:34:37.1201756Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-06T20:34:37.1253739Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-06T20:34:37.1307783Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-06T20:34:37.1358736Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-06T20:34:37.1409580Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-06T20:34:37.1461914Z Entering 'third_party/mimalloc' 2024-08-06T20:34:37.1512909Z Entering 'third_party/nccl/nccl' 2024-08-06T20:34:37.1564088Z Entering 'third_party/nlohmann' 2024-08-06T20:34:37.1616899Z Entering 'third_party/onnx' 2024-08-06T20:34:37.1684891Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-06T20:34:37.1735497Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-06T20:34:37.1789154Z Entering 'third_party/opentelemetry-cpp' 2024-08-06T20:34:37.1841266Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-06T20:34:37.1891549Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-06T20:34:37.1941353Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-06T20:34:37.1990376Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-06T20:34:37.2043409Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-06T20:34:37.2094350Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-06T20:34:37.2145291Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-06T20:34:37.2194049Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-06T20:34:37.2247793Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-06T20:34:37.2300082Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-06T20:34:37.2371389Z Entering 'third_party/pocketfft' 2024-08-06T20:34:37.2422768Z Entering 'third_party/protobuf' 2024-08-06T20:34:37.2476862Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-06T20:34:37.2526927Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-06T20:34:37.2578890Z Entering 'third_party/psimd' 2024-08-06T20:34:37.2631468Z Entering 'third_party/pthreadpool' 2024-08-06T20:34:37.2684613Z Entering 'third_party/pybind11' 2024-08-06T20:34:37.2736248Z Entering 'third_party/python-peachpy' 2024-08-06T20:34:37.2788473Z Entering 'third_party/sleef' 2024-08-06T20:34:37.2840495Z Entering 'third_party/tensorpipe' 2024-08-06T20:34:37.2891490Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-06T20:34:37.2943616Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-06T20:34:37.2993225Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-06T20:34:37.3044006Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-06T20:34:37.3094460Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-06T20:34:37.3163905Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-06T20:34:37.3476520Z Entering 'android/libs/fbjni' 2024-08-06T20:34:37.3525645Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-06T20:34:37.3544737Z Entering 'third_party/FP16' 2024-08-06T20:34:37.3593479Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-06T20:34:37.3611973Z Entering 'third_party/FXdiv' 2024-08-06T20:34:37.3659362Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-06T20:34:37.3678301Z Entering 'third_party/NNPACK' 2024-08-06T20:34:37.3727320Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-06T20:34:37.3747508Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-06T20:34:37.3795587Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-06T20:34:37.3814964Z Entering 'third_party/XNNPACK' 2024-08-06T20:34:37.3863778Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-06T20:34:37.3901268Z Entering 'third_party/benchmark' 2024-08-06T20:34:37.3947966Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-06T20:34:37.3966236Z Entering 'third_party/cpp-httplib' 2024-08-06T20:34:37.4014020Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-06T20:34:37.4033270Z Entering 'third_party/cpuinfo' 2024-08-06T20:34:37.4081034Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-06T20:34:37.4100767Z Entering 'third_party/cudnn_frontend' 2024-08-06T20:34:37.4147794Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-06T20:34:37.4166252Z Entering 'third_party/cutlass' 2024-08-06T20:34:37.4214171Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-06T20:34:37.4241562Z Entering 'third_party/eigen' 2024-08-06T20:34:37.4289047Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-06T20:34:37.4310798Z Entering 'third_party/fbgemm' 2024-08-06T20:34:37.4358169Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-06T20:34:37.4377259Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-06T20:34:37.4425128Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-06T20:34:37.4443456Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-06T20:34:37.4490885Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-06T20:34:37.4510341Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-06T20:34:37.4556893Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-06T20:34:37.4582524Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-06T20:34:37.4629245Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-06T20:34:37.4647440Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-06T20:34:37.4694091Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-06T20:34:37.4713703Z Entering 'third_party/flatbuffers' 2024-08-06T20:34:37.4761099Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-06T20:34:37.4782927Z Entering 'third_party/fmt' 2024-08-06T20:34:37.4829880Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-06T20:34:37.4848312Z Entering 'third_party/foxi' 2024-08-06T20:34:37.4895308Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-08-06T20:34:37.4914222Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-06T20:34:37.4961272Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-06T20:34:37.4979876Z Entering 'third_party/gloo' 2024-08-06T20:34:37.5026630Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-06T20:34:37.5045327Z Entering 'third_party/googletest' 2024-08-06T20:34:37.5093493Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-06T20:34:37.5123935Z Entering 'third_party/ideep' 2024-08-06T20:34:37.5159499Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-06T20:34:37.5176387Z Entering 'third_party/ideep/mkl-dnn' 2024-08-06T20:34:37.5223626Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-06T20:34:37.5249670Z Entering 'third_party/ittapi' 2024-08-06T20:34:37.5296762Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-06T20:34:37.5316049Z Entering 'third_party/kineto' 2024-08-06T20:34:37.5363715Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-06T20:34:37.5382301Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-06T20:34:37.5430147Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-06T20:34:37.5447324Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-06T20:34:37.5495444Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-06T20:34:37.5515625Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-06T20:34:37.5563253Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-06T20:34:37.5581700Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-06T20:34:37.5629687Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-06T20:34:37.5648171Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-06T20:34:37.5696214Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-06T20:34:37.5713620Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-06T20:34:37.5761102Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-06T20:34:37.5781433Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-06T20:34:37.5828106Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-06T20:34:37.5846579Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-06T20:34:37.5894788Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-06T20:34:37.5913664Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-06T20:34:37.5962389Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-06T20:34:37.5982356Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-06T20:34:37.6030418Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-06T20:34:37.6049598Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-06T20:34:37.6097099Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-06T20:34:37.6115253Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-06T20:34:37.6161589Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-06T20:34:37.6181317Z Entering 'third_party/mimalloc' 2024-08-06T20:34:37.6228575Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-06T20:34:37.6247298Z Entering 'third_party/nccl/nccl' 2024-08-06T20:34:37.6294230Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-06T20:34:37.6313950Z Entering 'third_party/nlohmann' 2024-08-06T20:34:37.6361105Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-06T20:34:37.6381477Z Entering 'third_party/onnx' 2024-08-06T20:34:37.6429538Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-06T20:34:37.6465350Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-06T20:34:37.6512495Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-06T20:34:37.6531252Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-06T20:34:37.6578744Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-06T20:34:37.6599439Z Entering 'third_party/opentelemetry-cpp' 2024-08-06T20:34:37.6648360Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-06T20:34:37.6669039Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-06T20:34:37.6716534Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-06T20:34:37.6734287Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-06T20:34:37.6781915Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-06T20:34:37.6800592Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-06T20:34:37.6847566Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-06T20:34:37.6865703Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-06T20:34:37.6912004Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-06T20:34:37.6931975Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-06T20:34:37.6979009Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-06T20:34:37.6996961Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-06T20:34:37.7044384Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-06T20:34:37.7063227Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-06T20:34:37.7111906Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-06T20:34:37.7129191Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-06T20:34:37.7176446Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-06T20:34:37.7197714Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-06T20:34:37.7245423Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-06T20:34:37.7266082Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-06T20:34:37.7312813Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-06T20:34:37.7354113Z Entering 'third_party/pocketfft' 2024-08-06T20:34:37.7401156Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-06T20:34:37.7419501Z Entering 'third_party/protobuf' 2024-08-06T20:34:37.7466863Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-06T20:34:37.7488133Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-06T20:34:37.7535687Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-06T20:34:37.7553543Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-06T20:34:37.7600497Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-06T20:34:37.7620167Z Entering 'third_party/psimd' 2024-08-06T20:34:37.7666957Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-06T20:34:37.7685491Z Entering 'third_party/pthreadpool' 2024-08-06T20:34:37.7732450Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-06T20:34:37.7751018Z Entering 'third_party/pybind11' 2024-08-06T20:34:37.7798248Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-06T20:34:37.7816720Z Entering 'third_party/python-peachpy' 2024-08-06T20:34:37.7864187Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-06T20:34:37.7882398Z Entering 'third_party/sleef' 2024-08-06T20:34:37.7929101Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-06T20:34:37.7948426Z Entering 'third_party/tensorpipe' 2024-08-06T20:34:37.7995787Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-06T20:34:37.8014054Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-06T20:34:37.8060168Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-06T20:34:37.8078125Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-06T20:34:37.8125495Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-06T20:34:37.8143275Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-06T20:34:37.8189308Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-06T20:34:37.8207071Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-06T20:34:37.8253227Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-06T20:34:37.8270669Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-06T20:34:37.8317610Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-06T20:34:37.9418091Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-06T20:34:37.9727883Z Entering 'android/libs/fbjni' 2024-08-06T20:34:37.9769372Z Entering 'third_party/FP16' 2024-08-06T20:34:37.9810520Z Entering 'third_party/FXdiv' 2024-08-06T20:34:37.9851901Z Entering 'third_party/NNPACK' 2024-08-06T20:34:37.9893571Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-06T20:34:37.9934610Z Entering 'third_party/XNNPACK' 2024-08-06T20:34:37.9993869Z Entering 'third_party/benchmark' 2024-08-06T20:34:38.0035154Z Entering 'third_party/cpp-httplib' 2024-08-06T20:34:38.0075846Z Entering 'third_party/cpuinfo' 2024-08-06T20:34:38.0118823Z Entering 'third_party/cudnn_frontend' 2024-08-06T20:34:38.0161122Z Entering 'third_party/cutlass' 2024-08-06T20:34:38.0210542Z Entering 'third_party/eigen' 2024-08-06T20:34:38.0254050Z Entering 'third_party/fbgemm' 2024-08-06T20:34:38.0294942Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-06T20:34:38.0336700Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-06T20:34:38.0377573Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-06T20:34:38.0425376Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-06T20:34:38.0465824Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-06T20:34:38.0506993Z Entering 'third_party/flatbuffers' 2024-08-06T20:34:38.0551072Z Entering 'third_party/fmt' 2024-08-06T20:34:38.0591940Z Entering 'third_party/foxi' 2024-08-06T20:34:38.0632425Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-06T20:34:38.0674562Z Entering 'third_party/gloo' 2024-08-06T20:34:38.0715683Z Entering 'third_party/googletest' 2024-08-06T20:34:38.0757553Z Entering 'third_party/ideep' 2024-08-06T20:34:38.0798274Z Entering 'third_party/ideep/mkl-dnn' 2024-08-06T20:34:38.0847169Z Entering 'third_party/ittapi' 2024-08-06T20:34:38.0888395Z Entering 'third_party/kineto' 2024-08-06T20:34:38.0929195Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-06T20:34:38.0970284Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-06T20:34:38.1012787Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-06T20:34:38.1053846Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-06T20:34:38.1095275Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-06T20:34:38.1135873Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-06T20:34:38.1179045Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-06T20:34:38.1221891Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-06T20:34:38.1262890Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-06T20:34:38.1307308Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-06T20:34:38.1349744Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-06T20:34:38.1390464Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-06T20:34:38.1432752Z Entering 'third_party/mimalloc' 2024-08-06T20:34:38.1474569Z Entering 'third_party/nccl/nccl' 2024-08-06T20:34:38.1515591Z Entering 'third_party/nlohmann' 2024-08-06T20:34:38.1558717Z Entering 'third_party/onnx' 2024-08-06T20:34:38.1616332Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-06T20:34:38.1657749Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-06T20:34:38.1701452Z Entering 'third_party/opentelemetry-cpp' 2024-08-06T20:34:38.1744859Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-06T20:34:38.1786306Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-06T20:34:38.1827582Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-06T20:34:38.1867619Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-06T20:34:38.1909453Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-06T20:34:38.1949390Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-06T20:34:38.1990073Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-06T20:34:38.2029637Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-06T20:34:38.2073517Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-06T20:34:38.2115358Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-06T20:34:38.2178002Z Entering 'third_party/pocketfft' 2024-08-06T20:34:38.2221283Z Entering 'third_party/protobuf' 2024-08-06T20:34:38.2266098Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-06T20:34:38.2306981Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-06T20:34:38.2349978Z Entering 'third_party/psimd' 2024-08-06T20:34:38.2391033Z Entering 'third_party/pthreadpool' 2024-08-06T20:34:38.2432235Z Entering 'third_party/pybind11' 2024-08-06T20:34:38.2473578Z Entering 'third_party/python-peachpy' 2024-08-06T20:34:38.2514538Z Entering 'third_party/sleef' 2024-08-06T20:34:38.2555454Z Entering 'third_party/tensorpipe' 2024-08-06T20:34:38.2597738Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-06T20:34:38.2638971Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-06T20:34:38.2679969Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-06T20:34:38.2720866Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-06T20:34:38.2761016Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-06T20:34:38.2819282Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-06T20:34:38.3138685Z Entering 'android/libs/fbjni' 2024-08-06T20:34:38.3181221Z Entering 'third_party/FP16' 2024-08-06T20:34:38.3224088Z Entering 'third_party/FXdiv' 2024-08-06T20:34:38.3266608Z Entering 'third_party/NNPACK' 2024-08-06T20:34:38.3308735Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-06T20:34:38.3351036Z Entering 'third_party/XNNPACK' 2024-08-06T20:34:38.3412464Z Entering 'third_party/benchmark' 2024-08-06T20:34:38.3454284Z Entering 'third_party/cpp-httplib' 2024-08-06T20:34:38.3496579Z Entering 'third_party/cpuinfo' 2024-08-06T20:34:38.3541418Z Entering 'third_party/cudnn_frontend' 2024-08-06T20:34:38.3582974Z Entering 'third_party/cutlass' 2024-08-06T20:34:38.3633504Z Entering 'third_party/eigen' 2024-08-06T20:34:38.3677008Z Entering 'third_party/fbgemm' 2024-08-06T20:34:38.3720059Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-06T20:34:38.3760649Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-06T20:34:38.3801455Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-06T20:34:38.3848581Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-06T20:34:38.3888495Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-06T20:34:38.3929429Z Entering 'third_party/flatbuffers' 2024-08-06T20:34:38.3973710Z Entering 'third_party/fmt' 2024-08-06T20:34:38.4014844Z Entering 'third_party/foxi' 2024-08-06T20:34:38.4055485Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-06T20:34:38.4097274Z Entering 'third_party/gloo' 2024-08-06T20:34:38.4139638Z Entering 'third_party/googletest' 2024-08-06T20:34:38.4181672Z Entering 'third_party/ideep' 2024-08-06T20:34:38.4223215Z Entering 'third_party/ideep/mkl-dnn' 2024-08-06T20:34:38.4274144Z Entering 'third_party/ittapi' 2024-08-06T20:34:38.4314845Z Entering 'third_party/kineto' 2024-08-06T20:34:38.4357417Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-06T20:34:38.4399359Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-06T20:34:38.4443626Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-06T20:34:38.4485508Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-06T20:34:38.4526872Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-06T20:34:38.4567016Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-06T20:34:38.4610001Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-06T20:34:38.4650658Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-06T20:34:38.4692426Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-06T20:34:38.4735319Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-06T20:34:38.4778025Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-06T20:34:38.4820250Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-06T20:34:38.4861674Z Entering 'third_party/mimalloc' 2024-08-06T20:34:38.4903400Z Entering 'third_party/nccl/nccl' 2024-08-06T20:34:38.4945072Z Entering 'third_party/nlohmann' 2024-08-06T20:34:38.4987677Z Entering 'third_party/onnx' 2024-08-06T20:34:38.5045865Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-06T20:34:38.5087141Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-06T20:34:38.5132580Z Entering 'third_party/opentelemetry-cpp' 2024-08-06T20:34:38.5175263Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-06T20:34:38.5215959Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-06T20:34:38.5257152Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-06T20:34:38.5298088Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-06T20:34:38.5341023Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-06T20:34:38.5381140Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-06T20:34:38.5422358Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-06T20:34:38.5461879Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-06T20:34:38.5505654Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-06T20:34:38.5548567Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-06T20:34:38.5611541Z Entering 'third_party/pocketfft' 2024-08-06T20:34:38.5653509Z Entering 'third_party/protobuf' 2024-08-06T20:34:38.5699352Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-06T20:34:38.5740762Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-06T20:34:38.5783493Z Entering 'third_party/psimd' 2024-08-06T20:34:38.5825265Z Entering 'third_party/pthreadpool' 2024-08-06T20:34:38.5866655Z Entering 'third_party/pybind11' 2024-08-06T20:34:38.5908101Z Entering 'third_party/python-peachpy' 2024-08-06T20:34:38.5949602Z Entering 'third_party/sleef' 2024-08-06T20:34:38.5992031Z Entering 'third_party/tensorpipe' 2024-08-06T20:34:38.6033033Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-06T20:34:38.6074254Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-06T20:34:38.6115518Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-06T20:34:38.6156176Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-06T20:34:38.6195754Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-06T20:34:38.6250237Z ##[endgroup] 2024-08-06T20:34:38.6293315Z [command]/usr/bin/git log -1 --format='%H' 2024-08-06T20:34:38.6319912Z 'b9d86fa89636e301796d4201f36d86c73f6e49bc' 2024-08-06T20:34:38.6505582Z Prepare all required actions 2024-08-06T20:34:38.6506258Z Getting action download info 2024-08-06T20:34:38.7851795Z ##[group]Run ./.github/actions/setup-linux 2024-08-06T20:34:38.7852218Z env: 2024-08-06T20:34:38.7852496Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:34:38.7852847Z ##[endgroup] 2024-08-06T20:34:38.7945273Z ##[group]Run set -euo pipefail 2024-08-06T20:34:38.7945734Z set -euo pipefail 2024-08-06T20:34:38.7946344Z function get_ec2_metadata() { 2024-08-06T20:34:38.7946877Z  # Pulled from instance metadata endpoint for EC2 2024-08-06T20:34:38.7947736Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-06T20:34:38.7948508Z  category=$1 2024-08-06T20:34:38.7949003Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-06T20:34:38.7949614Z  runner_name_str=i-06f045a861c998624 2024-08-06T20:34:38.7950150Z  if [[ -f /.inarc ]]; then 2024-08-06T20:34:38.7950651Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-06T20:34:38.7951186Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-06T20:34:38.7951838Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-06T20:34:38.7952425Z  else 2024-08-06T20:34:38.7952896Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-06T20:34:38.7953477Z  fi 2024-08-06T20:34:38.7953786Z } 2024-08-06T20:34:38.7954140Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-06T20:34:38.7954733Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-06T20:34:38.7955394Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-06T20:34:38.7955952Z echo "system info $(uname -a)" 2024-08-06T20:34:38.7964200Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:34:38.7964718Z env: 2024-08-06T20:34:38.7965019Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:34:38.7965388Z ##[endgroup] 2024-08-06T20:34:38.8064322Z ami-id: ami-0ce0c36d7a00b20e2 2024-08-06T20:34:38.8128397Z instance-id: i-06f045a861c998624 2024-08-06T20:34:38.8192805Z instance-type: g3.8xlarge 2024-08-06T20:34:38.8201301Z system info Linux ip-10-0-56-92.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-06T20:34:38.8233732Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-06T20:34:38.8234521Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-06T20:34:38.8242149Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:34:38.8242633Z env: 2024-08-06T20:34:38.8242926Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:34:38.8243255Z ##[endgroup] 2024-08-06T20:34:38.8329375Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-06T20:34:38.8329959Z if systemctl is-active --quiet docker; then 2024-08-06T20:34:38.8330470Z  echo "Docker daemon is running..."; 2024-08-06T20:34:38.8330900Z else 2024-08-06T20:34:38.8331340Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-06T20:34:38.8331877Z fi 2024-08-06T20:34:38.8339598Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:34:38.8340069Z env: 2024-08-06T20:34:38.8340382Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:34:38.8340732Z ##[endgroup] 2024-08-06T20:34:38.8386547Z Docker daemon is running... 2024-08-06T20:34:38.8445942Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-06T20:34:38.8446457Z with: 2024-08-06T20:34:38.8446733Z shell: bash 2024-08-06T20:34:38.8447031Z timeout_minutes: 5 2024-08-06T20:34:38.8447342Z max_attempts: 3 2024-08-06T20:34:38.8447646Z retry_wait_seconds: 30 2024-08-06T20:34:38.8449067Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-08-06T20:34:38.8450392Z polling_interval_seconds: 1 2024-08-06T20:34:38.8450747Z warning_on_retry: true 2024-08-06T20:34:38.8451090Z continue_on_error: false 2024-08-06T20:34:38.8451423Z env: 2024-08-06T20:34:38.8451687Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:34:38.8452176Z AWS_RETRY_MODE: standard 2024-08-06T20:34:38.8452517Z AWS_MAX_ATTEMPTS: 5 2024-08-06T20:34:38.8452841Z AWS_DEFAULT_REGION: us-east-1 2024-08-06T20:34:38.8453191Z ##[endgroup] 2024-08-06T20:34:39.8481494Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-06T20:34:39.8482264Z Configure a credential helper to remove this warning. See 2024-08-06T20:34:39.8482972Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-06T20:34:39.8483472Z 2024-08-06T20:34:39.8483644Z Login Succeeded 2024-08-06T20:34:39.9175936Z Command completed after 1 attempt(s). 2024-08-06T20:34:39.9546805Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-06T20:34:39.9547469Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-06T20:34:39.9548035Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-06T20:34:39.9556029Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:34:39.9556540Z env: 2024-08-06T20:34:39.9556820Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:34:39.9557168Z ##[endgroup] 2024-08-06T20:34:39.9839774Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-06T20:34:39.9840491Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-06T20:34:39.9841020Z # shellcheck disable=SC2046 2024-08-06T20:34:39.9841458Z docker stop $(docker ps -q) || true 2024-08-06T20:34:39.9841927Z # Prune all of the docker images 2024-08-06T20:34:39.9842337Z docker system prune -af 2024-08-06T20:34:39.9849598Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:34:39.9850077Z env: 2024-08-06T20:34:39.9850350Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:34:39.9850702Z ##[endgroup] 2024-08-06T20:34:40.0193176Z "docker stop" requires at least 1 argument. 2024-08-06T20:34:40.0193659Z See 'docker stop --help'. 2024-08-06T20:34:40.0193918Z 2024-08-06T20:34:40.0194169Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-06T20:34:40.0194531Z 2024-08-06T20:34:40.0194689Z Stop one or more running containers 2024-08-06T20:34:40.0896887Z Total reclaimed space: 0B 2024-08-06T20:34:40.1028114Z ##[group]Run set +e 2024-08-06T20:34:40.1028520Z set +e 2024-08-06T20:34:40.1028842Z set -x 2024-08-06T20:34:40.1029129Z  2024-08-06T20:34:40.1029463Z PT_DOMAIN=download.pytorch.org 2024-08-06T20:34:40.1030221Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-06T20:34:40.1031135Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-06T20:34:40.1031799Z # one is returned at random 2024-08-06T20:34:40.1032314Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-06T20:34:40.1032782Z  2024-08-06T20:34:40.1033101Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-06T20:34:40.1033675Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-06T20:34:40.1034313Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-06T20:34:40.1034828Z  2024-08-06T20:34:40.1035154Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-06T20:34:40.1035665Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-06T20:34:40.1036116Z  exit 1 2024-08-06T20:34:40.1036438Z  fi 2024-08-06T20:34:40.1036736Z fi 2024-08-06T20:34:40.1037190Z  2024-08-06T20:34:40.1037566Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-06T20:34:40.1038042Z  # Clean up any old records first 2024-08-06T20:34:40.1038502Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-06T20:34:40.1038930Z fi 2024-08-06T20:34:40.1039218Z  2024-08-06T20:34:40.1039611Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-06T20:34:40.1040322Z cat /etc/hosts 2024-08-06T20:34:40.1048564Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:34:40.1049034Z env: 2024-08-06T20:34:40.1049331Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:34:40.1049689Z ##[endgroup] 2024-08-06T20:34:40.1074664Z + PT_DOMAIN=download.pytorch.org 2024-08-06T20:34:40.1080458Z ++ dig -4 +short download.pytorch.org 2024-08-06T20:34:40.1080974Z ++ tail -n1 2024-08-06T20:34:40.1368390Z + RESOLVED_IP=18.160.10.22 2024-08-06T20:34:40.1368774Z + '[' -z 18.160.10.22 ']' 2024-08-06T20:34:40.1369174Z + grep -r download.pytorch.org /etc/hosts 2024-08-06T20:34:40.1377943Z 18.160.10.76 download.pytorch.org 2024-08-06T20:34:40.1379184Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-08-06T20:34:40.1527715Z + echo '18.160.10.22 download.pytorch.org' 2024-08-06T20:34:40.1528473Z + sudo tee -a /etc/hosts 2024-08-06T20:34:40.1622500Z 18.160.10.22 download.pytorch.org 2024-08-06T20:34:40.1639133Z + cat /etc/hosts 2024-08-06T20:34:40.1645142Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-06T20:34:40.1651825Z ::1 localhost6 localhost6.localdomain6 2024-08-06T20:34:40.1652315Z 18.160.10.22 download.pytorch.org 2024-08-06T20:34:40.2090217Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-06T20:34:40.2090851Z with: 2024-08-06T20:34:40.2091755Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-06T20:34:40.2092753Z docker-build-dir: .ci/docker 2024-08-06T20:34:40.2093133Z working-directory: . 2024-08-06T20:34:40.2093589Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-06T20:34:40.2094080Z force-push: false 2024-08-06T20:34:40.2094390Z env: 2024-08-06T20:34:40.2094677Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:34:40.2095002Z ##[endgroup] 2024-08-06T20:34:40.2401584Z ##[group]Run set -ex 2024-08-06T20:34:40.2402037Z set -ex 2024-08-06T20:34:40.2402356Z  2024-08-06T20:34:40.2402868Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-06T20:34:40.2403736Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-06T20:34:40.2404428Z # job could then download the pre-built image as usual 2024-08-06T20:34:40.2405071Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-06T20:34:40.2405669Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-06T20:34:40.2406233Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-06T20:34:40.2406738Z  2024-08-06T20:34:40.2407197Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-06T20:34:40.2407750Z  exit 0 2024-08-06T20:34:40.2408037Z else 2024-08-06T20:34:40.2408389Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-06T20:34:40.2408827Z fi 2024-08-06T20:34:40.2409097Z  2024-08-06T20:34:40.2409538Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-06T20:34:40.2410297Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-06T20:34:40.2410953Z  # use it as it is, but first let's extract the tag 2024-08-06T20:34:40.2411560Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-06T20:34:40.2412199Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-06T20:34:40.2412797Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-06T20:34:40.2413320Z else 2024-08-06T20:34:40.2413729Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-06T20:34:40.2414295Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-06T20:34:40.2415333Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-06T20:34:40.2416026Z fi 2024-08-06T20:34:40.2424306Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:34:40.2424776Z env: 2024-08-06T20:34:40.2425076Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:34:40.2425414Z REPO_NAME: pytorch 2024-08-06T20:34:40.2426357Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-06T20:34:40.2427370Z DOCKER_BUILD_DIR: .ci/docker 2024-08-06T20:34:40.2427854Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-06T20:34:40.2428334Z ##[endgroup] 2024-08-06T20:34:40.2453606Z + [[ ! -d .ci/docker ]] 2024-08-06T20:34:40.2453982Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-06T20:34:40.2454359Z + echo skip=false 2024-08-06T20:34:40.2455551Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-06T20:34:40.2461591Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-06T20:34:40.2462593Z ++ awk -F '[:,]' '{print $2}' 2024-08-06T20:34:40.2472510Z + DOCKER_TAG=02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-06T20:34:40.2473069Z + echo docker-tag=02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-06T20:34:40.2474428Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-06T20:34:40.2515632Z ##[group]Run set +e 2024-08-06T20:34:40.2516063Z set +e 2024-08-06T20:34:40.2516379Z set -x 2024-08-06T20:34:40.2516702Z  2024-08-06T20:34:40.2517002Z login() { 2024-08-06T20:34:40.2517662Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-06T20:34:40.2518375Z } 2024-08-06T20:34:40.2518674Z  2024-08-06T20:34:40.2518961Z retry () { 2024-08-06T20:34:40.2519344Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-06T20:34:40.2519784Z } 2024-08-06T20:34:40.2520064Z  2024-08-06T20:34:40.2520395Z retry login "${DOCKER_REGISTRY}" 2024-08-06T20:34:40.2520811Z  2024-08-06T20:34:40.2521280Z # Check if image already exists, if it does then skip building it 2024-08-06T20:34:40.2521975Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-06T20:34:40.2522487Z  exit 0 2024-08-06T20:34:40.2522790Z fi 2024-08-06T20:34:40.2523091Z  2024-08-06T20:34:40.2523564Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-06T20:34:40.2524379Z # be empty. The default action would be to continue rebuild the image 2024-08-06T20:34:40.2525120Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-06T20:34:40.2525761Z  # if we're on the base branch then use the parent commit 2024-08-06T20:34:40.2526312Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-06T20:34:40.2526761Z else 2024-08-06T20:34:40.2527220Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-06T20:34:40.2527866Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-06T20:34:40.2528370Z fi 2024-08-06T20:34:40.2528671Z  2024-08-06T20:34:40.2528985Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-06T20:34:40.2529491Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-06T20:34:40.2529958Z  2024-08-06T20:34:40.2530624Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-06T20:34:40.2531573Z  exit 0 2024-08-06T20:34:40.2531897Z fi 2024-08-06T20:34:40.2532199Z  2024-08-06T20:34:40.2532619Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-06T20:34:40.2533574Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-06T20:34:40.2534384Z  exit 1 2024-08-06T20:34:40.2534688Z fi 2024-08-06T20:34:40.2534990Z  2024-08-06T20:34:40.2535497Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-06T20:34:40.2536376Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-06T20:34:40.2537201Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-06T20:34:40.2538467Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-06T20:34:40.2539469Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-06T20:34:40.2540028Z fi 2024-08-06T20:34:40.2540320Z  2024-08-06T20:34:40.2540831Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-06T20:34:40.2548131Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:34:40.2548636Z env: 2024-08-06T20:34:40.2548931Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:34:40.2549304Z DOCKER_BUILD_DIR: .ci/docker 2024-08-06T20:34:40.2549755Z BASE_REVISION: 1736af7cf736184c356be1bb00f59fb2feea6d7d 2024-08-06T20:34:40.2550795Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-06T20:34:40.2551845Z DOCKER_TAG: 02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-06T20:34:40.2552424Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-06T20:34:40.2552898Z ##[endgroup] 2024-08-06T20:34:40.2576942Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-06T20:34:40.2577514Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-06T20:34:40.2579714Z + aws ecr get-login-password --region us-east-1 2024-08-06T20:34:40.2580723Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-06T20:34:40.7303017Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-06T20:34:40.7303784Z Configure a credential helper to remove this warning. See 2024-08-06T20:34:40.7304502Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-06T20:34:40.7304981Z 2024-08-06T20:34:40.7305143Z Login Succeeded 2024-08-06T20:34:40.7317019Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-06T20:34:40.9445119Z { 2024-08-06T20:34:40.9445930Z "schemaVersion": 2, 2024-08-06T20:34:40.9446612Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-06T20:34:40.9447168Z "config": { 2024-08-06T20:34:40.9447628Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-06T20:34:40.9448156Z "size": 47454, 2024-08-06T20:34:40.9448668Z "digest": "sha256:8e4e19ae0377c273bc603b24bc49d406d6f755072d4534d040c86afbe761a129" 2024-08-06T20:34:40.9449260Z }, 2024-08-06T20:34:40.9449534Z "layers": [ 2024-08-06T20:34:40.9449799Z { 2024-08-06T20:34:40.9450230Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9450775Z "size": 28580681, 2024-08-06T20:34:40.9451304Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-08-06T20:34:40.9451889Z }, 2024-08-06T20:34:40.9452134Z { 2024-08-06T20:34:40.9452566Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9453093Z "size": 7943353, 2024-08-06T20:34:40.9453604Z "digest": "sha256:09d415c238d76b32a7ea4a6e6add9542db9a5641f7f183af70aae185d0709e58" 2024-08-06T20:34:40.9454483Z }, 2024-08-06T20:34:40.9454748Z { 2024-08-06T20:34:40.9455167Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9455699Z "size": 56226389, 2024-08-06T20:34:40.9456230Z "digest": "sha256:9fe6e2e61518cba6844870c03b285737daec35e62baf25ae7744629ed3a7b470" 2024-08-06T20:34:40.9457099Z }, 2024-08-06T20:34:40.9457368Z { 2024-08-06T20:34:40.9457797Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9458925Z "size": 187, 2024-08-06T20:34:40.9459566Z "digest": "sha256:41f16248e682693ff20b3032c1d5e5541cc87c5af898ae2ff9b24d2940e59100" 2024-08-06T20:34:40.9460283Z }, 2024-08-06T20:34:40.9460720Z { 2024-08-06T20:34:40.9461266Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9461902Z "size": 6885, 2024-08-06T20:34:40.9462545Z "digest": "sha256:95d7b781703928cf3c4eece39d800cccb76728c375fedf51ecd83833fb25e458" 2024-08-06T20:34:40.9463267Z }, 2024-08-06T20:34:40.9463609Z { 2024-08-06T20:34:40.9464144Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9464979Z "size": 1383662475, 2024-08-06T20:34:40.9465600Z "digest": "sha256:8f6c9048534734f4c873935293b7296225846ceb31c1a158400a67ea170dde7f" 2024-08-06T20:34:40.9466279Z }, 2024-08-06T20:34:40.9466615Z { 2024-08-06T20:34:40.9493287Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9493925Z "size": 62282, 2024-08-06T20:34:40.9494438Z "digest": "sha256:ab17245097e491b9368790714f9d90ed447bf0973bd677cfe6f2456d62b72a13" 2024-08-06T20:34:40.9495216Z }, 2024-08-06T20:34:40.9495485Z { 2024-08-06T20:34:40.9495904Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9496443Z "size": 1684, 2024-08-06T20:34:40.9496977Z "digest": "sha256:dfecd7e9912b76ed460b8edd5a85f1943666e38a973ab5458177cf2c7c3110e3" 2024-08-06T20:34:40.9497570Z }, 2024-08-06T20:34:40.9497837Z { 2024-08-06T20:34:40.9498737Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9499289Z "size": 1522, 2024-08-06T20:34:40.9499891Z "digest": "sha256:464a8f74544589bf7b57f9a4cadcb6681e5ed00758f6c35025e691df4e88e890" 2024-08-06T20:34:40.9500483Z }, 2024-08-06T20:34:40.9500731Z { 2024-08-06T20:34:40.9501159Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9501673Z "size": 2445884909, 2024-08-06T20:34:40.9502202Z "digest": "sha256:d67c111fa58886d8904335b5122039816c8e51af5844cb6fe4e14154d9a7c016" 2024-08-06T20:34:40.9502777Z }, 2024-08-06T20:34:40.9503023Z { 2024-08-06T20:34:40.9503447Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9503978Z "size": 85882, 2024-08-06T20:34:40.9504487Z "digest": "sha256:0723abde19c09431db80b5d0f0e003541f6232c2da5c4f35c635c861875937ae" 2024-08-06T20:34:40.9505064Z }, 2024-08-06T20:34:40.9505337Z { 2024-08-06T20:34:40.9505741Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9506262Z "size": 1820, 2024-08-06T20:34:40.9506792Z "digest": "sha256:ceceb076946ee95300822daccb5d3e4e45e6857d34996ef89305d9a078410f8e" 2024-08-06T20:34:40.9507362Z }, 2024-08-06T20:34:40.9507621Z { 2024-08-06T20:34:40.9508043Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9508551Z "size": 654860881, 2024-08-06T20:34:40.9509078Z "digest": "sha256:889eb8142c5187639e02cdc07adc75024843fd85a1f5c77d24c1a1123f5246bb" 2024-08-06T20:34:40.9509655Z }, 2024-08-06T20:34:40.9509898Z { 2024-08-06T20:34:40.9510325Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9510849Z "size": 542, 2024-08-06T20:34:40.9511358Z "digest": "sha256:af8081143f008c5cb1bd61f6d92fde22553cfc007b6a77c2fd2aaf6a9f0088a9" 2024-08-06T20:34:40.9511951Z }, 2024-08-06T20:34:40.9512193Z { 2024-08-06T20:34:40.9512829Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9513355Z "size": 1284, 2024-08-06T20:34:40.9513885Z "digest": "sha256:cc64a79090ad4059e2f3dbc1949b010b2733ab614bc9518ff6256ce2bed52115" 2024-08-06T20:34:40.9514452Z }, 2024-08-06T20:34:40.9514711Z { 2024-08-06T20:34:40.9515135Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9515645Z "size": 483, 2024-08-06T20:34:40.9516149Z "digest": "sha256:ad26542e0d01609840496a12691483f2c50eb8e23002a205becb5b695f9159fb" 2024-08-06T20:34:40.9516721Z }, 2024-08-06T20:34:40.9516965Z { 2024-08-06T20:34:40.9517390Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9517920Z "size": 91711921, 2024-08-06T20:34:40.9518437Z "digest": "sha256:7b42d8ae23671f21f6523a7e71cc50d5f87c4e993027a9bd924985cda6b57de9" 2024-08-06T20:34:40.9519097Z }, 2024-08-06T20:34:40.9519478Z + exit 0 2024-08-06T20:34:40.9519734Z { 2024-08-06T20:34:40.9520174Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9520703Z "size": 3229, 2024-08-06T20:34:40.9521361Z "digest": "sha256:6d46e3b97d9fd2bd0e7ceec44e4884756b26c594a1c1c51d1426ddd13f749db5" 2024-08-06T20:34:40.9521979Z }, 2024-08-06T20:34:40.9522226Z { 2024-08-06T20:34:40.9522653Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9523183Z "size": 1907, 2024-08-06T20:34:40.9523689Z "digest": "sha256:91f9dddffbee46981d9e74382704c1b53c87f1808350cf1dd518687805d3aa59" 2024-08-06T20:34:40.9524269Z }, 2024-08-06T20:34:40.9524533Z { 2024-08-06T20:34:40.9524943Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9525467Z "size": 699, 2024-08-06T20:34:40.9525995Z "digest": "sha256:03fb78f207e3cbcfedfc7619a1d60fa576d3293d508a7e97dc09a5e7a9c65ab0" 2024-08-06T20:34:40.9526574Z }, 2024-08-06T20:34:40.9526839Z { 2024-08-06T20:34:40.9527267Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9527797Z "size": 2813413845, 2024-08-06T20:34:40.9528358Z "digest": "sha256:eeedca475f8cafec096dfa0abc5c6f08938eb47c5d9d96fb001fd7f13a2531ca" 2024-08-06T20:34:40.9528967Z }, 2024-08-06T20:34:40.9529210Z { 2024-08-06T20:34:40.9529630Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9530152Z "size": 378, 2024-08-06T20:34:40.9530647Z "digest": "sha256:202c71391ade27762d5e9cafde793db67b4d5e4d6869debe2643d5092101c461" 2024-08-06T20:34:40.9531231Z }, 2024-08-06T20:34:40.9531491Z { 2024-08-06T20:34:40.9531891Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9532411Z "size": 12893, 2024-08-06T20:34:40.9532922Z "digest": "sha256:5af882ef27a2db3ee5e89c8730de161c0718b951f3f8477e63fe21517c9167f7" 2024-08-06T20:34:40.9533489Z }, 2024-08-06T20:34:40.9533748Z { 2024-08-06T20:34:40.9534170Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9534687Z "size": 802, 2024-08-06T20:34:40.9535208Z "digest": "sha256:bac001ac2fee5df3367ff9332bc1fcd706a8de3ad05a1885903f338197619520" 2024-08-06T20:34:40.9535799Z }, 2024-08-06T20:34:40.9536042Z { 2024-08-06T20:34:40.9536473Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9536999Z "size": 106, 2024-08-06T20:34:40.9537497Z "digest": "sha256:05896f1d52f7a8e2c4a53777e5575da3dc8fca8a786d14075b5bb204bcb4cf04" 2024-08-06T20:34:40.9538077Z }, 2024-08-06T20:34:40.9538774Z { 2024-08-06T20:34:40.9539211Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9539740Z "size": 502, 2024-08-06T20:34:40.9540237Z "digest": "sha256:49a0e418a175ac12f1b4b503e80e22679bc321e6163ed5b203af848204f0606a" 2024-08-06T20:34:40.9540819Z }, 2024-08-06T20:34:40.9541082Z { 2024-08-06T20:34:40.9541490Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9542022Z "size": 121477364, 2024-08-06T20:34:40.9542706Z "digest": "sha256:b5632121ff7a146647683888294bcca31252dd90ca681099dfb81d75c4710d52" 2024-08-06T20:34:40.9543264Z }, 2024-08-06T20:34:40.9543524Z { 2024-08-06T20:34:40.9543959Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9544467Z "size": 109, 2024-08-06T20:34:40.9544980Z "digest": "sha256:deceb46806b535404e025b9b532009eb211e79f85021171997af825253327d1a" 2024-08-06T20:34:40.9545554Z }, 2024-08-06T20:34:40.9545801Z { 2024-08-06T20:34:40.9546224Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9546750Z "size": 487, 2024-08-06T20:34:40.9547255Z "digest": "sha256:a2630e5fe166ae8ff146e4c14db0bdc1299ed2b6120108351c961adcd2375288" 2024-08-06T20:34:40.9547842Z }, 2024-08-06T20:34:40.9548105Z { 2024-08-06T20:34:40.9548512Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9549040Z "size": 297, 2024-08-06T20:34:40.9549562Z "digest": "sha256:603261fb379fff6cf0eb63de1787e7c98932f81e44e6af3f1645c9a977e0ca8c" 2024-08-06T20:34:40.9550141Z }, 2024-08-06T20:34:40.9550405Z { 2024-08-06T20:34:40.9550943Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9551467Z "size": 103, 2024-08-06T20:34:40.9551981Z "digest": "sha256:d8ac71337d62da7c4624f267380da27d834fcf174b873c8c53a963e6cc7193a4" 2024-08-06T20:34:40.9552560Z }, 2024-08-06T20:34:40.9552806Z { 2024-08-06T20:34:40.9553228Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9553751Z "size": 1470, 2024-08-06T20:34:40.9554260Z "digest": "sha256:2c2e35b979f5fc0ecccef6efc157a97f33db01a77958fa33855dc367544a5c46" 2024-08-06T20:34:40.9554853Z }, 2024-08-06T20:34:40.9555114Z { 2024-08-06T20:34:40.9555524Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9556054Z "size": 424146101, 2024-08-06T20:34:40.9556588Z "digest": "sha256:e559222c0ec62d839f78e4215ae553de028be354b7031e7393d1c6be6e91fd70" 2024-08-06T20:34:40.9557165Z }, 2024-08-06T20:34:40.9557428Z { 2024-08-06T20:34:40.9557834Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9558365Z "size": 159, 2024-08-06T20:34:40.9558886Z "digest": "sha256:c4dcb7e9d189eca2e414c757b5cf6d158055b29b0e71dbd082a5fb3585dba665" 2024-08-06T20:34:40.9559459Z }, 2024-08-06T20:34:40.9559724Z { 2024-08-06T20:34:40.9560149Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9560656Z "size": 563, 2024-08-06T20:34:40.9561169Z "digest": "sha256:2b9731d295d60278e361d491fc274e5d558a89ec5adeb9fdf342d7a8cc17bfec" 2024-08-06T20:34:40.9561755Z }, 2024-08-06T20:34:40.9562000Z { 2024-08-06T20:34:40.9562430Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9562954Z "size": 35807525, 2024-08-06T20:34:40.9563457Z "digest": "sha256:19da43312ebac2831028913760327e53d9812e80ab6b806cd7a920c549a3b554" 2024-08-06T20:34:40.9564044Z }, 2024-08-06T20:34:40.9564307Z { 2024-08-06T20:34:40.9564717Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9565243Z "size": 104, 2024-08-06T20:34:40.9565764Z "digest": "sha256:b846263d066405189d21552b7288e3efbc47e6cd2f236aa39f39d88ca576a33a" 2024-08-06T20:34:40.9566324Z }, 2024-08-06T20:34:40.9566582Z { 2024-08-06T20:34:40.9567010Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9567524Z "size": 424, 2024-08-06T20:34:40.9568047Z "digest": "sha256:3886e8f57dc887a87bb710fbda080ef4fd521f1ed2e560a895b000c05f6e9b0a" 2024-08-06T20:34:40.9568637Z }, 2024-08-06T20:34:40.9568883Z { 2024-08-06T20:34:40.9569310Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9569838Z "size": 20262048, 2024-08-06T20:34:40.9570351Z "digest": "sha256:58e13e7b6d7b78400d84805c843f08fb8f1d6583bccdb3373c8122b317e4d15a" 2024-08-06T20:34:40.9570947Z }, 2024-08-06T20:34:40.9571327Z { 2024-08-06T20:34:40.9571739Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9572265Z "size": 438, 2024-08-06T20:34:40.9572791Z "digest": "sha256:0eb80a0ac0006a45aa6b4581263cc1613ddf9545bd4a1d5a544310efe898ad81" 2024-08-06T20:34:40.9573360Z }, 2024-08-06T20:34:40.9573618Z { 2024-08-06T20:34:40.9574020Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9574543Z "size": 699, 2024-08-06T20:34:40.9575074Z "digest": "sha256:03fb78f207e3cbcfedfc7619a1d60fa576d3293d508a7e97dc09a5e7a9c65ab0" 2024-08-06T20:34:40.9575645Z }, 2024-08-06T20:34:40.9575905Z { 2024-08-06T20:34:40.9576328Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9576833Z "size": 141, 2024-08-06T20:34:40.9577349Z "digest": "sha256:a9f11b2b3622f5530c5cfefe973e96e028323fab1ff02e7a6337eb86fadebeb5" 2024-08-06T20:34:40.9577936Z }, 2024-08-06T20:34:40.9578561Z { 2024-08-06T20:34:40.9579076Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9579612Z "size": 135, 2024-08-06T20:34:40.9580255Z "digest": "sha256:d7e94279f8b2fe549e3f23637275d9dca8c831563ce84f1d44117eb297ddc871" 2024-08-06T20:34:40.9580855Z }, 2024-08-06T20:34:40.9581119Z { 2024-08-06T20:34:40.9581525Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9582049Z "size": 32, 2024-08-06T20:34:40.9582571Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-06T20:34:40.9583143Z }, 2024-08-06T20:34:40.9583409Z { 2024-08-06T20:34:40.9583845Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9584355Z "size": 190, 2024-08-06T20:34:40.9584879Z "digest": "sha256:0a3560fcd26c387cb3ff0638718fa9da02d3b08ad75b23c434b06d6a6d21cb55" 2024-08-06T20:34:40.9585474Z }, 2024-08-06T20:34:40.9585723Z { 2024-08-06T20:34:40.9586147Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9586685Z "size": 562, 2024-08-06T20:34:40.9587195Z "digest": "sha256:adf9d181cfbca5b0544f23797995d4ea6500cc4a635db9f7e18954c9044e9f4a" 2024-08-06T20:34:40.9587786Z }, 2024-08-06T20:34:40.9588055Z { 2024-08-06T20:34:40.9588468Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9588999Z "size": 43163050, 2024-08-06T20:34:40.9589527Z "digest": "sha256:bc18dd8315979255e24cf91e83757510baeb22f29a64d7579b02aca642215b58" 2024-08-06T20:34:40.9590132Z }, 2024-08-06T20:34:40.9590402Z { 2024-08-06T20:34:40.9590828Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9591337Z "size": 106, 2024-08-06T20:34:40.9591859Z "digest": "sha256:d62f2a6086195c833ef952a43eed18216c6cde79b07f8deeb4fb3aeea648f868" 2024-08-06T20:34:40.9592448Z }, 2024-08-06T20:34:40.9592695Z { 2024-08-06T20:34:40.9593123Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9593645Z "size": 1210, 2024-08-06T20:34:40.9594177Z "digest": "sha256:719a1c2abcfd7e4113aa2f03adeb54350a5e381c720688ea53bc0e18401ad33d" 2024-08-06T20:34:40.9594770Z }, 2024-08-06T20:34:40.9595017Z { 2024-08-06T20:34:40.9595448Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9596015Z "size": 699, 2024-08-06T20:34:40.9596531Z "digest": "sha256:03fb78f207e3cbcfedfc7619a1d60fa576d3293d508a7e97dc09a5e7a9c65ab0" 2024-08-06T20:34:40.9597123Z }, 2024-08-06T20:34:40.9597381Z { 2024-08-06T20:34:40.9597787Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9598307Z "size": 138, 2024-08-06T20:34:40.9598811Z "digest": "sha256:8e1355d516c5e0527c25402c664fc648a8a2b814090e6ee0fdd60f49a8826ac1" 2024-08-06T20:34:40.9599373Z }, 2024-08-06T20:34:40.9599633Z { 2024-08-06T20:34:40.9600057Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9600567Z "size": 117, 2024-08-06T20:34:40.9601209Z "digest": "sha256:701898e66f54fcf3b57d837ab651a876666391d9f9b51b9969ec842a23fd86f0" 2024-08-06T20:34:40.9601786Z }, 2024-08-06T20:34:40.9602030Z { 2024-08-06T20:34:40.9602459Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9602989Z "size": 1916643487, 2024-08-06T20:34:40.9603507Z "digest": "sha256:129b38cfed12360bb7c4d882141b5fb7f7337791383b9f75e4ffed521548f060" 2024-08-06T20:34:40.9604086Z }, 2024-08-06T20:34:40.9604346Z { 2024-08-06T20:34:40.9604753Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9605278Z "size": 172, 2024-08-06T20:34:40.9605780Z "digest": "sha256:1751651a072f518b34289de9dfa631c4e82c7958a5331061379684f9b9b11720" 2024-08-06T20:34:40.9606327Z }, 2024-08-06T20:34:40.9606584Z { 2024-08-06T20:34:40.9607010Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9607515Z "size": 905, 2024-08-06T20:34:40.9608042Z "digest": "sha256:fc956bf4e5b49119cab7cc7bda867c61bed76c76c1e113dfa7c7b184251f8173" 2024-08-06T20:34:40.9608656Z }, 2024-08-06T20:34:40.9608902Z { 2024-08-06T20:34:40.9609326Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9609940Z "size": 699, 2024-08-06T20:34:40.9610462Z "digest": "sha256:03fb78f207e3cbcfedfc7619a1d60fa576d3293d508a7e97dc09a5e7a9c65ab0" 2024-08-06T20:34:40.9611051Z }, 2024-08-06T20:34:40.9611294Z { 2024-08-06T20:34:40.9611723Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9612246Z "size": 134, 2024-08-06T20:34:40.9612741Z "digest": "sha256:874491333fb5b81d9cbbd71520163a6baf07e6f011dc148b1634b94e301972b8" 2024-08-06T20:34:40.9613326Z }, 2024-08-06T20:34:40.9613585Z { 2024-08-06T20:34:40.9613993Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9614521Z "size": 32, 2024-08-06T20:34:40.9615044Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-06T20:34:40.9615634Z }, 2024-08-06T20:34:40.9615898Z { 2024-08-06T20:34:40.9616323Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9616836Z "size": 156, 2024-08-06T20:34:40.9617378Z "digest": "sha256:365b4a0eec21fb9dfc58baf7b43a229ae3f8b638b17e9b4db5ee1cfaecb82de4" 2024-08-06T20:34:40.9617976Z }, 2024-08-06T20:34:40.9618624Z { 2024-08-06T20:34:40.9619083Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9619604Z "size": 1839, 2024-08-06T20:34:40.9620110Z "digest": "sha256:c1d4cac78640fc7d364011da807a363d6b248a2c46ccf99a6e1488c4974c17e3" 2024-08-06T20:34:40.9620696Z }, 2024-08-06T20:34:40.9620957Z { 2024-08-06T20:34:40.9621366Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9621891Z "size": 7529788, 2024-08-06T20:34:40.9622416Z "digest": "sha256:29afe2181ca6174db390fd0d9e8c4b8675fb99bb961c93936d4080ae92c8af57" 2024-08-06T20:34:40.9622997Z }, 2024-08-06T20:34:40.9623260Z { 2024-08-06T20:34:40.9623687Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9624201Z "size": 161, 2024-08-06T20:34:40.9624717Z "digest": "sha256:190dccfa687dc1e698f27965d8065a183168d7c66361505f368765456ff9f4bc" 2024-08-06T20:34:40.9625292Z }, 2024-08-06T20:34:40.9625532Z { 2024-08-06T20:34:40.9625958Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9626480Z "size": 7942, 2024-08-06T20:34:40.9626991Z "digest": "sha256:b01bea4b6634f47c51ede2b4a95ceade4e61356ca29551168b493ae6d4492f44" 2024-08-06T20:34:40.9627584Z }, 2024-08-06T20:34:40.9627829Z { 2024-08-06T20:34:40.9628253Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9628786Z "size": 8066, 2024-08-06T20:34:40.9629296Z "digest": "sha256:f13a9cc5968861df73c96e4ad67eef7f58576eeba8cdbee3d39e42ed3d6fe96d" 2024-08-06T20:34:40.9629894Z }, 2024-08-06T20:34:40.9630312Z { 2024-08-06T20:34:40.9630725Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9631256Z "size": 302, 2024-08-06T20:34:40.9631786Z "digest": "sha256:96d6dcd01b9f6c5557ba405d028442d808dd329f4f93e75f2c3ba0554e703a52" 2024-08-06T20:34:40.9632354Z }, 2024-08-06T20:34:40.9632607Z { 2024-08-06T20:34:40.9633026Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9633532Z "size": 7629762, 2024-08-06T20:34:40.9634046Z "digest": "sha256:fe134554f1b8a0509798c1f49e6f973f9b8b7f954eb1fda146f35a37c9c71599" 2024-08-06T20:34:40.9634627Z }, 2024-08-06T20:34:40.9634868Z { 2024-08-06T20:34:40.9635286Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9635806Z "size": 108, 2024-08-06T20:34:40.9636299Z "digest": "sha256:8068d364963fe87857fbb7c3aad42ddee962351a57eddc01b4282be12c69a074" 2024-08-06T20:34:40.9636885Z }, 2024-08-06T20:34:40.9637143Z { 2024-08-06T20:34:40.9637546Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9638073Z "size": 54145777, 2024-08-06T20:34:40.9638601Z "digest": "sha256:f307f545c1a5e086f18e23247c890ddd5b135b1cb7157807aeeaef38ceb3b7dc" 2024-08-06T20:34:40.9639285Z }, 2024-08-06T20:34:40.9639562Z { 2024-08-06T20:34:40.9639994Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9640505Z "size": 471, 2024-08-06T20:34:40.9641021Z "digest": "sha256:51c0b7eab50584e37e08f303a4bf3ba3141998800971452b2024a8b897858877" 2024-08-06T20:34:40.9641590Z }, 2024-08-06T20:34:40.9641832Z { 2024-08-06T20:34:40.9642262Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9642788Z "size": 1373229322, 2024-08-06T20:34:40.9643304Z "digest": "sha256:60aba96f31c695ec1e1433578783bab1257233794a6e8b5cc5c6365fdf68669e" 2024-08-06T20:34:40.9643885Z }, 2024-08-06T20:34:40.9644147Z { 2024-08-06T20:34:40.9644556Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9645092Z "size": 106, 2024-08-06T20:34:40.9645615Z "digest": "sha256:46db894a106ccd6c28198b9236ee84ac2973b37ae608caeb380dfaef91f13961" 2024-08-06T20:34:40.9646188Z }, 2024-08-06T20:34:40.9646459Z { 2024-08-06T20:34:40.9646867Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9647396Z "size": 557, 2024-08-06T20:34:40.9647893Z "digest": "sha256:0052583728e40920c255902250fe4a615b10ff00a5c0e7ed51a0075121d42fad" 2024-08-06T20:34:40.9648443Z }, 2024-08-06T20:34:40.9648705Z { 2024-08-06T20:34:40.9649127Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9649637Z "size": 36532090, 2024-08-06T20:34:40.9650177Z "digest": "sha256:8c6d352a3b51edbb546310cc3d97ded6aa8fd29a7d610e4d13fc54b5828fdd05" 2024-08-06T20:34:40.9650769Z }, 2024-08-06T20:34:40.9651013Z { 2024-08-06T20:34:40.9651435Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9651967Z "size": 111, 2024-08-06T20:34:40.9652478Z "digest": "sha256:380c4e16278f4e3bf0eb6aee3ff4a730e3a1adb82aeb1cfb8e4114f1759a2bcf" 2024-08-06T20:34:40.9653072Z }, 2024-08-06T20:34:40.9653331Z { 2024-08-06T20:34:40.9653747Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9654273Z "size": 32, 2024-08-06T20:34:40.9654793Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-06T20:34:40.9655365Z }, 2024-08-06T20:34:40.9655624Z { 2024-08-06T20:34:40.9656042Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9656556Z "size": 32, 2024-08-06T20:34:40.9657078Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-06T20:34:40.9657661Z }, 2024-08-06T20:34:40.9657904Z { 2024-08-06T20:34:40.9658762Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9659300Z "size": 32, 2024-08-06T20:34:40.9659945Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-06T20:34:40.9660535Z }, 2024-08-06T20:34:40.9660800Z { 2024-08-06T20:34:40.9661219Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-06T20:34:40.9661749Z "size": 32, 2024-08-06T20:34:40.9662271Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-06T20:34:40.9662846Z } 2024-08-06T20:34:40.9663106Z ] 2024-08-06T20:34:40.9663346Z } 2024-08-06T20:34:40.9742241Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-06T20:34:40.9742749Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-06T20:34:40.9743260Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-06T20:34:40.9751454Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:34:40.9751919Z env: 2024-08-06T20:34:40.9752215Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:34:40.9753169Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-06T20:34:40.9754158Z ##[endgroup] 2024-08-06T20:34:40.9780418Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9-02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-06T20:34:40.9840224Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-06T20:34:40.9840823Z with: 2024-08-06T20:34:40.9841680Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-06T20:34:40.9842767Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-06T20:34:40.9843268Z env: 2024-08-06T20:34:40.9843541Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:34:40.9843888Z ##[endgroup] 2024-08-06T20:34:40.9871521Z ##[group]Run set -x 2024-08-06T20:34:40.9871892Z set -x 2024-08-06T20:34:40.9872192Z set +e 2024-08-06T20:34:40.9872498Z  2024-08-06T20:34:40.9872801Z login() { 2024-08-06T20:34:40.9873386Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-06T20:34:40.9874036Z } 2024-08-06T20:34:40.9874326Z  2024-08-06T20:34:40.9874655Z retry () { 2024-08-06T20:34:40.9875007Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-06T20:34:40.9875427Z } 2024-08-06T20:34:40.9875713Z  2024-08-06T20:34:40.9876013Z retry login "${DOCKER_REGISTRY}" 2024-08-06T20:34:40.9876425Z  2024-08-06T20:34:40.9876710Z set -e 2024-08-06T20:34:40.9877135Z # ignore output since only exit code is used for conditional 2024-08-06T20:34:40.9877755Z # only pull docker image if it's not available locally 2024-08-06T20:34:40.9878438Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-06T20:34:40.9879046Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-06T20:34:40.9879476Z fi 2024-08-06T20:34:40.9886895Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:34:40.9887360Z env: 2024-08-06T20:34:40.9887656Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:34:40.9888605Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-06T20:34:40.9889685Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-06T20:34:40.9890227Z ##[endgroup] 2024-08-06T20:34:40.9913361Z + set +e 2024-08-06T20:34:40.9913926Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-06T20:34:40.9914493Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-06T20:34:40.9916695Z + aws ecr get-login-password --region us-east-1 2024-08-06T20:34:40.9918070Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-06T20:34:41.4632073Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-06T20:34:41.4633545Z Configure a credential helper to remove this warning. See 2024-08-06T20:34:41.4634765Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-06T20:34:41.4635295Z 2024-08-06T20:34:41.4635482Z Login Succeeded 2024-08-06T20:34:41.4645452Z + set -e 2024-08-06T20:34:41.4646426Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-06T20:34:41.4819973Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-06T20:34:41.4821657Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-06T20:34:41.7441511Z 02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9: Pulling from pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9 2024-08-06T20:34:41.7442604Z 7a2c55901189: Pulling fs layer 2024-08-06T20:34:41.7443167Z 09d415c238d7: Pulling fs layer 2024-08-06T20:34:41.7443528Z 9fe6e2e61518: Pulling fs layer 2024-08-06T20:34:41.7443959Z 41f16248e682: Pulling fs layer 2024-08-06T20:34:41.7444880Z 95d7b7817039: Pulling fs layer 2024-08-06T20:34:41.7445284Z 8f6c90485347: Pulling fs layer 2024-08-06T20:34:41.7445637Z 41f16248e682: Waiting 2024-08-06T20:34:41.7445953Z ab17245097e4: Pulling fs layer 2024-08-06T20:34:41.7446331Z dfecd7e9912b: Pulling fs layer 2024-08-06T20:34:41.7446702Z 464a8f745445: Pulling fs layer 2024-08-06T20:34:41.7447437Z 95d7b7817039: Waiting 2024-08-06T20:34:41.7448029Z d67c111fa588: Pulling fs layer 2024-08-06T20:34:41.7448485Z 0723abde19c0: Pulling fs layer 2024-08-06T20:34:41.7448858Z ceceb076946e: Pulling fs layer 2024-08-06T20:34:41.7449416Z 8f6c90485347: Waiting 2024-08-06T20:34:41.7449820Z 889eb8142c51: Pulling fs layer 2024-08-06T20:34:41.7450183Z dfecd7e9912b: Waiting 2024-08-06T20:34:41.7450623Z ab17245097e4: Waiting 2024-08-06T20:34:41.7451196Z af8081143f00: Pulling fs layer 2024-08-06T20:34:41.7451557Z cc64a79090ad: Pulling fs layer 2024-08-06T20:34:41.7451913Z d67c111fa588: Waiting 2024-08-06T20:34:41.7452232Z 464a8f745445: Waiting 2024-08-06T20:34:41.7452557Z ad26542e0d01: Pulling fs layer 2024-08-06T20:34:41.7452915Z ceceb076946e: Waiting 2024-08-06T20:34:41.7453252Z 7b42d8ae2367: Pulling fs layer 2024-08-06T20:34:41.7453592Z 889eb8142c51: Waiting 2024-08-06T20:34:41.7453910Z af8081143f00: Waiting 2024-08-06T20:34:41.7454241Z 6d46e3b97d9f: Pulling fs layer 2024-08-06T20:34:41.7454997Z 91f9dddffbee: Pulling fs layer 2024-08-06T20:34:41.7455365Z cc64a79090ad: Waiting 2024-08-06T20:34:41.7455703Z 03fb78f207e3: Pulling fs layer 2024-08-06T20:34:41.7456060Z eeedca475f8c: Pulling fs layer 2024-08-06T20:34:41.7456569Z 91f9dddffbee: Waiting 2024-08-06T20:34:41.7457005Z 202c71391ade: Pulling fs layer 2024-08-06T20:34:41.7457578Z 5af882ef27a2: Pulling fs layer 2024-08-06T20:34:41.7458553Z bac001ac2fee: Pulling fs layer 2024-08-06T20:34:41.7459022Z 05896f1d52f7: Pulling fs layer 2024-08-06T20:34:41.7459375Z 49a0e418a175: Pulling fs layer 2024-08-06T20:34:41.7459920Z b5632121ff7a: Pulling fs layer 2024-08-06T20:34:41.7460393Z 03fb78f207e3: Waiting 2024-08-06T20:34:41.7460714Z 6d46e3b97d9f: Waiting 2024-08-06T20:34:41.7461040Z 0723abde19c0: Waiting 2024-08-06T20:34:41.7461367Z deceb46806b5: Pulling fs layer 2024-08-06T20:34:41.7461741Z a2630e5fe166: Pulling fs layer 2024-08-06T20:34:41.7462101Z 5af882ef27a2: Waiting 2024-08-06T20:34:41.7462411Z eeedca475f8c: Waiting 2024-08-06T20:34:41.7462736Z 49a0e418a175: Waiting 2024-08-06T20:34:41.7463059Z bac001ac2fee: Waiting 2024-08-06T20:34:41.7463380Z 603261fb379f: Pulling fs layer 2024-08-06T20:34:41.7463752Z d8ac71337d62: Pulling fs layer 2024-08-06T20:34:41.7464117Z deceb46806b5: Waiting 2024-08-06T20:34:41.7464427Z a2630e5fe166: Waiting 2024-08-06T20:34:41.7464754Z 2c2e35b979f5: Pulling fs layer 2024-08-06T20:34:41.7465123Z e559222c0ec6: Pulling fs layer 2024-08-06T20:34:41.7466192Z d8ac71337d62: Waiting 2024-08-06T20:34:41.7466637Z 7b42d8ae2367: Waiting 2024-08-06T20:34:41.7466973Z c4dcb7e9d189: Pulling fs layer 2024-08-06T20:34:41.7467326Z 2b9731d295d6: Pulling fs layer 2024-08-06T20:34:41.7467677Z 202c71391ade: Waiting 2024-08-06T20:34:41.7468017Z 19da43312eba: Pulling fs layer 2024-08-06T20:34:41.7468367Z b846263d0664: Pulling fs layer 2024-08-06T20:34:41.7468724Z c4dcb7e9d189: Waiting 2024-08-06T20:34:41.7469044Z 3886e8f57dc8: Pulling fs layer 2024-08-06T20:34:41.7469417Z 58e13e7b6d7b: Pulling fs layer 2024-08-06T20:34:41.7469786Z 0eb80a0ac000: Pulling fs layer 2024-08-06T20:34:41.7470139Z a9f11b2b3622: Pulling fs layer 2024-08-06T20:34:41.7470496Z 19da43312eba: Waiting 2024-08-06T20:34:41.7470816Z 603261fb379f: Waiting 2024-08-06T20:34:41.7471163Z 58e13e7b6d7b: Waiting 2024-08-06T20:34:41.7471464Z b846263d0664: Waiting 2024-08-06T20:34:41.7471795Z d7e94279f8b2: Pulling fs layer 2024-08-06T20:34:41.7472331Z 3886e8f57dc8: Waiting 2024-08-06T20:34:41.7472706Z 4f4fb700ef54: Pulling fs layer 2024-08-06T20:34:41.7473064Z 0a3560fcd26c: Pulling fs layer 2024-08-06T20:34:41.7473677Z adf9d181cfbc: Pulling fs layer 2024-08-06T20:34:41.7474092Z bc18dd831597: Pulling fs layer 2024-08-06T20:34:41.7474437Z 4f4fb700ef54: Waiting 2024-08-06T20:34:41.7475270Z d62f2a608619: Pulling fs layer 2024-08-06T20:34:41.7475916Z 719a1c2abcfd: Pulling fs layer 2024-08-06T20:34:41.7476259Z 0a3560fcd26c: Waiting 2024-08-06T20:34:41.7476584Z adf9d181cfbc: Waiting 2024-08-06T20:34:41.7476906Z 8e1355d516c5: Pulling fs layer 2024-08-06T20:34:41.7477266Z 719a1c2abcfd: Waiting 2024-08-06T20:34:41.7477599Z 701898e66f54: Pulling fs layer 2024-08-06T20:34:41.7477951Z 129b38cfed12: Pulling fs layer 2024-08-06T20:34:41.7478308Z 8e1355d516c5: Waiting 2024-08-06T20:34:41.7478625Z 701898e66f54: Waiting 2024-08-06T20:34:41.7478938Z 1751651a072f: Pulling fs layer 2024-08-06T20:34:41.7479310Z fc956bf4e5b4: Pulling fs layer 2024-08-06T20:34:41.7479671Z e559222c0ec6: Waiting 2024-08-06T20:34:41.7479986Z 1751651a072f: Waiting 2024-08-06T20:34:41.7480318Z 874491333fb5: Pulling fs layer 2024-08-06T20:34:41.7480674Z fc956bf4e5b4: Waiting 2024-08-06T20:34:41.7480981Z d62f2a608619: Waiting 2024-08-06T20:34:41.7481315Z 365b4a0eec21: Pulling fs layer 2024-08-06T20:34:41.7481678Z 874491333fb5: Waiting 2024-08-06T20:34:41.7482353Z c1d4cac78640: Pulling fs layer 2024-08-06T20:34:41.7482757Z 29afe2181ca6: Pulling fs layer 2024-08-06T20:34:41.7483115Z a9f11b2b3622: Waiting 2024-08-06T20:34:41.7483432Z 190dccfa687d: Pulling fs layer 2024-08-06T20:34:41.7483804Z b01bea4b6634: Pulling fs layer 2024-08-06T20:34:41.7484172Z f13a9cc59688: Pulling fs layer 2024-08-06T20:34:41.7484524Z 96d6dcd01b9f: Pulling fs layer 2024-08-06T20:34:41.7484884Z 365b4a0eec21: Waiting 2024-08-06T20:34:41.7485186Z 29afe2181ca6: Waiting 2024-08-06T20:34:41.7485519Z fe134554f1b8: Pulling fs layer 2024-08-06T20:34:41.7485877Z 190dccfa687d: Waiting 2024-08-06T20:34:41.7486190Z 8068d364963f: Pulling fs layer 2024-08-06T20:34:41.7486545Z c1d4cac78640: Waiting 2024-08-06T20:34:41.7486878Z 96d6dcd01b9f: Waiting 2024-08-06T20:34:41.7487193Z f307f545c1a5: Pulling fs layer 2024-08-06T20:34:41.7487550Z fe134554f1b8: Waiting 2024-08-06T20:34:41.7487883Z 51c0b7eab505: Pulling fs layer 2024-08-06T20:34:41.7488246Z 60aba96f31c6: Pulling fs layer 2024-08-06T20:34:41.7488617Z 46db894a106c: Pulling fs layer 2024-08-06T20:34:41.7488984Z 0052583728e4: Pulling fs layer 2024-08-06T20:34:41.7489522Z 8c6d352a3b51: Pulling fs layer 2024-08-06T20:34:41.7489891Z 380c4e16278f: Pulling fs layer 2024-08-06T20:34:41.7490437Z 46db894a106c: Waiting 2024-08-06T20:34:41.7490741Z 0052583728e4: Waiting 2024-08-06T20:34:41.7491061Z 8c6d352a3b51: Waiting 2024-08-06T20:34:41.7491378Z 380c4e16278f: Waiting 2024-08-06T20:34:41.7491947Z 60aba96f31c6: Waiting 2024-08-06T20:34:41.7492275Z b01bea4b6634: Waiting 2024-08-06T20:34:41.7492581Z f13a9cc59688: Waiting 2024-08-06T20:34:41.7492906Z 51c0b7eab505: Waiting 2024-08-06T20:34:41.8767547Z 09d415c238d7: Verifying Checksum 2024-08-06T20:34:41.8768278Z 09d415c238d7: Download complete 2024-08-06T20:34:41.9368083Z 41f16248e682: Download complete 2024-08-06T20:34:42.0149612Z 95d7b7817039: Verifying Checksum 2024-08-06T20:34:42.0150074Z 95d7b7817039: Download complete 2024-08-06T20:34:42.0869611Z 7a2c55901189: Download complete 2024-08-06T20:34:42.1640174Z ab17245097e4: Download complete 2024-08-06T20:34:42.2495651Z dfecd7e9912b: Verifying Checksum 2024-08-06T20:34:42.2496132Z dfecd7e9912b: Download complete 2024-08-06T20:34:42.3445155Z 464a8f745445: Verifying Checksum 2024-08-06T20:34:42.3445993Z 464a8f745445: Download complete 2024-08-06T20:34:42.4085649Z 9fe6e2e61518: Download complete 2024-08-06T20:34:42.4891788Z 0723abde19c0: Download complete 2024-08-06T20:34:42.5523334Z ceceb076946e: Verifying Checksum 2024-08-06T20:34:42.5523820Z ceceb076946e: Download complete 2024-08-06T20:34:43.4451419Z 7a2c55901189: Pull complete 2024-08-06T20:34:43.8318988Z 09d415c238d7: Pull complete 2024-08-06T20:34:45.0020332Z 9fe6e2e61518: Pull complete 2024-08-06T20:34:45.0534452Z 41f16248e682: Pull complete 2024-08-06T20:34:45.1220510Z 95d7b7817039: Pull complete 2024-08-06T20:34:49.1587822Z 889eb8142c51: Verifying Checksum 2024-08-06T20:34:49.1588330Z 889eb8142c51: Download complete 2024-08-06T20:34:49.2685035Z af8081143f00: Download complete 2024-08-06T20:34:49.3392466Z cc64a79090ad: Verifying Checksum 2024-08-06T20:34:49.3393102Z cc64a79090ad: Download complete 2024-08-06T20:34:49.3998584Z ad26542e0d01: Download complete 2024-08-06T20:34:50.3717215Z 7b42d8ae2367: Verifying Checksum 2024-08-06T20:34:50.3717667Z 7b42d8ae2367: Download complete 2024-08-06T20:34:50.4345642Z 6d46e3b97d9f: Verifying Checksum 2024-08-06T20:34:50.4346072Z 6d46e3b97d9f: Download complete 2024-08-06T20:34:50.5193248Z 91f9dddffbee: Verifying Checksum 2024-08-06T20:34:50.5193686Z 91f9dddffbee: Download complete 2024-08-06T20:34:50.6002146Z 03fb78f207e3: Verifying Checksum 2024-08-06T20:34:50.6002557Z 03fb78f207e3: Download complete 2024-08-06T20:34:55.8961841Z 8f6c90485347: Verifying Checksum 2024-08-06T20:34:55.8962348Z 8f6c90485347: Download complete 2024-08-06T20:34:55.9712895Z 202c71391ade: Verifying Checksum 2024-08-06T20:34:55.9713474Z 202c71391ade: Download complete 2024-08-06T20:34:56.0392034Z 5af882ef27a2: Download complete 2024-08-06T20:34:56.1354297Z bac001ac2fee: Verifying Checksum 2024-08-06T20:34:56.1354894Z bac001ac2fee: Download complete 2024-08-06T20:34:56.2220968Z 05896f1d52f7: Verifying Checksum 2024-08-06T20:34:56.2221439Z 05896f1d52f7: Download complete 2024-08-06T20:34:56.3150044Z 49a0e418a175: Verifying Checksum 2024-08-06T20:34:56.3150479Z 49a0e418a175: Download complete 2024-08-06T20:34:57.5822191Z b5632121ff7a: Verifying Checksum 2024-08-06T20:34:57.5822651Z b5632121ff7a: Download complete 2024-08-06T20:34:57.6594350Z deceb46806b5: Verifying Checksum 2024-08-06T20:34:57.6595193Z deceb46806b5: Download complete 2024-08-06T20:34:57.7384564Z a2630e5fe166: Download complete 2024-08-06T20:34:57.8301289Z 603261fb379f: Verifying Checksum 2024-08-06T20:34:57.8301723Z 603261fb379f: Download complete 2024-08-06T20:34:57.8912791Z d8ac71337d62: Verifying Checksum 2024-08-06T20:34:57.8913251Z d8ac71337d62: Download complete 2024-08-06T20:34:57.9850290Z 2c2e35b979f5: Verifying Checksum 2024-08-06T20:34:57.9850747Z 2c2e35b979f5: Download complete 2024-08-06T20:35:02.2843883Z e559222c0ec6: Verifying Checksum 2024-08-06T20:35:02.2844700Z e559222c0ec6: Download complete 2024-08-06T20:35:02.3614669Z c4dcb7e9d189: Verifying Checksum 2024-08-06T20:35:02.3615521Z c4dcb7e9d189: Download complete 2024-08-06T20:35:02.4609089Z 2b9731d295d6: Verifying Checksum 2024-08-06T20:35:02.4609923Z 2b9731d295d6: Download complete 2024-08-06T20:35:02.9688131Z 19da43312eba: Verifying Checksum 2024-08-06T20:35:02.9688940Z 19da43312eba: Download complete 2024-08-06T20:35:03.0534324Z b846263d0664: Verifying Checksum 2024-08-06T20:35:03.0535168Z b846263d0664: Download complete 2024-08-06T20:35:03.1482944Z 3886e8f57dc8: Verifying Checksum 2024-08-06T20:35:03.1483758Z 3886e8f57dc8: Download complete 2024-08-06T20:35:03.4090502Z 58e13e7b6d7b: Verifying Checksum 2024-08-06T20:35:03.4091354Z 58e13e7b6d7b: Download complete 2024-08-06T20:35:03.4948897Z 0eb80a0ac000: Verifying Checksum 2024-08-06T20:35:03.4949741Z 0eb80a0ac000: Download complete 2024-08-06T20:35:03.5933768Z a9f11b2b3622: Verifying Checksum 2024-08-06T20:35:03.5934648Z a9f11b2b3622: Download complete 2024-08-06T20:35:03.6848582Z d7e94279f8b2: Verifying Checksum 2024-08-06T20:35:03.6849406Z d7e94279f8b2: Download complete 2024-08-06T20:35:03.6926818Z 4f4fb700ef54: Verifying Checksum 2024-08-06T20:35:03.6927567Z 4f4fb700ef54: Download complete 2024-08-06T20:35:03.7705950Z 0a3560fcd26c: Verifying Checksum 2024-08-06T20:35:03.7706774Z 0a3560fcd26c: Download complete 2024-08-06T20:35:03.8466650Z adf9d181cfbc: Verifying Checksum 2024-08-06T20:35:03.8467439Z adf9d181cfbc: Download complete 2024-08-06T20:35:04.3350852Z bc18dd831597: Verifying Checksum 2024-08-06T20:35:04.3351680Z bc18dd831597: Download complete 2024-08-06T20:35:04.4071277Z d62f2a608619: Verifying Checksum 2024-08-06T20:35:04.4072251Z d62f2a608619: Download complete 2024-08-06T20:35:04.4686915Z 719a1c2abcfd: Verifying Checksum 2024-08-06T20:35:04.4687728Z 719a1c2abcfd: Download complete 2024-08-06T20:35:04.5226116Z 8e1355d516c5: Download complete 2024-08-06T20:35:04.6090934Z 701898e66f54: Verifying Checksum 2024-08-06T20:35:04.6091776Z 701898e66f54: Download complete 2024-08-06T20:35:06.8622356Z d67c111fa588: Verifying Checksum 2024-08-06T20:35:06.8622857Z d67c111fa588: Download complete 2024-08-06T20:35:06.9464697Z 1751651a072f: Verifying Checksum 2024-08-06T20:35:06.9465194Z 1751651a072f: Download complete 2024-08-06T20:35:07.0133022Z fc956bf4e5b4: Download complete 2024-08-06T20:35:07.0950541Z 874491333fb5: Verifying Checksum 2024-08-06T20:35:07.0950993Z 874491333fb5: Download complete 2024-08-06T20:35:07.1798573Z 365b4a0eec21: Verifying Checksum 2024-08-06T20:35:07.1799041Z 365b4a0eec21: Download complete 2024-08-06T20:35:07.2506414Z c1d4cac78640: Verifying Checksum 2024-08-06T20:35:07.2506902Z c1d4cac78640: Download complete 2024-08-06T20:35:07.3650009Z 29afe2181ca6: Verifying Checksum 2024-08-06T20:35:07.3650481Z 29afe2181ca6: Download complete 2024-08-06T20:35:07.4352530Z 190dccfa687d: Download complete 2024-08-06T20:35:07.5103026Z b01bea4b6634: Verifying Checksum 2024-08-06T20:35:07.5103571Z b01bea4b6634: Download complete 2024-08-06T20:35:07.5812711Z f13a9cc59688: Verifying Checksum 2024-08-06T20:35:07.5813192Z f13a9cc59688: Download complete 2024-08-06T20:35:07.6533020Z 96d6dcd01b9f: Verifying Checksum 2024-08-06T20:35:07.6533865Z 96d6dcd01b9f: Download complete 2024-08-06T20:35:07.8926116Z fe134554f1b8: Verifying Checksum 2024-08-06T20:35:07.8926598Z fe134554f1b8: Download complete 2024-08-06T20:35:07.9640614Z 8068d364963f: Verifying Checksum 2024-08-06T20:35:07.9641057Z 8068d364963f: Download complete 2024-08-06T20:35:08.6048226Z f307f545c1a5: Verifying Checksum 2024-08-06T20:35:08.6048723Z f307f545c1a5: Download complete 2024-08-06T20:35:08.6822139Z 51c0b7eab505: Verifying Checksum 2024-08-06T20:35:08.6822672Z 51c0b7eab505: Download complete 2024-08-06T20:35:14.4988265Z 8f6c90485347: Pull complete 2024-08-06T20:35:14.5606048Z ab17245097e4: Pull complete 2024-08-06T20:35:14.6337332Z dfecd7e9912b: Pull complete 2024-08-06T20:35:14.6986245Z 464a8f745445: Pull complete 2024-08-06T20:35:18.7905655Z eeedca475f8c: Verifying Checksum 2024-08-06T20:35:18.7906132Z eeedca475f8c: Download complete 2024-08-06T20:35:18.8788753Z 46db894a106c: Verifying Checksum 2024-08-06T20:35:18.8789517Z 46db894a106c: Download complete 2024-08-06T20:35:18.9565171Z 0052583728e4: Verifying Checksum 2024-08-06T20:35:18.9565606Z 0052583728e4: Download complete 2024-08-06T20:35:19.3744826Z 8c6d352a3b51: Verifying Checksum 2024-08-06T20:35:19.3745269Z 8c6d352a3b51: Download complete 2024-08-06T20:35:19.4422165Z 380c4e16278f: Verifying Checksum 2024-08-06T20:35:19.4422603Z 380c4e16278f: Download complete 2024-08-06T20:35:22.4688745Z 60aba96f31c6: Verifying Checksum 2024-08-06T20:35:22.4689203Z 60aba96f31c6: Download complete 2024-08-06T20:35:23.8293483Z 129b38cfed12: Verifying Checksum 2024-08-06T20:35:23.8293970Z 129b38cfed12: Download complete 2024-08-06T20:35:53.4496749Z d67c111fa588: Pull complete 2024-08-06T20:35:55.8923180Z 0723abde19c0: Pull complete 2024-08-06T20:35:58.5582485Z ceceb076946e: Pull complete 2024-08-06T20:36:14.9009223Z 889eb8142c51: Pull complete 2024-08-06T20:36:17.3429361Z af8081143f00: Pull complete 2024-08-06T20:36:19.7834260Z cc64a79090ad: Pull complete 2024-08-06T20:36:22.2269767Z ad26542e0d01: Pull complete 2024-08-06T20:36:27.9937209Z 7b42d8ae2367: Pull complete 2024-08-06T20:36:30.4049329Z 6d46e3b97d9f: Pull complete 2024-08-06T20:36:32.8166386Z 91f9dddffbee: Pull complete 2024-08-06T20:36:35.2100426Z 03fb78f207e3: Pull complete 2024-08-06T20:37:40.9732199Z eeedca475f8c: Pull complete 2024-08-06T20:37:45.2446300Z 202c71391ade: Pull complete 2024-08-06T20:37:49.6146151Z 5af882ef27a2: Pull complete 2024-08-06T20:37:52.5707193Z bac001ac2fee: Pull complete 2024-08-06T20:37:55.3586057Z 05896f1d52f7: Pull complete 2024-08-06T20:37:58.2334257Z 49a0e418a175: Pull complete 2024-08-06T20:38:03.2764139Z b5632121ff7a: Pull complete 2024-08-06T20:38:03.3280523Z deceb46806b5: Pull complete 2024-08-06T20:38:03.3681770Z a2630e5fe166: Pull complete 2024-08-06T20:38:03.4091019Z 603261fb379f: Pull complete 2024-08-06T20:38:03.4530800Z d8ac71337d62: Pull complete 2024-08-06T20:38:03.4998906Z 2c2e35b979f5: Pull complete 2024-08-06T20:38:15.3267903Z e559222c0ec6: Pull complete 2024-08-06T20:38:15.3754593Z c4dcb7e9d189: Pull complete 2024-08-06T20:38:15.4199550Z 2b9731d295d6: Pull complete 2024-08-06T20:38:16.3745675Z 19da43312eba: Pull complete 2024-08-06T20:38:16.4226475Z b846263d0664: Pull complete 2024-08-06T20:38:16.4720982Z 3886e8f57dc8: Pull complete 2024-08-06T20:38:16.8490371Z 58e13e7b6d7b: Pull complete 2024-08-06T20:38:16.8878932Z 0eb80a0ac000: Pull complete 2024-08-06T20:38:17.0213161Z a9f11b2b3622: Pull complete 2024-08-06T20:38:17.0688313Z d7e94279f8b2: Pull complete 2024-08-06T20:38:17.1104704Z 4f4fb700ef54: Pull complete 2024-08-06T20:38:17.1521748Z 0a3560fcd26c: Pull complete 2024-08-06T20:38:17.1918066Z adf9d181cfbc: Pull complete 2024-08-06T20:38:19.6865222Z bc18dd831597: Pull complete 2024-08-06T20:38:19.7405805Z d62f2a608619: Pull complete 2024-08-06T20:38:19.7721791Z 719a1c2abcfd: Pull complete 2024-08-06T20:38:19.8919905Z 8e1355d516c5: Pull complete 2024-08-06T20:38:19.9376081Z 701898e66f54: Pull complete 2024-08-06T20:39:05.0689114Z 129b38cfed12: Pull complete 2024-08-06T20:39:07.5041650Z 1751651a072f: Pull complete 2024-08-06T20:39:09.8831674Z fc956bf4e5b4: Pull complete 2024-08-06T20:39:14.7264585Z 874491333fb5: Pull complete 2024-08-06T20:39:19.6354786Z 365b4a0eec21: Pull complete 2024-08-06T20:39:22.0473331Z c1d4cac78640: Pull complete 2024-08-06T20:39:24.8003503Z 29afe2181ca6: Pull complete 2024-08-06T20:39:27.8452789Z 190dccfa687d: Pull complete 2024-08-06T20:39:30.2994920Z b01bea4b6634: Pull complete 2024-08-06T20:39:33.0128112Z f13a9cc59688: Pull complete 2024-08-06T20:39:35.7541552Z 96d6dcd01b9f: Pull complete 2024-08-06T20:39:38.1551199Z fe134554f1b8: Pull complete 2024-08-06T20:39:38.1954518Z 8068d364963f: Pull complete 2024-08-06T20:39:40.5092003Z f307f545c1a5: Pull complete 2024-08-06T20:39:40.5564239Z 51c0b7eab505: Pull complete 2024-08-06T20:39:57.4315174Z 60aba96f31c6: Pull complete 2024-08-06T20:39:57.4960026Z 46db894a106c: Pull complete 2024-08-06T20:39:57.5368518Z 0052583728e4: Pull complete 2024-08-06T20:39:58.1861514Z 8c6d352a3b51: Pull complete 2024-08-06T20:39:58.2244006Z 380c4e16278f: Pull complete 2024-08-06T20:39:58.5004100Z Digest: sha256:66181a134b3ebc35388dabcc46dc11e767152fd72514dad76711590f50b650f7 2024-08-06T20:39:58.5038924Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-06T20:39:58.5070050Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-06T20:39:58.5121089Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-06T20:39:58.5122204Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-06T20:39:58.5130594Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:39:58.5131082Z env: 2024-08-06T20:39:58.5131362Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:39:58.5131715Z ##[endgroup] 2024-08-06T20:39:58.5398039Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-08-06T20:39:58.5398607Z with: 2024-08-06T20:39:58.5398890Z driver-version: 550.54.15 2024-08-06T20:39:58.5399240Z env: 2024-08-06T20:39:58.5399532Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:39:58.5399856Z ##[endgroup] 2024-08-06T20:39:58.5451024Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-06T20:39:58.5451555Z with: 2024-08-06T20:39:58.5451850Z timeout_minutes: 10 2024-08-06T20:39:58.5452166Z max_attempts: 3 2024-08-06T20:39:58.5482900Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-08-06T20:39:58.5513636Z retry_wait_seconds: 10 2024-08-06T20:39:58.5514017Z polling_interval_seconds: 1 2024-08-06T20:39:58.5514384Z warning_on_retry: true 2024-08-06T20:39:58.5514741Z continue_on_error: false 2024-08-06T20:39:58.5515088Z env: 2024-08-06T20:39:58.5515364Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:39:58.5515718Z DRIVER_VERSION: 550.54.15 2024-08-06T20:39:58.5516050Z ##[endgroup] 2024-08-06T20:39:58.6287787Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-08-06T20:39:58.6290137Z + pre_install_nvidia_driver_amzn2 2024-08-06T20:39:58.6292350Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-08-06T20:39:59.1373709Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-08-06T20:39:59.1987523Z No Match for argument: nvidia-driver-latest-dkms 2024-08-06T20:39:59.2376367Z No Packages marked for removal 2024-08-06T20:39:59.2540073Z + install_nvidia_driver_common 2024-08-06T20:39:59.2543657Z + echo 'Before installing NVIDIA driver' 2024-08-06T20:39:59.2544443Z + lspci 2024-08-06T20:39:59.2546321Z Before installing NVIDIA driver 2024-08-06T20:39:59.2723349Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-08-06T20:39:59.2724052Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-06T20:39:59.2724744Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-08-06T20:39:59.2725408Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-08-06T20:39:59.2725966Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-08-06T20:39:59.2726903Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-06T20:39:59.2727643Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-06T20:39:59.2728384Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-06T20:39:59.2729122Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-08-06T20:39:59.2729826Z + lsmod 2024-08-06T20:39:59.2747656Z Module Size Used by 2024-08-06T20:39:59.2748080Z ib_core 266240 0 2024-08-06T20:39:59.2748444Z veth 16384 0 2024-08-06T20:39:59.2748809Z nvidia_modeset 1339392 0 2024-08-06T20:39:59.2749154Z nvidia_uvm 4599808 0 2024-08-06T20:39:59.2749578Z nvidia 53989376 11 nvidia_uvm,nvidia_modeset 2024-08-06T20:39:59.2750039Z drm 421888 1 nvidia 2024-08-06T20:39:59.2750537Z i2c_core 77824 2 nvidia,drm 2024-08-06T20:39:59.2750995Z backlight 16384 1 nvidia_modeset 2024-08-06T20:39:59.2751424Z xt_conntrack 16384 1 2024-08-06T20:39:59.2751782Z ipt_MASQUERADE 16384 1 2024-08-06T20:39:59.2752192Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-08-06T20:39:59.2752643Z nf_conntrack_netlink 49152 0 2024-08-06T20:39:59.2753056Z nfnetlink 16384 2 nf_conntrack_netlink 2024-08-06T20:39:59.2753513Z xfrm_user 45056 1 2024-08-06T20:39:59.2753895Z xfrm_algo 16384 1 xfrm_user 2024-08-06T20:39:59.2754278Z iptable_nat 16384 1 2024-08-06T20:39:59.2754645Z nf_conntrack_ipv4 16384 3 2024-08-06T20:39:59.2755056Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-08-06T20:39:59.2755495Z nf_nat_ipv4 16384 1 iptable_nat 2024-08-06T20:39:59.2756024Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-08-06T20:39:59.2756883Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-08-06T20:39:59.2757684Z xt_addrtype 16384 2 2024-08-06T20:39:59.2758057Z iptable_filter 16384 1 2024-08-06T20:39:59.2758429Z br_netfilter 24576 0 2024-08-06T20:39:59.2758799Z bridge 172032 1 br_netfilter 2024-08-06T20:39:59.2759216Z stp 16384 1 bridge 2024-08-06T20:39:59.2759620Z llc 16384 2 bridge,stp 2024-08-06T20:39:59.2760004Z overlay 86016 0 2024-08-06T20:39:59.2760360Z sunrpc 393216 1 2024-08-06T20:39:59.2760718Z dm_mirror 28672 0 2024-08-06T20:39:59.2761076Z dm_region_hash 20480 1 dm_mirror 2024-08-06T20:39:59.2761524Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-08-06T20:39:59.2761986Z dm_mod 143360 2 dm_log,dm_mirror 2024-08-06T20:39:59.2762385Z dax 69632 1 dm_mod 2024-08-06T20:39:59.2762771Z sb_edac 24576 0 2024-08-06T20:39:59.2763128Z crc32_pclmul 16384 0 2024-08-06T20:39:59.2763489Z ghash_clmulni_intel 16384 0 2024-08-06T20:39:59.2763860Z pcbc 16384 0 2024-08-06T20:39:59.2764307Z aesni_intel 188416 0 2024-08-06T20:39:59.2764668Z aes_x86_64 20480 1 aesni_intel 2024-08-06T20:39:59.2765090Z crypto_simd 16384 1 aesni_intel 2024-08-06T20:39:59.2765546Z glue_helper 16384 1 aesni_intel 2024-08-06T20:39:59.2766049Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-08-06T20:39:59.2766564Z mousedev 24576 0 2024-08-06T20:39:59.2766924Z evdev 20480 3 2024-08-06T20:39:59.2767266Z psmouse 32768 0 2024-08-06T20:39:59.2767670Z pcc_cpufreq 16384 0 2024-08-06T20:39:59.2768032Z button 16384 0 2024-08-06T20:39:59.2768368Z ena 139264 0 2024-08-06T20:39:59.2768796Z ptp 20480 1 ena 2024-08-06T20:39:59.2769179Z pps_core 20480 1 ptp 2024-08-06T20:39:59.2769725Z ata_piix 36864 0 2024-08-06T20:39:59.2770101Z libata 266240 1 ata_piix 2024-08-06T20:39:59.2770500Z xen_blkfront 49152 2 2024-08-06T20:39:59.2770844Z crc32c_intel 24576 0 2024-08-06T20:39:59.2771212Z scsi_mod 245760 1 libata 2024-08-06T20:39:59.2771597Z autofs4 49152 2 2024-08-06T20:39:59.2771932Z + modinfo nvidia 2024-08-06T20:39:59.2772572Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-08-06T20:39:59.2773197Z alias: char-major-195-* 2024-08-06T20:39:59.2773558Z version: 550.54.15 2024-08-06T20:39:59.2773946Z supported: external 2024-08-06T20:39:59.2774293Z license: NVIDIA 2024-08-06T20:39:59.2774649Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-06T20:39:59.2775117Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-06T20:39:59.2775559Z srcversion: 833721318DA517F0C2FEC97 2024-08-06T20:39:59.2775985Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-06T20:39:59.2776460Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-06T20:39:59.2776922Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-06T20:39:59.2777336Z depends: i2c-core,drm 2024-08-06T20:39:59.2777701Z retpoline: Y 2024-08-06T20:39:59.2778016Z name: nvidia 2024-08-06T20:39:59.2778804Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-08-06T20:39:59.2779403Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-06T20:39:59.2780025Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-06T20:39:59.2780582Z parm: NVreg_ResmanDebugLevel:int 2024-08-06T20:39:59.2781179Z parm: NVreg_RmLogonRC:int 2024-08-06T20:39:59.2781608Z parm: NVreg_ModifyDeviceFiles:int 2024-08-06T20:39:59.2782034Z parm: NVreg_DeviceFileUID:int 2024-08-06T20:39:59.2782466Z parm: NVreg_DeviceFileGID:int 2024-08-06T20:39:59.2783012Z parm: NVreg_DeviceFileMode:int 2024-08-06T20:39:59.2783518Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-06T20:39:59.2784033Z parm: NVreg_UsePageAttributeTable:int 2024-08-06T20:39:59.2784497Z parm: NVreg_EnablePCIeGen3:int 2024-08-06T20:39:59.2784919Z parm: NVreg_EnableMSI:int 2024-08-06T20:39:59.2785316Z parm: NVreg_TCEBypassMode:int 2024-08-06T20:39:59.2785767Z parm: NVreg_EnableStreamMemOPs:int 2024-08-06T20:39:59.2786266Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-06T20:39:59.2786789Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-06T20:39:59.2787320Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-06T20:39:59.2787883Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-06T20:39:59.2788424Z parm: NVreg_DynamicPowerManagement:int 2024-08-06T20:39:59.2788995Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-06T20:39:59.2789565Z parm: NVreg_EnableGpuFirmware:int 2024-08-06T20:39:59.2790016Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-06T20:39:59.2790524Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-06T20:39:59.2791036Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-06T20:39:59.2791493Z parm: NVreg_MemoryPoolSize:int 2024-08-06T20:39:59.2791948Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-06T20:39:59.2792409Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-06T20:39:59.2792847Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-06T20:39:59.2793285Z parm: NVreg_NvLinkDisable:int 2024-08-06T20:39:59.2793768Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-06T20:39:59.2794242Z parm: NVreg_RegisterPCIDriver:int 2024-08-06T20:39:59.2794696Z parm: NVreg_EnableResizableBar:int 2024-08-06T20:39:59.2795159Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-06T20:39:59.2795613Z parm: NVreg_EnableNonblockingOpen:int 2024-08-06T20:39:59.2796254Z parm: NVreg_RegistryDwords:charp 2024-08-06T20:39:59.2796724Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-06T20:39:59.2797161Z parm: NVreg_RmMsg:charp 2024-08-06T20:39:59.2797560Z parm: NVreg_GpuBlacklist:charp 2024-08-06T20:39:59.2798007Z parm: NVreg_TemporaryFilePath:charp 2024-08-06T20:39:59.2798667Z parm: NVreg_ExcludedGpus:charp 2024-08-06T20:39:59.2799124Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-06T20:39:59.2799583Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-06T20:39:59.2800023Z parm: NVreg_ImexChannelCount:int 2024-08-06T20:39:59.2800457Z parm: rm_firmware_active:charp 2024-08-06T20:39:59.2800859Z + HAS_NVIDIA_DRIVER=0 2024-08-06T20:39:59.2801239Z ++ command -v nvidia-smi 2024-08-06T20:39:59.2801612Z + '[' -x /usr/bin/nvidia-smi ']' 2024-08-06T20:39:59.2802058Z + set +e 2024-08-06T20:39:59.2802467Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-08-06T20:39:59.3028038Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-08-06T20:39:59.3028464Z + NVIDIA_SMI_STATUS=0 2024-08-06T20:39:59.3028778Z + '[' 0 -ne 0 ']' 2024-08-06T20:39:59.3029091Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-08-06T20:39:59.3029458Z + HAS_NVIDIA_DRIVER=1 2024-08-06T20:39:59.3030083Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-08-06T20:39:59.3030726Z + set -e 2024-08-06T20:39:59.3031012Z + '[' 1 -eq 0 ']' 2024-08-06T20:39:59.3031522Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-08-06T20:39:59.3032154Z + post_install_nvidia_driver_common 2024-08-06T20:39:59.3035078Z + sudo modprobe nvidia 2024-08-06T20:39:59.3151715Z + echo 'After installing NVIDIA driver' 2024-08-06T20:39:59.3152150Z + lspci 2024-08-06T20:39:59.3153200Z After installing NVIDIA driver 2024-08-06T20:39:59.3341745Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-08-06T20:39:59.3342472Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-06T20:39:59.3343163Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-08-06T20:39:59.3343807Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-08-06T20:39:59.3344374Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-08-06T20:39:59.3345002Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-06T20:39:59.3345721Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-06T20:39:59.3346477Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-06T20:39:59.3347210Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-08-06T20:39:59.3347750Z + lsmod 2024-08-06T20:39:59.3364299Z Module Size Used by 2024-08-06T20:39:59.3364728Z ib_core 266240 0 2024-08-06T20:39:59.3365107Z veth 16384 0 2024-08-06T20:39:59.3365457Z nvidia_modeset 1339392 0 2024-08-06T20:39:59.3365821Z nvidia_uvm 4599808 0 2024-08-06T20:39:59.3366250Z nvidia 53989376 11 nvidia_uvm,nvidia_modeset 2024-08-06T20:39:59.3366690Z drm 421888 1 nvidia 2024-08-06T20:39:59.3367093Z i2c_core 77824 2 nvidia,drm 2024-08-06T20:39:59.3367540Z backlight 16384 1 nvidia_modeset 2024-08-06T20:39:59.3367956Z xt_conntrack 16384 1 2024-08-06T20:39:59.3368332Z ipt_MASQUERADE 16384 1 2024-08-06T20:39:59.3368748Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-08-06T20:39:59.3369186Z nf_conntrack_netlink 49152 0 2024-08-06T20:39:59.3369617Z nfnetlink 16384 2 nf_conntrack_netlink 2024-08-06T20:39:59.3370055Z xfrm_user 45056 1 2024-08-06T20:39:59.3370413Z xfrm_algo 16384 1 xfrm_user 2024-08-06T20:39:59.3370815Z iptable_nat 16384 1 2024-08-06T20:39:59.3371433Z nf_conntrack_ipv4 16384 3 2024-08-06T20:39:59.3371828Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-08-06T20:39:59.3372288Z nf_nat_ipv4 16384 1 iptable_nat 2024-08-06T20:39:59.3372767Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-08-06T20:39:59.3373754Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-08-06T20:39:59.3374580Z xt_addrtype 16384 2 2024-08-06T20:39:59.3374956Z iptable_filter 16384 1 2024-08-06T20:39:59.3375312Z br_netfilter 24576 0 2024-08-06T20:39:59.3375701Z bridge 172032 1 br_netfilter 2024-08-06T20:39:59.3376115Z stp 16384 1 bridge 2024-08-06T20:39:59.3376494Z llc 16384 2 bridge,stp 2024-08-06T20:39:59.3376888Z overlay 86016 0 2024-08-06T20:39:59.3377242Z sunrpc 393216 1 2024-08-06T20:39:59.3377581Z dm_mirror 28672 0 2024-08-06T20:39:59.3377968Z dm_region_hash 20480 1 dm_mirror 2024-08-06T20:39:59.3378779Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-08-06T20:39:59.3379224Z dm_mod 143360 2 dm_log,dm_mirror 2024-08-06T20:39:59.3379643Z dax 69632 1 dm_mod 2024-08-06T20:39:59.3380024Z sb_edac 24576 0 2024-08-06T20:39:59.3380364Z crc32_pclmul 16384 0 2024-08-06T20:39:59.3380738Z ghash_clmulni_intel 16384 0 2024-08-06T20:39:59.3381107Z pcbc 16384 0 2024-08-06T20:39:59.3381446Z aesni_intel 188416 0 2024-08-06T20:39:59.3381821Z aes_x86_64 20480 1 aesni_intel 2024-08-06T20:39:59.3382242Z crypto_simd 16384 1 aesni_intel 2024-08-06T20:39:59.3382646Z glue_helper 16384 1 aesni_intel 2024-08-06T20:39:59.3383160Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-08-06T20:39:59.3383667Z mousedev 24576 0 2024-08-06T20:39:59.3384017Z evdev 20480 3 2024-08-06T20:39:59.3384372Z psmouse 32768 0 2024-08-06T20:39:59.3384716Z pcc_cpufreq 16384 0 2024-08-06T20:39:59.3385081Z button 16384 0 2024-08-06T20:39:59.3385433Z ena 139264 0 2024-08-06T20:39:59.3385769Z ptp 20480 1 ena 2024-08-06T20:39:59.3386152Z pps_core 20480 1 ptp 2024-08-06T20:39:59.3386539Z ata_piix 36864 0 2024-08-06T20:39:59.3386894Z libata 266240 1 ata_piix 2024-08-06T20:39:59.3387297Z xen_blkfront 49152 2 2024-08-06T20:39:59.3387660Z crc32c_intel 24576 0 2024-08-06T20:39:59.3388008Z scsi_mod 245760 1 libata 2024-08-06T20:39:59.3388394Z autofs4 49152 2 2024-08-06T20:39:59.3388798Z + modinfo nvidia 2024-08-06T20:39:59.3389483Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-08-06T20:39:59.3390066Z alias: char-major-195-* 2024-08-06T20:39:59.3390455Z version: 550.54.15 2024-08-06T20:39:59.3390915Z supported: external 2024-08-06T20:39:59.3391244Z license: NVIDIA 2024-08-06T20:39:59.3391623Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-06T20:39:59.3392069Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-06T20:39:59.3392508Z srcversion: 833721318DA517F0C2FEC97 2024-08-06T20:39:59.3392954Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-06T20:39:59.3393403Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-06T20:39:59.3393868Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-06T20:39:59.3394306Z depends: i2c-core,drm 2024-08-06T20:39:59.3394651Z retpoline: Y 2024-08-06T20:39:59.3394963Z name: nvidia 2024-08-06T20:39:59.3395412Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-08-06T20:39:59.3395984Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-06T20:39:59.3396588Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-06T20:39:59.3397339Z parm: NVreg_ResmanDebugLevel:int 2024-08-06T20:39:59.3397749Z parm: NVreg_RmLogonRC:int 2024-08-06T20:39:59.3398173Z parm: NVreg_ModifyDeviceFiles:int 2024-08-06T20:39:59.3398608Z parm: NVreg_DeviceFileUID:int 2024-08-06T20:39:59.3399013Z parm: NVreg_DeviceFileGID:int 2024-08-06T20:39:59.3399538Z parm: NVreg_DeviceFileMode:int 2024-08-06T20:39:59.3400051Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-06T20:39:59.3400561Z parm: NVreg_UsePageAttributeTable:int 2024-08-06T20:39:59.3401024Z parm: NVreg_EnablePCIeGen3:int 2024-08-06T20:39:59.3401443Z parm: NVreg_EnableMSI:int 2024-08-06T20:39:59.3401830Z parm: NVreg_TCEBypassMode:int 2024-08-06T20:39:59.3402274Z parm: NVreg_EnableStreamMemOPs:int 2024-08-06T20:39:59.3402773Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-06T20:39:59.3403292Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-06T20:39:59.3403822Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-06T20:39:59.3404376Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-06T20:39:59.3404915Z parm: NVreg_DynamicPowerManagement:int 2024-08-06T20:39:59.3405481Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-06T20:39:59.3406046Z parm: NVreg_EnableGpuFirmware:int 2024-08-06T20:39:59.3406496Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-06T20:39:59.3406999Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-06T20:39:59.3407511Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-06T20:39:59.3407966Z parm: NVreg_MemoryPoolSize:int 2024-08-06T20:39:59.3408411Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-06T20:39:59.3408877Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-06T20:39:59.3409312Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-06T20:39:59.3409757Z parm: NVreg_NvLinkDisable:int 2024-08-06T20:39:59.3410233Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-06T20:39:59.3410716Z parm: NVreg_RegisterPCIDriver:int 2024-08-06T20:39:59.3411173Z parm: NVreg_EnableResizableBar:int 2024-08-06T20:39:59.3411635Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-06T20:39:59.3412096Z parm: NVreg_EnableNonblockingOpen:int 2024-08-06T20:39:59.3412563Z parm: NVreg_RegistryDwords:charp 2024-08-06T20:39:59.3413034Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-06T20:39:59.3413476Z parm: NVreg_RmMsg:charp 2024-08-06T20:39:59.3413877Z parm: NVreg_GpuBlacklist:charp 2024-08-06T20:39:59.3414328Z parm: NVreg_TemporaryFilePath:charp 2024-08-06T20:39:59.3414767Z parm: NVreg_ExcludedGpus:charp 2024-08-06T20:39:59.3415205Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-06T20:39:59.3415662Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-06T20:39:59.3416115Z parm: NVreg_ImexChannelCount:int 2024-08-06T20:39:59.3416549Z parm: rm_firmware_active:charp 2024-08-06T20:39:59.3416946Z + set +e 2024-08-06T20:39:59.3417234Z + nvidia-smi 2024-08-06T20:39:59.3629762Z Tue Aug 6 20:39:59 2024 2024-08-06T20:39:59.3630273Z +-----------------------------------------------------------------------------------------+ 2024-08-06T20:39:59.3630965Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-06T20:39:59.3631611Z |-----------------------------------------+------------------------+----------------------+ 2024-08-06T20:39:59.3632302Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-06T20:39:59.3633020Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-06T20:39:59.3633603Z | | | MIG M. | 2024-08-06T20:39:59.3634264Z |=========================================+========================+======================| 2024-08-06T20:39:59.3715827Z | 0 Tesla M60 On | 00000000:00:1D.0 Off | 0 | 2024-08-06T20:39:59.3716433Z | N/A 37C P8 14W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-06T20:39:59.3717109Z | | | N/A | 2024-08-06T20:39:59.3717680Z +-----------------------------------------+------------------------+----------------------+ 2024-08-06T20:39:59.3780405Z | 1 Tesla M60 On | 00000000:00:1E.0 Off | 0 | 2024-08-06T20:39:59.3781016Z | N/A 34C P8 15W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-06T20:39:59.3781526Z | | | N/A | 2024-08-06T20:39:59.3782084Z +-----------------------------------------+------------------------+----------------------+ 2024-08-06T20:39:59.3782639Z 2024-08-06T20:39:59.3783159Z +-----------------------------------------------------------------------------------------+ 2024-08-06T20:39:59.3783745Z | Processes: | 2024-08-06T20:39:59.3784372Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-06T20:39:59.3784938Z | ID ID Usage | 2024-08-06T20:39:59.3785422Z |=========================================================================================| 2024-08-06T20:39:59.3786927Z | No running processes found | 2024-08-06T20:39:59.3788328Z +-----------------------------------------------------------------------------------------+ 2024-08-06T20:39:59.4883662Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-08-06T20:39:59.5065042Z Tesla M60 2024-08-06T20:39:59.5128786Z + NVIDIA_SMI_STATUS=0 2024-08-06T20:39:59.5129483Z + '[' 0 -eq 0 ']' 2024-08-06T20:39:59.5130176Z + echo 'INFO: Ignoring allowed status 0' 2024-08-06T20:39:59.5131224Z + set -e 2024-08-06T20:39:59.5131874Z INFO: Ignoring allowed status 0 2024-08-06T20:39:59.5136517Z == Installing nvidia container toolkit for amzn2 == 2024-08-06T20:39:59.5142690Z + sudo yum install -y yum-utils 2024-08-06T20:40:00.0034188Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-08-06T20:40:01.7557887Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-08-06T20:40:01.7558534Z Nothing to do 2024-08-06T20:40:01.9064551Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]] 2024-08-06T20:40:01.9065719Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-08-06T20:40:01.9067556Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-08-06T20:40:02.4151931Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-08-06T20:40:02.4470531Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-08-06T20:40:02.4471534Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-08-06T20:40:02.4472427Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-08-06T20:40:02.4619205Z + sudo yum install -y nvidia-docker2 2024-08-06T20:40:02.9535682Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-08-06T20:40:04.5825116Z Package nvidia-docker2-2.14.0-1.noarch already installed and latest version 2024-08-06T20:40:04.5825733Z Nothing to do 2024-08-06T20:40:04.7254481Z + sudo systemctl restart docker 2024-08-06T20:40:23.8415549Z nvidia-persistenced failed to initialize. Check syslog for more details. 2024-08-06T20:40:23.8870321Z Tue Aug 6 20:40:23 2024 2024-08-06T20:40:23.8871160Z +-----------------------------------------------------------------------------------------+ 2024-08-06T20:40:23.8871841Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-06T20:40:23.8872503Z |-----------------------------------------+------------------------+----------------------+ 2024-08-06T20:40:23.8873327Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-06T20:40:23.8874072Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-06T20:40:23.8874662Z | | | MIG M. | 2024-08-06T20:40:23.8875125Z |=========================================+========================+======================| 2024-08-06T20:40:23.8956271Z | 0 Tesla M60 On | 00000000:00:1D.0 Off | 0 | 2024-08-06T20:40:23.8956875Z | N/A 37C P8 12W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-06T20:40:23.8957422Z | | | N/A | 2024-08-06T20:40:23.8957949Z +-----------------------------------------+------------------------+----------------------+ 2024-08-06T20:40:23.9023872Z | 1 Tesla M60 On | 00000000:00:1E.0 Off | 0 | 2024-08-06T20:40:23.9024542Z | N/A 34C P8 14W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-06T20:40:23.9025067Z | | | N/A | 2024-08-06T20:40:23.9025615Z +-----------------------------------------+------------------------+----------------------+ 2024-08-06T20:40:23.9026153Z 2024-08-06T20:40:23.9026687Z +-----------------------------------------------------------------------------------------+ 2024-08-06T20:40:23.9027273Z | Processes: | 2024-08-06T20:40:23.9027881Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-06T20:40:23.9028464Z | ID ID Usage | 2024-08-06T20:40:23.9028933Z |=========================================================================================| 2024-08-06T20:40:23.9029612Z | No running processes found | 2024-08-06T20:40:23.9030256Z +-----------------------------------------------------------------------------------------+ 2024-08-06T20:40:24.6543403Z Command completed after 1 attempt(s). 2024-08-06T20:40:24.6614434Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-06T20:40:24.6615180Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-06T20:40:24.6615843Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-06T20:40:24.6616452Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-06T20:40:24.6624835Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:40:24.6625331Z env: 2024-08-06T20:40:24.6625626Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:40:24.6626074Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:40:24.6626537Z ##[endgroup] 2024-08-06T20:40:24.9655566Z Defaulting to user installation because normal site-packages is not writeable 2024-08-06T20:40:24.9869298Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-08-06T20:40:24.9998845Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-08-06T20:40:25.1182768Z Prepare all required actions 2024-08-06T20:40:25.1183846Z Getting action download info 2024-08-06T20:40:25.2532465Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-06T20:40:25.4392444Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-06T20:40:25.5748288Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-06T20:40:25.5748764Z with: 2024-08-06T20:40:25.5749091Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-08-06T20:40:25.5749506Z s3-bucket: gha-artifacts 2024-08-06T20:40:25.5749855Z env: 2024-08-06T20:40:25.5750153Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:40:25.5750574Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:40:25.5751039Z ##[endgroup] 2024-08-06T20:40:25.5797220Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-06T20:40:25.5797669Z with: 2024-08-06T20:40:25.5798061Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-08-06T20:40:25.5798490Z s3-bucket: gha-artifacts 2024-08-06T20:40:25.5798858Z region: us-east-1 2024-08-06T20:40:25.5799178Z env: 2024-08-06T20:40:25.5799475Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:40:25.5799916Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:40:25.5800387Z ##[endgroup] 2024-08-06T20:40:26.1946482Z (node:66940) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-06T20:40:26.1947091Z 2024-08-06T20:40:26.1947340Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-06T20:40:26.1948010Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-06T20:40:26.1948717Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-06T20:40:26.2840930Z Found 1 objects with prefix pytorch/pytorch/10273124344/linux-focal-cuda11.8-py3.10-gcc9/ 2024-08-06T20:40:26.2841891Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-06T20:40:36.8623024Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-06T20:40:36.8634579Z Artifact download has finished successfully 2024-08-06T20:40:36.8849036Z ##[group]Run unzip -o artifacts.zip 2024-08-06T20:40:36.8849476Z unzip -o artifacts.zip 2024-08-06T20:40:36.8857552Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:40:36.8858024Z env: 2024-08-06T20:40:36.8858706Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:40:36.8859150Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:40:36.8859638Z ##[endgroup] 2024-08-06T20:40:36.8897145Z Archive: artifacts.zip 2024-08-06T20:40:36.8899451Z creating: dist/ 2024-08-06T20:40:39.3551786Z inflating: dist/torch-2.5.0a0+gitb9d86fa-cp310-cp310-linux_x86_64.whl 2024-08-06T20:40:39.3552440Z creating: build/custom_test_artifacts/ 2024-08-06T20:40:39.3552962Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-06T20:40:39.3553601Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-06T20:40:39.3554342Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-06T20:40:39.3562352Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-06T20:40:39.3563184Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-06T20:40:39.3563986Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-06T20:40:39.3564860Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-06T20:40:39.3565715Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-06T20:40:39.3567005Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-06T20:40:39.3569555Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-06T20:40:39.3570484Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-06T20:40:39.3571547Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-06T20:40:39.3572600Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-06T20:40:39.3574466Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-06T20:40:39.3576876Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-06T20:40:39.3577982Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-06T20:40:39.3579670Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-06T20:40:39.3580746Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-06T20:40:39.3581670Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-06T20:40:39.3582546Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-06T20:40:39.3640166Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-06T20:40:39.3641429Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-06T20:40:39.3642706Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-06T20:40:39.3643997Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-06T20:40:39.3645304Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-06T20:40:39.3646538Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-06T20:40:39.3647797Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-06T20:40:39.3649027Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-06T20:40:39.3650276Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-06T20:40:39.3696906Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-06T20:40:39.3747339Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-06T20:40:39.3748598Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-06T20:40:39.3749759Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-06T20:40:39.3750884Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-06T20:40:39.3751979Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-06T20:40:39.3753062Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-06T20:40:39.3754132Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-06T20:40:39.3755227Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-06T20:40:39.3831151Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-06T20:40:39.3907365Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-06T20:40:39.3908632Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-06T20:40:39.3909530Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-06T20:40:39.3910288Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-06T20:40:39.3911098Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-06T20:40:39.3911937Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-06T20:40:39.3912853Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-06T20:40:39.3913869Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-06T20:40:39.3914878Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-06T20:40:39.3915808Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-06T20:40:39.3916746Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-06T20:40:39.3917706Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-06T20:40:39.3918680Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-06T20:40:39.3919640Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-06T20:40:39.3920568Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-06T20:40:39.3942665Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-06T20:40:39.4122681Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-06T20:40:39.4123570Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-06T20:40:39.4124530Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-06T20:40:39.4125620Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-06T20:40:39.4126660Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-06T20:40:39.4127618Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-06T20:40:39.4128633Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-06T20:40:39.4129656Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-06T20:40:39.4130672Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-06T20:40:39.4131727Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-06T20:40:39.4132733Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-06T20:40:39.4155924Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-06T20:40:39.4260578Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-06T20:40:39.4261651Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-06T20:40:39.4262624Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-06T20:40:39.4263679Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-06T20:40:39.4264504Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-06T20:40:39.4265452Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-06T20:40:39.4266259Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-08-06T20:40:39.4268759Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-06T20:40:39.4269733Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-06T20:40:39.4270751Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-06T20:40:39.4420479Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-06T20:40:39.4499141Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-06T20:40:39.4499824Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-06T20:40:39.4500451Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-06T20:40:39.4501180Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-06T20:40:39.4509216Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-06T20:40:39.4510048Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-06T20:40:39.4510857Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-06T20:40:39.4511705Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-06T20:40:39.4512548Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-06T20:40:39.4513777Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-06T20:40:39.4514800Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-06T20:40:39.4515664Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-06T20:40:39.4516530Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-06T20:40:39.4519173Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-06T20:40:39.4520220Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-06T20:40:39.4522949Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-06T20:40:39.4523973Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-06T20:40:39.4525828Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-06T20:40:39.4526859Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-06T20:40:39.4527776Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-06T20:40:39.4528652Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-06T20:40:39.4586786Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-06T20:40:39.4588035Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-06T20:40:39.4589558Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-06T20:40:39.4590867Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-06T20:40:39.4592180Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-06T20:40:39.4593605Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-06T20:40:39.4594869Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-06T20:40:39.4596281Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-06T20:40:39.4597513Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-06T20:40:39.4643375Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-06T20:40:39.4693253Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-06T20:40:39.4694687Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-06T20:40:39.4695832Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-06T20:40:39.4696964Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-06T20:40:39.4698578Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-06T20:40:39.4699935Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-06T20:40:39.4701012Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-06T20:40:39.4702335Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-06T20:40:39.4777365Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-06T20:40:39.4853559Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-06T20:40:39.4854606Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-06T20:40:39.4855496Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-06T20:40:39.4856258Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-06T20:40:39.4857052Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-06T20:40:39.4857868Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-06T20:40:39.4859056Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-06T20:40:39.4860135Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-06T20:40:39.4861157Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-06T20:40:39.4862113Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-06T20:40:39.4863210Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-06T20:40:39.4864199Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-06T20:40:39.4865170Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-06T20:40:39.4866183Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-06T20:40:39.4867158Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-06T20:40:39.4888943Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-06T20:40:39.4969674Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-06T20:40:39.4970769Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-06T20:40:39.4972144Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-06T20:40:39.4972996Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-06T20:40:39.4973800Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-06T20:40:39.4974586Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-06T20:40:39.4975361Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-08-06T20:40:39.4976888Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-06T20:40:39.4977950Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-06T20:40:39.4978992Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-06T20:40:39.5041492Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-06T20:40:39.5042196Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-06T20:40:39.5042861Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-06T20:40:39.5043617Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-06T20:40:39.5051559Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-06T20:40:39.5052455Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-06T20:40:39.5053325Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-06T20:40:39.5054235Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-06T20:40:39.5055200Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-06T20:40:39.5056466Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-06T20:40:39.5057559Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-06T20:40:39.5058765Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-06T20:40:39.5059712Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-06T20:40:39.5061485Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-06T20:40:39.5063329Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-06T20:40:39.5065727Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-06T20:40:39.5066822Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-06T20:40:39.5068234Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-06T20:40:39.5069356Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-06T20:40:39.5070375Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-06T20:40:39.5071340Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-06T20:40:39.5128897Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-06T20:40:39.5130209Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-06T20:40:39.5131775Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-06T20:40:39.5133192Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-06T20:40:39.5134709Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-06T20:40:39.5135991Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-06T20:40:39.5137319Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-06T20:40:39.5138931Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-06T20:40:39.5140305Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-06T20:40:39.5201505Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-06T20:40:39.5251591Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-06T20:40:39.5252894Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-06T20:40:39.5254114Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-06T20:40:39.5255268Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-06T20:40:39.5256428Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-06T20:40:39.5257600Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-06T20:40:39.5259017Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-06T20:40:39.5260165Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-06T20:40:39.5335272Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-06T20:40:39.5411297Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-06T20:40:39.5412422Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-06T20:40:39.5413356Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-06T20:40:39.5414183Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-06T20:40:39.5415041Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-06T20:40:39.5415919Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-06T20:40:39.5416937Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-06T20:40:39.5418082Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-06T20:40:39.5419467Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-06T20:40:39.5420480Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-06T20:40:39.5421539Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-06T20:40:39.5422784Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-06T20:40:39.5423867Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-06T20:40:39.5425085Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-06T20:40:39.5426136Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-06T20:40:39.5427263Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-06T20:40:39.5584742Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-06T20:40:39.5585800Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-06T20:40:39.5586870Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-06T20:40:39.5588082Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-06T20:40:39.5589235Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-06T20:40:39.5590320Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-06T20:40:39.5591440Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-06T20:40:39.5592548Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-06T20:40:39.5593666Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-06T20:40:39.5594787Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-06T20:40:39.5595911Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-06T20:40:39.5617796Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-06T20:40:39.5687412Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-06T20:40:39.5688673Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-06T20:40:39.5689811Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-06T20:40:39.5690734Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-06T20:40:39.5691600Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-06T20:40:39.5692460Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-06T20:40:39.5693312Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-08-06T20:40:39.5695541Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-06T20:40:39.5696478Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-06T20:40:39.5697265Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-06T20:40:39.5826660Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-06T20:40:39.5880286Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-06T20:40:39.5880904Z creating: build/lib/ 2024-08-06T20:40:39.5892434Z inflating: build/lib/libpthreadpool.a 2024-08-06T20:40:39.5902820Z inflating: build/lib/libcpuinfo.a 2024-08-06T20:40:39.5912552Z inflating: build/lib/libcpuinfo_internals.a 2024-08-06T20:40:39.5913482Z inflating: build/lib/libclog.a 2024-08-06T20:40:39.6024870Z inflating: build/lib/libprotobuf-lite.a 2024-08-06T20:40:39.6027382Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-06T20:40:39.6595457Z inflating: build/lib/libprotobuf.a 2024-08-06T20:40:39.6676929Z inflating: build/lib/libgtest.a 2024-08-06T20:40:39.6769980Z inflating: build/lib/libbenchmark.a 2024-08-06T20:40:39.6779676Z inflating: build/lib/libittnotify.a 2024-08-06T20:40:39.6813979Z inflating: build/lib/libtensorpipe_uv.a 2024-08-06T20:40:39.6836842Z inflating: build/lib/libnnpack.a 2024-08-06T20:40:39.6916119Z inflating: build/lib/libasmjit.a 2024-08-06T20:40:39.7074842Z inflating: build/lib/libgloo.a 2024-08-06T20:40:39.7100996Z inflating: build/lib/libfmt.a 2024-08-06T20:40:39.7218706Z inflating: build/lib/libc10.so 2024-08-06T20:40:39.7221224Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-06T20:40:39.7221783Z inflating: build/lib/libfoxi_loader.a 2024-08-06T20:40:39.7224077Z inflating: build/lib/libtorch_global_deps.so 2024-08-06T20:40:39.7248114Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-06T20:40:39.7270767Z inflating: build/lib/libgmock.a 2024-08-06T20:40:39.7271460Z inflating: build/lib/libgtest_main.a 2024-08-06T20:40:39.7272453Z inflating: build/lib/libbenchmark_main.a 2024-08-06T20:40:39.7899724Z inflating: build/lib/libprotoc.a 2024-08-06T20:40:41.0545952Z inflating: build/lib/libdnnl.a 2024-08-06T20:40:41.1259432Z inflating: build/lib/libtensorpipe.a 2024-08-06T20:40:41.1330514Z inflating: build/lib/libc10_cuda.so 2024-08-06T20:40:41.1331315Z inflating: build/lib/libgmock_main.a 2024-08-06T20:40:41.2891183Z inflating: build/lib/libfbgemm.a 2024-08-06T20:40:41.3521244Z inflating: build/lib/libkineto.a 2024-08-06T20:40:41.3840124Z inflating: build/lib/libtensorpipe_cuda.a 2024-08-06T20:40:41.4069635Z inflating: build/lib/libXNNPACK.a 2024-08-06T20:40:41.4122364Z inflating: build/lib/libonnx_proto.a 2024-08-06T20:40:41.4606766Z inflating: build/lib/libgloo_cuda.a 2024-08-06T20:40:41.5468483Z inflating: build/lib/libonnx.a 2024-08-06T20:40:44.5645968Z inflating: build/lib/libtorch_cpu.so 2024-08-06T20:40:44.5650873Z inflating: build/lib/libshm.so 2024-08-06T20:40:44.5656780Z inflating: build/lib/libunbox_lib.a 2024-08-06T20:40:47.0388642Z inflating: build/lib/libtorch_cuda.so 2024-08-06T20:40:47.0389540Z inflating: build/lib/libtorch.so 2024-08-06T20:40:47.0393413Z inflating: build/lib/libc10d_cuda_test.so 2024-08-06T20:40:48.1772874Z inflating: build/lib/libtorch_cuda_linalg.so 2024-08-06T20:40:48.1798775Z inflating: build/lib/libjitbackend_test.so 2024-08-06T20:40:48.1888058Z inflating: build/lib/libtorchbind_test.so 2024-08-06T20:40:48.1920790Z inflating: build/lib/libaoti_custom_ops.so 2024-08-06T20:40:48.1952430Z inflating: build/lib/libbackend_with_compiler.so 2024-08-06T20:40:48.4431122Z inflating: build/lib/libtorch_python.so 2024-08-06T20:40:48.4474323Z inflating: build/lib/libnnapi_backend.so 2024-08-06T20:40:48.4474922Z creating: build/bin/ 2024-08-06T20:40:48.4537220Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-06T20:40:48.4601488Z inflating: build/bin/c10_DeviceGuard_test 2024-08-06T20:40:48.4664429Z inflating: build/bin/c10_Device_test 2024-08-06T20:40:48.4737966Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-06T20:40:48.4804764Z inflating: build/bin/c10_Scalar_test 2024-08-06T20:40:48.4866070Z inflating: build/bin/c10_StreamGuard_test 2024-08-06T20:40:48.4929550Z inflating: build/bin/c10_SymInt_test 2024-08-06T20:40:48.4997872Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-06T20:40:48.5067393Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-06T20:40:48.5137633Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-06T20:40:48.5226319Z inflating: build/bin/c10_cow_test 2024-08-06T20:40:48.5292545Z inflating: build/bin/c10_Bitset_test 2024-08-06T20:40:48.5354362Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-06T20:40:48.5416117Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-06T20:40:48.5479706Z inflating: build/bin/c10_Half_test 2024-08-06T20:40:48.5550230Z inflating: build/bin/c10_LeftRight_test 2024-08-06T20:40:48.5618924Z inflating: build/bin/c10_Metaprogramming_test 2024-08-06T20:40:48.5681358Z inflating: build/bin/c10_Synchronized_test 2024-08-06T20:40:48.5745653Z inflating: build/bin/c10_TypeIndex_test 2024-08-06T20:40:48.5815183Z inflating: build/bin/c10_ThreadLocal_test 2024-08-06T20:40:48.5878992Z inflating: build/bin/c10_TypeList_test 2024-08-06T20:40:48.5940233Z inflating: build/bin/c10_TypeTraits_test 2024-08-06T20:40:48.6004733Z inflating: build/bin/c10_accumulate_test 2024-08-06T20:40:48.6074435Z inflating: build/bin/c10_bfloat16_test 2024-08-06T20:40:48.6137447Z inflating: build/bin/c10_bit_cast_test 2024-08-06T20:40:48.6208271Z inflating: build/bin/c10_complex_math_test 2024-08-06T20:40:48.6278097Z inflating: build/bin/c10_complex_test 2024-08-06T20:40:48.6344023Z inflating: build/bin/c10_exception_test 2024-08-06T20:40:48.6406959Z inflating: build/bin/c10_flags_test 2024-08-06T20:40:48.6469650Z inflating: build/bin/c10_generic_math_test 2024-08-06T20:40:48.6675656Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-06T20:40:48.6739286Z inflating: build/bin/c10_irange_test 2024-08-06T20:40:48.6805958Z inflating: build/bin/c10_lazy_test 2024-08-06T20:40:48.6877404Z inflating: build/bin/c10_logging_test 2024-08-06T20:40:48.6971086Z inflating: build/bin/c10_optional_test 2024-08-06T20:40:48.7048967Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-06T20:40:48.7116370Z inflating: build/bin/c10_registry_test 2024-08-06T20:40:48.7305306Z inflating: build/bin/c10_small_vector_test 2024-08-06T20:40:48.7369722Z inflating: build/bin/c10_ssize_test 2024-08-06T20:40:48.7434733Z inflating: build/bin/c10_string_util_test 2024-08-06T20:40:48.7508329Z inflating: build/bin/c10_string_view_test 2024-08-06T20:40:48.7571080Z inflating: build/bin/c10_tempfile_test 2024-08-06T20:40:48.7641236Z inflating: build/bin/c10_typeid_test 2024-08-06T20:40:48.7701835Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-06T20:40:48.8258737Z inflating: build/bin/protoc-3.13.0.0 2024-08-06T20:40:48.8815743Z inflating: build/bin/protoc 2024-08-06T20:40:48.8881786Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-08-06T20:40:48.8947687Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-08-06T20:40:48.9014489Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-08-06T20:40:48.9080603Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-06T20:40:48.9145645Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-08-06T20:40:48.9211606Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-06T20:40:48.9273011Z inflating: build/bin/c10_cuda_CUDATest 2024-08-06T20:40:48.9338743Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-08-06T20:40:48.9751258Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-06T20:40:49.0181375Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-06T20:40:49.0630128Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-06T20:40:49.0696850Z inflating: build/bin/HashStoreTest 2024-08-06T20:40:49.0763250Z inflating: build/bin/FileStoreTest 2024-08-06T20:40:49.0829164Z inflating: build/bin/BackoffTest 2024-08-06T20:40:49.0898778Z inflating: build/bin/TCPStoreTest 2024-08-06T20:40:49.0915797Z inflating: build/bin/ProcessGroupMPITest 2024-08-06T20:40:49.0921047Z inflating: build/bin/torch_shm_manager 2024-08-06T20:40:49.0988465Z inflating: build/bin/test_edge_op_registration 2024-08-06T20:40:49.0992308Z inflating: build/bin/example_allreduce 2024-08-06T20:40:49.1056924Z inflating: build/bin/Dimname_test 2024-08-06T20:40:49.1149178Z inflating: build/bin/Dict_test 2024-08-06T20:40:49.1230002Z inflating: build/bin/MaybeOwned_test 2024-08-06T20:40:49.1301568Z inflating: build/bin/NamedTensor_test 2024-08-06T20:40:49.1375249Z inflating: build/bin/apply_utils_test 2024-08-06T20:40:49.1448924Z inflating: build/bin/atest 2024-08-06T20:40:49.1527609Z inflating: build/bin/basic 2024-08-06T20:40:49.1591192Z inflating: build/bin/cpu_allocator_test 2024-08-06T20:40:49.1660297Z inflating: build/bin/broadcast_test 2024-08-06T20:40:49.1732806Z inflating: build/bin/cpu_generator_test 2024-08-06T20:40:49.1799508Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-06T20:40:49.1862065Z inflating: build/bin/dispatch_key_set_test 2024-08-06T20:40:49.1924884Z inflating: build/bin/dlconvertor_test 2024-08-06T20:40:49.2041442Z inflating: build/bin/cpu_rng_test 2024-08-06T20:40:49.2114582Z inflating: build/bin/extension_backend_test 2024-08-06T20:40:49.2183708Z inflating: build/bin/half_test 2024-08-06T20:40:49.2303411Z inflating: build/bin/ivalue_test 2024-08-06T20:40:49.2365498Z inflating: build/bin/lazy_tensor_test 2024-08-06T20:40:49.2433295Z inflating: build/bin/memory_format_test 2024-08-06T20:40:49.2499726Z inflating: build/bin/memory_overlapping_test 2024-08-06T20:40:49.2568058Z inflating: build/bin/math_kernel_test 2024-08-06T20:40:49.2634679Z inflating: build/bin/mobile_memory_cleanup 2024-08-06T20:40:49.2705076Z inflating: build/bin/native_test 2024-08-06T20:40:49.2768396Z inflating: build/bin/operator_name_test 2024-08-06T20:40:49.2833610Z inflating: build/bin/operators_test 2024-08-06T20:40:49.2898331Z inflating: build/bin/packedtensoraccessor_test 2024-08-06T20:40:49.2983006Z inflating: build/bin/pow_test 2024-08-06T20:40:49.3054584Z inflating: build/bin/quantized_test 2024-08-06T20:40:49.3119280Z inflating: build/bin/reportMemoryUsage_test 2024-08-06T20:40:49.3182193Z inflating: build/bin/reduce_ops_test 2024-08-06T20:40:49.3253265Z inflating: build/bin/scalar_tensor_test 2024-08-06T20:40:49.3318418Z inflating: build/bin/StorageUtils_test 2024-08-06T20:40:49.3392022Z inflating: build/bin/scalar_test 2024-08-06T20:40:49.3458021Z inflating: build/bin/stride_properties_test 2024-08-06T20:40:49.3557674Z inflating: build/bin/tensor_iterator_test 2024-08-06T20:40:49.3625750Z inflating: build/bin/test_parallel 2024-08-06T20:40:49.3629495Z inflating: build/bin/thread_init_test 2024-08-06T20:40:49.3699719Z inflating: build/bin/type_ptr_test 2024-08-06T20:40:49.3774858Z inflating: build/bin/type_test 2024-08-06T20:40:49.3840464Z inflating: build/bin/undefined_tensor_test 2024-08-06T20:40:49.3842533Z inflating: build/bin/verify_api_visibility 2024-08-06T20:40:49.3907600Z inflating: build/bin/weakref_test 2024-08-06T20:40:49.3994567Z inflating: build/bin/legacy_vmap_test 2024-08-06T20:40:49.4059161Z inflating: build/bin/wrapdim_test 2024-08-06T20:40:49.4193866Z inflating: build/bin/List_test 2024-08-06T20:40:49.4269425Z inflating: build/bin/IListRef_test 2024-08-06T20:40:49.4334139Z inflating: build/bin/xla_tensor_test 2024-08-06T20:40:49.4487291Z inflating: build/bin/kernel_function_legacy_test 2024-08-06T20:40:49.4608198Z inflating: build/bin/kernel_function_test 2024-08-06T20:40:49.4691547Z inflating: build/bin/KernelFunction_test 2024-08-06T20:40:49.4851736Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-06T20:40:49.4981737Z inflating: build/bin/kernel_lambda_test 2024-08-06T20:40:49.5057862Z inflating: build/bin/kernel_stackbased_test 2024-08-06T20:40:49.5121955Z inflating: build/bin/CppSignature_test 2024-08-06T20:40:49.5242415Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-06T20:40:49.5303616Z inflating: build/bin/op_allowlist_test 2024-08-06T20:40:49.5373548Z inflating: build/bin/backend_fallback_test 2024-08-06T20:40:49.5452991Z inflating: build/bin/inline_container_test 2024-08-06T20:40:49.5830642Z inflating: build/bin/op_registration_test 2024-08-06T20:40:49.5896794Z inflating: build/bin/cuda_apply_test 2024-08-06T20:40:49.5961724Z inflating: build/bin/cuda_allocator_test 2024-08-06T20:40:49.6030156Z inflating: build/bin/cuda_caching_host_allocator_test 2024-08-06T20:40:49.6118363Z inflating: build/bin/cuda_complex_math_test 2024-08-06T20:40:49.6192545Z inflating: build/bin/cuda_atomic_ops_test 2024-08-06T20:40:49.6266549Z inflating: build/bin/cuda_complex_test 2024-08-06T20:40:49.6328488Z inflating: build/bin/cuda_device_test 2024-08-06T20:40:49.6400886Z inflating: build/bin/cuda_cub_test 2024-08-06T20:40:49.6464616Z inflating: build/bin/cuda_dlconvertor_test 2024-08-06T20:40:49.6528338Z inflating: build/bin/cuda_integer_divider_test 2024-08-06T20:40:49.6610163Z inflating: build/bin/cuda_distributions_test 2024-08-06T20:40:49.6681479Z inflating: build/bin/cuda_generator_test 2024-08-06T20:40:49.6743770Z inflating: build/bin/cuda_half_test 2024-08-06T20:40:49.6805597Z inflating: build/bin/cuda_optional_test 2024-08-06T20:40:49.6868056Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-08-06T20:40:49.6943911Z inflating: build/bin/cuda_stream_test 2024-08-06T20:40:49.7009489Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-08-06T20:40:49.7074069Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-08-06T20:40:49.7135572Z inflating: build/bin/cuda_cudnn_test 2024-08-06T20:40:49.7200825Z inflating: build/bin/cuda_vectorized_test 2024-08-06T20:40:49.7218851Z inflating: build/bin/tutorial_tensorexpr 2024-08-06T20:40:49.7300886Z inflating: build/bin/ProcessGroupGlooTest 2024-08-06T20:40:49.7372669Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-08-06T20:40:49.7438498Z inflating: build/bin/ProcessGroupUCCTest 2024-08-06T20:40:49.7516146Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-08-06T20:40:49.7595749Z inflating: build/bin/ProcessGroupNCCLTest 2024-08-06T20:40:49.7664488Z inflating: build/bin/test_dist_autograd 2024-08-06T20:40:49.7667679Z inflating: build/bin/parallel_benchmark 2024-08-06T20:40:49.7751421Z inflating: build/bin/test_mobile_nnc 2024-08-06T20:40:49.7837673Z inflating: build/bin/test_cpp_rpc 2024-08-06T20:40:49.7848921Z inflating: build/bin/aot_model_compiler_test 2024-08-06T20:40:49.8881964Z inflating: build/bin/test_tensorexpr 2024-08-06T20:40:49.9315944Z inflating: build/bin/test_lazy 2024-08-06T20:40:50.0789901Z inflating: build/bin/test_api 2024-08-06T20:40:50.1514588Z inflating: build/bin/test_jit 2024-08-06T20:40:50.1515006Z creating: .additional_ci_files/ 2024-08-06T20:40:50.1587224Z inflating: .additional_ci_files/test-times.json 2024-08-06T20:40:50.1874215Z inflating: .additional_ci_files/test-class-times.json 2024-08-06T20:40:50.1917174Z ##[group]Run rm artifacts.zip 2024-08-06T20:40:50.1917670Z rm artifacts.zip 2024-08-06T20:40:50.1925600Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:40:50.1926109Z env: 2024-08-06T20:40:50.1926391Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:40:50.1926825Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:40:50.1927288Z ##[endgroup] 2024-08-06T20:40:50.3149753Z ##[group]Run df -H 2024-08-06T20:40:50.3150102Z df -H 2024-08-06T20:40:50.3157939Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:40:50.3158418Z env: 2024-08-06T20:40:50.3158712Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:40:50.3159126Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:40:50.3159590Z ##[endgroup] 2024-08-06T20:40:50.3192744Z Filesystem Size Used Avail Use% Mounted on 2024-08-06T20:40:50.3193224Z devtmpfs 129G 0 129G 0% /dev 2024-08-06T20:40:50.3193664Z tmpfs 129G 4.1M 129G 1% /dev/shm 2024-08-06T20:40:50.3194567Z tmpfs 129G 594k 129G 1% /run 2024-08-06T20:40:50.3195024Z tmpfs 129G 0 129G 0% /sys/fs/cgroup 2024-08-06T20:40:50.3195468Z /dev/xvda1 162G 38G 125G 23% / 2024-08-06T20:40:50.3196084Z tmpfs 26G 0 26G 0% /run/user/0 2024-08-06T20:40:50.3235502Z Prepare all required actions 2024-08-06T20:40:50.3236003Z Getting action download info 2024-08-06T20:40:50.4435299Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-06T20:40:50.4435762Z with: 2024-08-06T20:40:50.4436043Z env: 2024-08-06T20:40:50.4436318Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:40:50.4436760Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:40:50.4437230Z ##[endgroup] 2024-08-06T20:40:50.4482716Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-06T20:40:50.4483156Z with: 2024-08-06T20:40:50.4483445Z name: td_results 2024-08-06T20:40:50.4483750Z s3-bucket: gha-artifacts 2024-08-06T20:40:50.4484107Z region: us-east-1 2024-08-06T20:40:50.4484411Z env: 2024-08-06T20:40:50.4484673Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:40:50.4485109Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:40:50.4485590Z ##[endgroup] 2024-08-06T20:40:51.0585152Z (node:66981) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-06T20:40:51.0585766Z 2024-08-06T20:40:51.0586039Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-06T20:40:51.0586690Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-06T20:40:51.0587597Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-06T20:40:51.1490965Z Found 1 objects with prefix pytorch/pytorch/10273124344/td_results/ 2024-08-06T20:40:51.1491945Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-08-06T20:40:51.2014318Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-08-06T20:40:51.2025913Z Artifact download has finished successfully 2024-08-06T20:40:51.2212815Z ##[group]Run mkdir -p .additional_ci_files 2024-08-06T20:40:51.2213298Z mkdir -p .additional_ci_files 2024-08-06T20:40:51.2213846Z mv td_results.json .additional_ci_files/td_results.json 2024-08-06T20:40:51.2222611Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:40:51.2223105Z env: 2024-08-06T20:40:51.2223384Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:40:51.2223823Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:40:51.2224290Z ##[endgroup] 2024-08-06T20:40:51.2308596Z ##[group]Run .github/scripts/parse_ref.py 2024-08-06T20:40:51.2309102Z .github/scripts/parse_ref.py 2024-08-06T20:40:51.2316616Z shell: /usr/bin/bash -e {0} 2024-08-06T20:40:51.2316958Z env: 2024-08-06T20:40:51.2317347Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:40:51.2317796Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:40:51.2318243Z ##[endgroup] 2024-08-06T20:40:51.2622869Z Prepare all required actions 2024-08-06T20:40:51.2667754Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-06T20:40:51.2668186Z with: 2024-08-06T20:40:51.2668705Z github-token: *** 2024-08-06T20:40:51.2669026Z env: 2024-08-06T20:40:51.2669318Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:40:51.2669735Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:40:51.2670198Z ##[endgroup] 2024-08-06T20:40:51.2694155Z ##[group]Run set -eux 2024-08-06T20:40:51.2694506Z set -eux 2024-08-06T20:40:51.2695051Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-06T20:40:51.2702816Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:40:51.2703305Z env: 2024-08-06T20:40:51.2703581Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:40:51.2704015Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:40:51.2704665Z GITHUB_TOKEN: *** 2024-08-06T20:40:51.2704968Z ##[endgroup] 2024-08-06T20:40:51.2727948Z + python3 .github/scripts/get_workflow_job_id.py 10273124344 i-06f045a861c998624 2024-08-06T20:40:55.0114047Z setting job-id=28428107750 2024-08-06T20:40:55.0115042Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-08-06T20:40:55.0323878Z Prepare all required actions 2024-08-06T20:40:55.0324419Z Getting action download info 2024-08-06T20:40:55.1484688Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-06T20:40:55.1485127Z with: 2024-08-06T20:40:55.1485651Z github-token: *** 2024-08-06T20:40:55.1486996Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]} 2024-08-06T20:40:55.1488640Z job-name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-08-06T20:40:55.1489271Z env: 2024-08-06T20:40:55.1489569Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:40:55.1490006Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:40:55.1490471Z ##[endgroup] 2024-08-06T20:40:55.1542865Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-06T20:40:55.1543408Z with: 2024-08-06T20:40:55.1543699Z shell: bash 2024-08-06T20:40:55.1543994Z timeout_minutes: 10 2024-08-06T20:40:55.1544334Z max_attempts: 5 2024-08-06T20:40:55.1544657Z retry_wait_seconds: 30 2024-08-06T20:40:55.1545690Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-06T20:40:55.1546752Z polling_interval_seconds: 1 2024-08-06T20:40:55.1547135Z warning_on_retry: true 2024-08-06T20:40:55.1547489Z continue_on_error: false 2024-08-06T20:40:55.1547813Z env: 2024-08-06T20:40:55.1548109Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:40:55.1548541Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:40:55.1549183Z GITHUB_TOKEN: *** 2024-08-06T20:40:55.1549531Z ##[endgroup] 2024-08-06T20:40:55.2274660Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-06T20:40:55.5206559Z Defaulting to user installation because normal site-packages is not writeable 2024-08-06T20:40:55.5416251Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1) 2024-08-06T20:40:55.5600547Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-08-06T20:40:55.5611446Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.7.4) 2024-08-06T20:40:55.5623129Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7) 2024-08-06T20:40:55.5639699Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19) 2024-08-06T20:40:55.5891553Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12) 2024-08-06T20:40:56.2261915Z Command completed after 1 attempt(s). 2024-08-06T20:40:56.2319620Z ##[group]Run set -x 2024-08-06T20:40:56.2319979Z set -x 2024-08-06T20:40:56.2320274Z  2024-08-06T20:40:56.2320775Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-06T20:40:56.2321384Z # in runner workspace 2024-08-06T20:40:56.2321870Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-06T20:40:56.2329942Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:40:56.2330442Z env: 2024-08-06T20:40:56.2330731Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:40:56.2331175Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:40:56.2331649Z ##[endgroup] 2024-08-06T20:40:56.2356956Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-06T20:40:56.2636697Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-06T20:40:56.2637234Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-06T20:40:56.2637654Z echo "Job name: ${JOB_NAME}" 2024-08-06T20:40:56.2638037Z  2024-08-06T20:40:56.2638519Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-06T20:40:56.2639101Z # in runner workspace 2024-08-06T20:40:56.2639643Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-06T20:40:56.2640245Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-06T20:40:56.2640663Z  --job-name "${JOB_NAME}" \ 2024-08-06T20:40:56.2642082Z  --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]}" \ 2024-08-06T20:40:56.2643565Z  --selected-test-configs "" \ 2024-08-06T20:40:56.2644008Z  --pr-number "${PR_NUMBER}" \ 2024-08-06T20:40:56.2644403Z  --tag "${TAG}" \ 2024-08-06T20:40:56.2644789Z  --event-name "${EVENT_NAME}" \ 2024-08-06T20:40:56.2645211Z  --schedule "${SCHEDULE}" \ 2024-08-06T20:40:56.2645604Z  --branch "${HEAD_BRANCH}" 2024-08-06T20:40:56.2652956Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:40:56.2653438Z env: 2024-08-06T20:40:56.2653777Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:40:56.2654457Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:40:56.2655225Z GITHUB_TOKEN: *** 2024-08-06T20:40:56.2655917Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-08-06T20:40:56.2656686Z PR_NUMBER: 132710 2024-08-06T20:40:56.2657095Z TAG: 2024-08-06T20:40:56.2657422Z EVENT_NAME: pull_request 2024-08-06T20:40:56.2657907Z SCHEDULE: 2024-08-06T20:40:56.2658535Z HEAD_BRANCH: 2024-08-06T20:40:56.2658888Z ##[endgroup] 2024-08-06T20:40:56.2698927Z Workflow: pull 2024-08-06T20:40:56.2699535Z Job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-08-06T20:40:56.5202972Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-06T20:40:56.7227787Z ##[group]Run echo "Filtered matrix:" 2024-08-06T20:40:56.7228230Z echo "Filtered matrix:" 2024-08-06T20:40:56.7229587Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]}" 2024-08-06T20:40:56.7230970Z  2024-08-06T20:40:56.7231272Z echo 2024-08-06T20:40:56.7231635Z echo "Is the current job unstable? False" 2024-08-06T20:40:56.7232072Z  2024-08-06T20:40:56.7232354Z echo 2024-08-06T20:40:56.7232683Z echo "Is keep-going label set? False" 2024-08-06T20:40:56.7233167Z  2024-08-06T20:40:56.7233451Z echo 2024-08-06T20:40:56.7233751Z echo "Renabled issues? " 2024-08-06T20:40:56.7241177Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:40:56.7241755Z env: 2024-08-06T20:40:56.7242035Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:40:56.7242472Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:40:56.7242934Z ##[endgroup] 2024-08-06T20:40:56.7266063Z Filtered matrix: 2024-08-06T20:40:56.7267577Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 2, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 3, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}]} 2024-08-06T20:40:56.7269036Z 2024-08-06T20:40:56.7269342Z Is the current job unstable? False 2024-08-06T20:40:56.7269626Z 2024-08-06T20:40:56.7269796Z Is keep-going label set? False 2024-08-06T20:40:56.7270042Z 2024-08-06T20:40:56.7270181Z Renabled issues? 2024-08-06T20:40:56.7322275Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-06T20:40:56.7322962Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-06T20:40:56.7330172Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:40:56.7330638Z env: 2024-08-06T20:40:56.7330939Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:40:56.7331362Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:40:56.7331833Z JOB_TIMEOUT: 360 2024-08-06T20:40:56.7332152Z ##[endgroup] 2024-08-06T20:40:56.7414366Z ##[group]Run set -x 2024-08-06T20:40:56.7414830Z set -x 2024-08-06T20:40:56.7415124Z  2024-08-06T20:40:56.7415469Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-06T20:40:56.7416004Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-06T20:40:56.7416508Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-06T20:40:56.7416982Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-06T20:40:56.7417382Z else 2024-08-06T20:40:56.7417703Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-06T20:40:56.7418110Z fi 2024-08-06T20:40:56.7418800Z  2024-08-06T20:40:56.7419232Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-06T20:40:56.7419905Z # TODO: Stop building test binaries as part of the build phase 2024-08-06T20:40:56.7420503Z # Used for GPU_FLAG since that doesn't play nice 2024-08-06T20:40:56.7421040Z # shellcheck disable=SC2086,SC2090 2024-08-06T20:40:56.7421476Z container_name=$(docker run \ 2024-08-06T20:40:56.7421890Z  ${GPU_FLAG:-} \ 2024-08-06T20:40:56.7422268Z  -e BUILD_ENVIRONMENT \ 2024-08-06T20:40:56.7422643Z  -e PR_NUMBER \ 2024-08-06T20:40:56.7423013Z  -e GITHUB_ACTIONS \ 2024-08-06T20:40:56.7423397Z  -e GITHUB_REPOSITORY \ 2024-08-06T20:40:56.7423775Z  -e GITHUB_WORKFLOW \ 2024-08-06T20:40:56.7424154Z  -e GITHUB_JOB \ 2024-08-06T20:40:56.7424510Z  -e GITHUB_RUN_ID \ 2024-08-06T20:40:56.7424863Z  -e GITHUB_RUN_NUMBER \ 2024-08-06T20:40:56.7425258Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-06T20:40:56.7425640Z  -e JOB_ID \ 2024-08-06T20:40:56.7425964Z  -e JOB_NAME \ 2024-08-06T20:40:56.7426313Z  -e BASE_SHA \ 2024-08-06T20:40:56.7426656Z  -e BRANCH \ 2024-08-06T20:40:56.7426980Z  -e SHA1 \ 2024-08-06T20:40:56.7427325Z  -e AWS_DEFAULT_REGION \ 2024-08-06T20:40:56.7427718Z  -e IN_WHEEL_TEST \ 2024-08-06T20:40:56.7428071Z  -e SHARD_NUMBER \ 2024-08-06T20:40:56.7428446Z  -e TEST_CONFIG \ 2024-08-06T20:40:56.7428811Z  -e NUM_TEST_SHARDS \ 2024-08-06T20:40:56.7429180Z  -e REENABLED_ISSUES \ 2024-08-06T20:40:56.7429579Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-06T20:40:56.7429992Z  -e VERBOSE_TEST_LOGS \ 2024-08-06T20:40:56.7430370Z  -e TEST_SHOWLOCALS \ 2024-08-06T20:40:56.7430751Z  -e NO_TEST_TIMEOUT \ 2024-08-06T20:40:56.7431118Z  -e NO_TD \ 2024-08-06T20:40:56.7431441Z  -e TD_DISTRIBUTED \ 2024-08-06T20:40:56.7431818Z  -e PR_LABELS \ 2024-08-06T20:40:56.7432209Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-06T20:40:56.7432629Z  -e SCCACHE_BUCKET \ 2024-08-06T20:40:56.7433063Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-06T20:40:56.7433458Z  -e XLA_CUDA \ 2024-08-06T20:40:56.7433833Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-06T20:40:56.7434303Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-06T20:40:56.7434993Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-06T20:40:56.7435452Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-06T20:40:56.7435903Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-06T20:40:56.7436314Z  -e DASHBOARD_TAG \ 2024-08-06T20:40:56.7436736Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-06T20:40:56.7437247Z  --security-opt seccomp=unconfined \ 2024-08-06T20:40:56.7437693Z  --cap-add=SYS_PTRACE \ 2024-08-06T20:40:56.7438066Z  --ipc=host \ 2024-08-06T20:40:56.7438424Z  --shm-size="${SHM_SIZE}" \ 2024-08-06T20:40:56.7438811Z  --tty \ 2024-08-06T20:40:56.7439116Z  --detach \ 2024-08-06T20:40:56.7439469Z  --name="${container_name}" \ 2024-08-06T20:40:56.7440004Z  --user jenkins \ 2024-08-06T20:40:56.7440468Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-06T20:40:56.7440983Z  -w /var/lib/jenkins/workspace \ 2024-08-06T20:40:56.7441411Z  "${DOCKER_IMAGE}" 2024-08-06T20:40:56.7441748Z ) 2024-08-06T20:40:56.7442137Z # Propagate download.pytorch.org IP to container 2024-08-06T20:40:56.7442960Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-06T20:40:56.7443828Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-06T20:40:56.7444642Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-06T20:40:56.7452102Z shell: /usr/bin/bash -e {0} 2024-08-06T20:40:56.7452455Z env: 2024-08-06T20:40:56.7452755Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:40:56.7453196Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:40:56.7453763Z BUILD_ENVIRONMENT: linux-focal-cuda11.8-py3.10-gcc9 2024-08-06T20:40:56.7454216Z PR_NUMBER: 132710 2024-08-06T20:40:56.7454559Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-06T20:40:56.7454957Z GITHUB_WORKFLOW: pull 2024-08-06T20:40:56.7455296Z GITHUB_JOB: test 2024-08-06T20:40:56.7455619Z GITHUB_RUN_ID: 10273124344 2024-08-06T20:40:56.7455960Z GITHUB_RUN_NUMBER: 233985 2024-08-06T20:40:56.7456313Z GITHUB_RUN_ATTEMPT: 1 2024-08-06T20:40:56.7456645Z JOB_ID: 28428107750 2024-08-06T20:40:56.7457207Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-08-06T20:40:56.7457865Z BRANCH: pull/132710 2024-08-06T20:40:56.7458524Z SHA1: b9d86fa89636e301796d4201f36d86c73f6e49bc 2024-08-06T20:40:56.7459004Z BASE_SHA: 1736af7cf736184c356be1bb00f59fb2feea6d7d 2024-08-06T20:40:56.7459447Z TEST_CONFIG: distributed 2024-08-06T20:40:56.7459793Z SHARD_NUMBER: 2 2024-08-06T20:40:56.7460085Z NUM_TEST_SHARDS: 3 2024-08-06T20:40:56.7460415Z REENABLED_ISSUES: 2024-08-06T20:40:56.7460754Z CONTINUE_THROUGH_ERROR: False 2024-08-06T20:40:56.7461111Z VERBOSE_TEST_LOGS: False 2024-08-06T20:40:56.7461466Z TEST_SHOWLOCALS: False 2024-08-06T20:40:56.7461822Z NO_TEST_TIMEOUT: False 2024-08-06T20:40:56.7462162Z NO_TD: False 2024-08-06T20:40:56.7462455Z TD_DISTRIBUTED: False 2024-08-06T20:40:56.7462869Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-06T20:40:56.7463342Z SCCACHE_S3_KEY_PREFIX: pull 2024-08-06T20:40:56.7463686Z SHM_SIZE: 2g 2024-08-06T20:40:56.7464570Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-06T20:40:56.7465536Z XLA_CUDA: 2024-08-06T20:40:56.7466000Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-06T20:40:56.7466563Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-06T20:40:56.7466984Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-06T20:40:56.7467398Z DASHBOARD_TAG: 2024-08-06T20:40:56.7467705Z HUGGING_FACE_HUB_TOKEN: 2024-08-06T20:40:56.7468058Z ##[endgroup] 2024-08-06T20:40:56.7491489Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-08-06T20:40:56.7492285Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *onnx* ]] 2024-08-06T20:40:56.7492767Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-06T20:40:56.7501894Z +++ nproc --ignore=2 2024-08-06T20:40:56.7517137Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10273124344 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-06T20:41:08.0167966Z + container_name=0e2f9f76c86bfdf6b10ad25e0546516e8d36c6bdfb223c138546808c0d6a4ac6 2024-08-06T20:41:08.0169634Z + grep download.pytorch.org /etc/hosts 2024-08-06T20:41:08.0172021Z + docker exec -i 0e2f9f76c86bfdf6b10ad25e0546516e8d36c6bdfb223c138546808c0d6a4ac6 sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-06T20:41:08.1060670Z + echo DOCKER_CONTAINER_ID=0e2f9f76c86bfdf6b10ad25e0546516e8d36c6bdfb223c138546808c0d6a4ac6 2024-08-06T20:41:08.1064658Z ++ echo dist/torch-2.5.0a0+gitb9d86fa-cp310-cp310-linux_x86_64.whl 2024-08-06T20:41:08.1066845Z + docker exec -t 0e2f9f76c86bfdf6b10ad25e0546516e8d36c6bdfb223c138546808c0d6a4ac6 sh -c 'pip install dist/torch-2.5.0a0+gitb9d86fa-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-06T20:41:08.6400458Z Processing ./dist/torch-2.5.0a0+gitb9d86fa-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+gitb9d86fa) 2024-08-06T20:41:09.7700053Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb9d86fa->torch==2.5.0a0+gitb9d86fa) (3.13.1) 2024-08-06T20:41:09.7707097Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb9d86fa->torch==2.5.0a0+gitb9d86fa) (4.12.2) 2024-08-06T20:41:09.7711815Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb9d86fa->torch==2.5.0a0+gitb9d86fa) (2.8.8) 2024-08-06T20:41:09.7716974Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb9d86fa->torch==2.5.0a0+gitb9d86fa) (3.1.4) 2024-08-06T20:41:09.7721872Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb9d86fa->torch==2.5.0a0+gitb9d86fa) (2024.6.1) 2024-08-06T20:41:09.7735619Z Requirement already satisfied: sympy>=1.13.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb9d86fa->torch==2.5.0a0+gitb9d86fa) (1.13.1) 2024-08-06T20:41:09.7777295Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb9d86fa->torch==2.5.0a0+gitb9d86fa) (3.3.0) 2024-08-06T20:41:09.7859629Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+gitb9d86fa->torch==2.5.0a0+gitb9d86fa) (1.21.2) 2024-08-06T20:41:09.7907509Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.0->torch==2.5.0a0+gitb9d86fa->torch==2.5.0a0+gitb9d86fa) (1.3.0) 2024-08-06T20:41:09.9194370Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+gitb9d86fa->torch==2.5.0a0+gitb9d86fa) (2.1.5) 2024-08-06T20:41:11.0766492Z Installing collected packages: torch 2024-08-06T20:41:24.0920629Z Successfully installed torch-2.5.0a0+gitb9d86fa 2024-08-06T20:41:24.1624365Z ++ dirname .ci/pytorch/test.sh 2024-08-06T20:41:24.1631467Z + source .ci/pytorch/common.sh 2024-08-06T20:41:24.1635051Z +++ dirname .ci/pytorch/common.sh 2024-08-06T20:41:24.1643011Z ++ source .ci/pytorch/common_utils.sh 2024-08-06T20:41:24.1645410Z +++ declare -f -t trap_add 2024-08-06T20:41:24.1652815Z ++ set -ex 2024-08-06T20:41:24.1655446Z ++ [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-08-06T20:41:24.1655976Z ++ BUILD_TEST_LIBTORCH=0 2024-08-06T20:41:24.1656706Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-08-06T20:41:24.1658014Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-06T20:41:24.1670337Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-06T20:41:24.1670753Z + trap_add cleanup_workspace EXIT 2024-08-06T20:41:24.1671155Z + trap_add_cmd=cleanup_workspace 2024-08-06T20:41:24.1671521Z + shift 2024-08-06T20:41:24.1671801Z + for trap_add_name in "$@" 2024-08-06T20:41:24.1679822Z +++ trap -p EXIT 2024-08-06T20:41:24.1682613Z ++ eval 'extract_trap_cmd ' 2024-08-06T20:41:24.1682979Z +++ extract_trap_cmd 2024-08-06T20:41:24.1683319Z +++ printf '%s\n' '' 2024-08-06T20:41:24.1683674Z ++ printf '%s\n' cleanup_workspace 2024-08-06T20:41:24.1686007Z + trap -- ' 2024-08-06T20:41:24.1686330Z cleanup_workspace' EXIT 2024-08-06T20:41:24.1686741Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-06T20:41:24.6996247Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-06T20:41:24.7013507Z + echo 'Environment variables:' 2024-08-06T20:41:24.7014014Z Environment variables: 2024-08-06T20:41:24.7014358Z + env 2024-08-06T20:41:24.7022537Z INSTALLED_DB=yes 2024-08-06T20:41:24.7023272Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-08-06T20:41:24.7023662Z NVIDIA_VISIBLE_DEVICES=all 2024-08-06T20:41:24.7024029Z NV_NVML_DEV_VERSION=11.8.86-1 2024-08-06T20:41:24.7024565Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-06T20:41:24.7025109Z CONTINUE_THROUGH_ERROR=False 2024-08-06T20:41:24.7025543Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-08-06T20:41:24.7026026Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-08-06T20:41:24.7026495Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-08-06T20:41:24.7026962Z HOSTNAME=0e2f9f76c86b 2024-08-06T20:41:24.7027672Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_7497a1d4-6367-4202-b78a-c3451f411b9a 2024-08-06T20:41:24.7028425Z GITHUB_ACTION=__self 2024-08-06T20:41:24.7028778Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-06T20:41:24.7030709Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-08-06T20:41:24.7032733Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-08-06T20:41:24.7033188Z NV_NVTX_VERSION=11.8.86-1 2024-08-06T20:41:24.7033550Z GITHUB_RUN_NUMBER=233985 2024-08-06T20:41:24.7033882Z TEST_CONFIG=distributed 2024-08-06T20:41:24.7034243Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-06T20:41:24.7034664Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-06T20:41:24.7035068Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-08-06T20:41:24.7035477Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-08-06T20:41:24.7035875Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-08-06T20:41:24.7036306Z GITHUB_TRIGGERING_ACTOR=drisspg 2024-08-06T20:41:24.7036746Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-06T20:41:24.7037204Z GITHUB_REF_TYPE=branch 2024-08-06T20:41:24.7037865Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-06T20:41:24.7038233Z NCCL_VERSION=2.16.2-1 2024-08-06T20:41:24.7038627Z BASE_SHA=1736af7cf736184c356be1bb00f59fb2feea6d7d 2024-08-06T20:41:24.7039039Z XLA_CUDA= 2024-08-06T20:41:24.7039334Z HUGGING_FACE_HUB_TOKEN= 2024-08-06T20:41:24.7039865Z *** 2024-08-06T20:41:24.7040153Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-06T20:41:24.7040582Z GITHUB_REPOSITORY_ID=65600975 2024-08-06T20:41:24.7040934Z GITHUB_ACTIONS=true 2024-08-06T20:41:24.7041278Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:41:24.7041688Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-08-06T20:41:24.7042164Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-08-06T20:41:24.7042616Z SHA1=b9d86fa89636e301796d4201f36d86c73f6e49bc 2024-08-06T20:41:24.7043048Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-06T20:41:24.7043689Z GITHUB_SHA=bf5bb5a1585a03379137fab341e87c02c77e76cd 2024-08-06T20:41:24.7044397Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/132710/merge 2024-08-06T20:41:24.7045002Z UCC_HOME=/usr 2024-08-06T20:41:24.7045391Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-08-06T20:41:24.7045793Z VERBOSE_TEST_LOGS=False 2024-08-06T20:41:24.7046129Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-06T20:41:24.7046541Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-08-06T20:41:24.7047001Z GITHUB_REF=refs/pull/132710/merge 2024-08-06T20:41:24.7047382Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-08-06T20:41:24.7047752Z SHARD_NUMBER=2 2024-08-06T20:41:24.7048072Z GITHUB_REF_PROTECTED=false 2024-08-06T20:41:24.7048512Z HOME=/var/lib/jenkins 2024-08-06T20:41:24.7048887Z GITHUB_API_URL=https://api.github.com 2024-08-06T20:41:24.7049325Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-06T20:41:24.7049763Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-06T20:41:24.7050228Z SCCACHE_S3_KEY_PREFIX=pull 2024-08-06T20:41:24.7050594Z CUDA_VERSION=11.8.0 2024-08-06T20:41:24.7050944Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-08-06T20:41:24.7051424Z NUM_TEST_SHARDS=3 2024-08-06T20:41:24.7051741Z UCX_HOME=/usr 2024-08-06T20:41:24.7052167Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-08-06T20:41:24.7053088Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_7497a1d4-6367-4202-b78a-c3451f411b9a 2024-08-06T20:41:24.7054139Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-08-06T20:41:24.7055125Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_7497a1d4-6367-4202-b78a-c3451f411b9a 2024-08-06T20:41:24.7056097Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-06T20:41:24.7056737Z GITHUB_EVENT_NAME=pull_request 2024-08-06T20:41:24.7057135Z DASHBOARD_TAG= 2024-08-06T20:41:24.7057430Z GITHUB_RUN_ID=10273124344 2024-08-06T20:41:24.7057839Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-08-06T20:41:24.7058634Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-08-06T20:41:24.7059470Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_7497a1d4-6367-4202-b78a-c3451f411b9a 2024-08-06T20:41:24.7060302Z GITHUB_ACTOR=drisspg 2024-08-06T20:41:24.7060648Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-08-06T20:41:24.7061010Z PR_NUMBER=132710 2024-08-06T20:41:24.7061328Z GITHUB_RUN_ATTEMPT=1 2024-08-06T20:41:24.7061672Z ANACONDA_PYTHON_VERSION=3.10 2024-08-06T20:41:24.7062100Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-06T20:41:24.7062553Z TERM=xterm 2024-08-06T20:41:24.7062873Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-08-06T20:41:24.7063263Z INSTALLED_VISION=yes 2024-08-06T20:41:24.7063598Z BRANCH=pull/132710 2024-08-06T20:41:24.7063923Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-06T20:41:24.7064326Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-06T20:41:24.7064751Z CUDA_PATH=/usr/local/cuda 2024-08-06T20:41:24.7065404Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-06T20:41:24.7066310Z GITHUB_SERVER_URL=https://github.com 2024-08-06T20:41:24.7066791Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-06T20:41:24.7067298Z REENABLED_ISSUES= 2024-08-06T20:41:24.7067613Z SHLVL=1 2024-08-06T20:41:24.7067891Z MAX_JOBS=30 2024-08-06T20:41:24.7068179Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-08-06T20:41:24.7068537Z NVARCH=x86_64 2024-08-06T20:41:24.7068849Z GITHUB_ACTOR_ID=32754868 2024-08-06T20:41:24.7069282Z GITHUB_WORKFLOW_SHA=bf5bb5a1585a03379137fab341e87c02c77e76cd 2024-08-06T20:41:24.7069798Z GITHUB_REF_NAME=132710/merge 2024-08-06T20:41:24.7070197Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-08-06T20:41:24.7070754Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-06T20:41:24.7071307Z GITHUB_JOB=test 2024-08-06T20:41:24.7071659Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-08-06T20:41:24.7072299Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-06T20:41:24.7072822Z NO_TEST_TIMEOUT=False 2024-08-06T20:41:24.7073163Z TD_DISTRIBUTED=False 2024-08-06T20:41:24.7073515Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-08-06T20:41:24.7073947Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-06T20:41:24.7074347Z NV_NVPROF_VERSION=11.8.87-1 2024-08-06T20:41:24.7074693Z GITHUB_RETENTION_DAYS=90 2024-08-06T20:41:24.7075048Z OPENSSL_DIR=/opt/openssl 2024-08-06T20:41:24.7075406Z GITHUB_ACTION_REPOSITORY= 2024-08-06T20:41:24.7076345Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-06T20:41:24.7077332Z GITHUB_BASE_REF=gh/drisspg/23/base 2024-08-06T20:41:24.7077743Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-06T20:41:24.7078089Z CI=true 2024-08-06T20:41:24.7078394Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-08-06T20:41:24.7078803Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-06T20:41:24.7079161Z JOB_ID=28428107750 2024-08-06T20:41:24.7079483Z INSTALLED_PROTOBUF=yes 2024-08-06T20:41:24.7079821Z GITHUB_HEAD_REF=gh/drisspg/23/head 2024-08-06T20:41:24.7080223Z GITHUB_ACTION_REF= 2024-08-06T20:41:24.7080612Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-06T20:41:24.7081052Z TEST_SHOWLOCALS=False 2024-08-06T20:41:24.7081393Z GITHUB_WORKFLOW=pull 2024-08-06T20:41:24.7081739Z DEBIAN_FRONTEND=noninteractive 2024-08-06T20:41:24.7082490Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_7497a1d4-6367-4202-b78a-c3451f411b9a 2024-08-06T20:41:24.7083269Z NO_TD=False 2024-08-06T20:41:24.7083584Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-06T20:41:24.7083931Z _=/usr/bin/env 2024-08-06T20:41:24.7084349Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-06T20:41:24.7243936Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-08-06T20:41:24.7244754Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-06T20:41:24.7245477Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-08-06T20:41:24.7246220Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-08-06T20:41:24.7246802Z + BUILD_DIR=build 2024-08-06T20:41:24.7247122Z + BUILD_RENAMED_DIR=build_renamed 2024-08-06T20:41:24.7247618Z + BUILD_BIN_DIR=build/bin 2024-08-06T20:41:24.7247972Z + SHARD_NUMBER=2 2024-08-06T20:41:24.7248276Z + NUM_TEST_SHARDS=3 2024-08-06T20:41:24.7248602Z + export VALGRIND=ON 2024-08-06T20:41:24.7248930Z + VALGRIND=ON 2024-08-06T20:41:24.7249288Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *clang9* ]] 2024-08-06T20:41:24.7249738Z + [[ 0 == \1 ]] 2024-08-06T20:41:24.7250023Z + [[ False == \1 ]] 2024-08-06T20:41:24.7250399Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *bazel* ]] 2024-08-06T20:41:24.7253312Z ++ realpath build/custom_test_artifacts 2024-08-06T20:41:24.7263234Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-06T20:41:24.7263853Z + [[ -n '' ]] 2024-08-06T20:41:24.7264181Z + echo 'Environment variables' 2024-08-06T20:41:24.7264568Z Environment variables 2024-08-06T20:41:24.7265258Z + env 2024-08-06T20:41:24.7271309Z INSTALLED_DB=yes 2024-08-06T20:41:24.7271691Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-08-06T20:41:24.7272066Z NVIDIA_VISIBLE_DEVICES=all 2024-08-06T20:41:24.7272435Z NV_NVML_DEV_VERSION=11.8.86-1 2024-08-06T20:41:24.7272961Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-06T20:41:24.7273506Z CONTINUE_THROUGH_ERROR=False 2024-08-06T20:41:24.7273938Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-08-06T20:41:24.7274416Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-08-06T20:41:24.7274881Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-08-06T20:41:24.7275343Z HOSTNAME=0e2f9f76c86b 2024-08-06T20:41:24.7276049Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_7497a1d4-6367-4202-b78a-c3451f411b9a 2024-08-06T20:41:24.7276967Z GITHUB_ACTION=__self 2024-08-06T20:41:24.7277352Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-06T20:41:24.7279268Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-08-06T20:41:24.7281275Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-08-06T20:41:24.7281735Z NV_NVTX_VERSION=11.8.86-1 2024-08-06T20:41:24.7282095Z GITHUB_RUN_NUMBER=233985 2024-08-06T20:41:24.7282453Z TEST_CONFIG=distributed 2024-08-06T20:41:24.7282795Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-06T20:41:24.7283224Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-06T20:41:24.7283651Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-08-06T20:41:24.7284059Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-08-06T20:41:24.7284465Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-08-06T20:41:24.7284830Z GITHUB_TRIGGERING_ACTOR=drisspg 2024-08-06T20:41:24.7285266Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-06T20:41:24.7285734Z GITHUB_REF_TYPE=branch 2024-08-06T20:41:24.7286069Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-06T20:41:24.7286440Z NCCL_VERSION=2.16.2-1 2024-08-06T20:41:24.7286915Z BASE_SHA=1736af7cf736184c356be1bb00f59fb2feea6d7d 2024-08-06T20:41:24.7287332Z XLA_CUDA= 2024-08-06T20:41:24.7287631Z HUGGING_FACE_HUB_TOKEN= 2024-08-06T20:41:24.7288081Z *** 2024-08-06T20:41:24.7288371Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-06T20:41:24.7288771Z GITHUB_REPOSITORY_ID=65600975 2024-08-06T20:41:24.7289145Z GITHUB_ACTIONS=true 2024-08-06T20:41:24.7289471Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:41:24.7289897Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-08-06T20:41:24.7290376Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-08-06T20:41:24.7290822Z SHA1=b9d86fa89636e301796d4201f36d86c73f6e49bc 2024-08-06T20:41:24.7291276Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-06T20:41:24.7291744Z GITHUB_SHA=bf5bb5a1585a03379137fab341e87c02c77e76cd 2024-08-06T20:41:24.7292412Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/132710/merge 2024-08-06T20:41:24.7293031Z UCC_HOME=/usr 2024-08-06T20:41:24.7293358Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-08-06T20:41:24.7293742Z VERBOSE_TEST_LOGS=False 2024-08-06T20:41:24.7294091Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-06T20:41:24.7294502Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-08-06T20:41:24.7294945Z GITHUB_REF=refs/pull/132710/merge 2024-08-06T20:41:24.7295343Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-08-06T20:41:24.7295717Z SHARD_NUMBER=2 2024-08-06T20:41:24.7296022Z GITHUB_REF_PROTECTED=false 2024-08-06T20:41:24.7296378Z HOME=/var/lib/jenkins 2024-08-06T20:41:24.7296747Z GITHUB_API_URL=https://api.github.com 2024-08-06T20:41:24.7297158Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-06T20:41:24.7297615Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-06T20:41:24.7298078Z SCCACHE_S3_KEY_PREFIX=pull 2024-08-06T20:41:24.7298751Z CUDA_VERSION=11.8.0 2024-08-06T20:41:24.7299305Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-08-06T20:41:24.7299738Z NUM_TEST_SHARDS=3 2024-08-06T20:41:24.7300029Z UCX_HOME=/usr 2024-08-06T20:41:24.7300515Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-08-06T20:41:24.7301440Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_7497a1d4-6367-4202-b78a-c3451f411b9a 2024-08-06T20:41:24.7302472Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-08-06T20:41:24.7303460Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_7497a1d4-6367-4202-b78a-c3451f411b9a 2024-08-06T20:41:24.7304435Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-06T20:41:24.7305073Z GITHUB_EVENT_NAME=pull_request 2024-08-06T20:41:24.7305575Z DASHBOARD_TAG= 2024-08-06T20:41:24.7305913Z GITHUB_RUN_ID=10273124344 2024-08-06T20:41:24.7306321Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-08-06T20:41:24.7306794Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-08-06T20:41:24.7307643Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_7497a1d4-6367-4202-b78a-c3451f411b9a 2024-08-06T20:41:24.7308487Z GITHUB_ACTOR=drisspg 2024-08-06T20:41:24.7308818Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-08-06T20:41:24.7309200Z PR_NUMBER=132710 2024-08-06T20:41:24.7309531Z GITHUB_RUN_ATTEMPT=1 2024-08-06T20:41:24.7309839Z VALGRIND=ON 2024-08-06T20:41:24.7310179Z ANACONDA_PYTHON_VERSION=3.10 2024-08-06T20:41:24.7310688Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-06T20:41:24.7311124Z TERM=xterm 2024-08-06T20:41:24.7311446Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-08-06T20:41:24.7311857Z INSTALLED_VISION=yes 2024-08-06T20:41:24.7312175Z BRANCH=pull/132710 2024-08-06T20:41:24.7312521Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-06T20:41:24.7312907Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-06T20:41:24.7313329Z CUDA_PATH=/usr/local/cuda 2024-08-06T20:41:24.7313997Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-06T20:41:24.7314710Z GITHUB_SERVER_URL=https://github.com 2024-08-06T20:41:24.7315181Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-06T20:41:24.7315642Z REENABLED_ISSUES= 2024-08-06T20:41:24.7315935Z SHLVL=1 2024-08-06T20:41:24.7316214Z MAX_JOBS=30 2024-08-06T20:41:24.7316523Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-08-06T20:41:24.7316869Z NVARCH=x86_64 2024-08-06T20:41:24.7317179Z GITHUB_ACTOR_ID=32754868 2024-08-06T20:41:24.7317630Z GITHUB_WORKFLOW_SHA=bf5bb5a1585a03379137fab341e87c02c77e76cd 2024-08-06T20:41:24.7318195Z GITHUB_REF_NAME=132710/merge 2024-08-06T20:41:24.7318591Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-08-06T20:41:24.7319183Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-06T20:41:24.7319715Z GITHUB_JOB=test 2024-08-06T20:41:24.7320066Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-08-06T20:41:24.7320592Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-06T20:41:24.7321073Z NO_TEST_TIMEOUT=False 2024-08-06T20:41:24.7321408Z TD_DISTRIBUTED=False 2024-08-06T20:41:24.7321763Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-08-06T20:41:24.7322171Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-06T20:41:24.7322566Z NV_NVPROF_VERSION=11.8.87-1 2024-08-06T20:41:24.7322928Z GITHUB_RETENTION_DAYS=90 2024-08-06T20:41:24.7323264Z OPENSSL_DIR=/opt/openssl 2024-08-06T20:41:24.7323622Z GITHUB_ACTION_REPOSITORY= 2024-08-06T20:41:24.7324565Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-06T20:41:24.7325545Z GITHUB_BASE_REF=gh/drisspg/23/base 2024-08-06T20:41:24.7325955Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-06T20:41:24.7326324Z CI=true 2024-08-06T20:41:24.7326617Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-08-06T20:41:24.7327028Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-06T20:41:24.7327396Z JOB_ID=28428107750 2024-08-06T20:41:24.7327813Z INSTALLED_PROTOBUF=yes 2024-08-06T20:41:24.7328259Z GITHUB_HEAD_REF=gh/drisspg/23/head 2024-08-06T20:41:24.7328629Z GITHUB_ACTION_REF= 2024-08-06T20:41:24.7329016Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-06T20:41:24.7329476Z TEST_SHOWLOCALS=False 2024-08-06T20:41:24.7329796Z GITHUB_WORKFLOW=pull 2024-08-06T20:41:24.7330179Z DEBIAN_FRONTEND=noninteractive 2024-08-06T20:41:24.7330948Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_7497a1d4-6367-4202-b78a-c3451f411b9a 2024-08-06T20:41:24.7331708Z NO_TD=False 2024-08-06T20:41:24.7332025Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-06T20:41:24.7332392Z _=/usr/bin/env 2024-08-06T20:41:24.7332690Z + echo 'Testing pytorch' 2024-08-06T20:41:24.7333040Z Testing pytorch 2024-08-06T20:41:24.7333366Z + export LANG=C.UTF-8 2024-08-06T20:41:24.7333772Z + LANG=C.UTF-8 2024-08-06T20:41:24.7334094Z + PR_NUMBER=132710 2024-08-06T20:41:24.7334435Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-08-06T20:41:24.7334850Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-06T20:41:24.7335323Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-08-06T20:41:24.7335788Z + [[ distributed == \s\l\o\w ]] 2024-08-06T20:41:24.7336289Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *slow-gradcheck* ]] 2024-08-06T20:41:24.7336914Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-08-06T20:41:24.7337401Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-06T20:41:24.7337843Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-06T20:41:24.7338593Z + [[ distributed == *crossref* ]] 2024-08-06T20:41:24.7339035Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-08-06T20:41:24.7339516Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *xpu* ]] 2024-08-06T20:41:24.7340051Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-08-06T20:41:24.7340546Z + pip_install --user ninja==1.10.2 2024-08-06T20:41:24.7340997Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-06T20:41:25.2593105Z Collecting ninja==1.10.2 2024-08-06T20:41:25.2782110Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-06T20:41:25.2957814Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-06T20:41:26.5160989Z Installing collected packages: ninja 2024-08-06T20:41:26.5259608Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-06T20:41:26.5260755Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-06T20:41:26.5313397Z Successfully installed ninja-1.10.2 2024-08-06T20:41:26.6019379Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-06T20:41:26.6021268Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-06T20:41:26.6022471Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-08-06T20:41:26.6022928Z + install_tlparse 2024-08-06T20:41:26.6023280Z + pip_install --user tlparse==0.3.7 2024-08-06T20:41:26.6023742Z + pip install --progress-bar off --user tlparse==0.3.7 2024-08-06T20:41:27.1139361Z Collecting tlparse==0.3.7 2024-08-06T20:41:27.1334488Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-08-06T20:41:27.1434887Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-06T20:41:28.4077366Z Installing collected packages: tlparse 2024-08-06T20:41:28.4572329Z Successfully installed tlparse-0.3.7 2024-08-06T20:41:28.5276189Z ++ python -m site --user-base 2024-08-06T20:41:28.5501751Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-06T20:41:28.5504143Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *asan* ]] 2024-08-06T20:41:28.5506447Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-debug* ]] 2024-08-06T20:41:28.5506990Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-08-06T20:41:28.5507680Z + echo 'We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass' 2024-08-06T20:41:28.5508538Z We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass 2024-08-06T20:41:28.5509141Z + cd test 2024-08-06T20:41:28.5509573Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-06T20:41:30.7074757Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-06T20:41:30.7075649Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-06T20:41:30.7080435Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-06T20:41:30.7080811Z + [[ distributed == *dynamo_eager* ]] 2024-08-06T20:41:30.7081247Z + [[ distributed == *aot_eager* ]] 2024-08-06T20:41:30.7081659Z + [[ distributed == *aot_inductor* ]] 2024-08-06T20:41:30.7082045Z + [[ distributed == *inductor* ]] 2024-08-06T20:41:30.7082434Z + [[ distributed == *dynamic* ]] 2024-08-06T20:41:30.7082813Z + [[ distributed == *cpu* ]] 2024-08-06T20:41:30.7083248Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-06T20:41:30.7124245Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-08-06T20:41:30.7124785Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-bazel-* ]] 2024-08-06T20:41:30.7127063Z + cd test 2024-08-06T20:41:30.7127521Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-06T20:41:32.6267068Z PyTorch built with: 2024-08-06T20:41:32.6267482Z - GCC 9.4 2024-08-06T20:41:32.6267791Z - C++ Version: 201703 2024-08-06T20:41:32.6268547Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-06T20:41:32.6269421Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-06T20:41:32.6270003Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-06T20:41:32.6270443Z - LAPACK is enabled (usually provided by MKL) 2024-08-06T20:41:32.6270863Z - NNPACK is enabled 2024-08-06T20:41:32.6271215Z - CPU capability usage: AVX2 2024-08-06T20:41:32.6271587Z - CUDA Runtime 11.8 2024-08-06T20:41:32.6272017Z - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52 2024-08-06T20:41:32.6272518Z - CuDNN 90.1 2024-08-06T20:41:32.6272821Z - Magma 2.6.1 2024-08-06T20:41:32.6278595Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-06T20:41:32.6284640Z 2024-08-06T20:41:32.8859689Z + cd test 2024-08-06T20:41:32.8860283Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-06T20:41:34.7671942Z ATen/Parallel: 2024-08-06T20:41:34.7672808Z at::get_num_threads() : 16 2024-08-06T20:41:34.7673200Z at::get_num_interop_threads() : 16 2024-08-06T20:41:34.7673622Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-06T20:41:34.7674017Z omp_get_max_threads() : 16 2024-08-06T20:41:34.7674704Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-06T20:41:34.7675447Z mkl_get_max_threads() : 16 2024-08-06T20:41:34.7675945Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-06T20:41:34.7676520Z std::thread::hardware_concurrency() : 32 2024-08-06T20:41:34.7676927Z Environment variables: 2024-08-06T20:41:34.7677277Z OMP_NUM_THREADS : [not set] 2024-08-06T20:41:34.7677629Z MKL_NUM_THREADS : [not set] 2024-08-06T20:41:34.7678004Z ATen parallel backend: OpenMP 2024-08-06T20:41:34.7678443Z 2024-08-06T20:41:35.0086758Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-08-06T20:41:35.0087311Z + [[ distributed == *backward* ]] 2024-08-06T20:41:35.0087750Z + [[ distributed == *xla* ]] 2024-08-06T20:41:35.0088109Z + [[ distributed == *executorch* ]] 2024-08-06T20:41:35.0088519Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-06T20:41:35.0089001Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-08-06T20:41:35.0089459Z + [[ distributed == distributed ]] 2024-08-06T20:41:35.0089845Z + test_distributed 2024-08-06T20:41:35.0090195Z + echo 'Testing distributed python tests' 2024-08-06T20:41:35.0090615Z Testing distributed python tests 2024-08-06T20:41:35.0091139Z + python test/run_test.py --distributed-tests --shard 2 3 --verbose 2024-08-06T20:41:35.1378739Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-06T20:41:35.1379786Z import pkg_resources 2024-08-06T20:41:39.6911243Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-08-06T20:41:39.7539600Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-06T20:41:39.8077037Z Ignoring disabled issues: [''] 2024-08-06T20:41:39.8218115Z Found test times from artifacts 2024-08-06T20:41:39.8754085Z Found test times from artifacts 2024-08-06T20:41:39.8773258Z Running 25% of tests based on TD 2024-08-06T20:41:39.8899843Z Running parallel tests on 2 processes 2024-08-06T20:41:39.8901470Z Name: tests to run (est. time: 6.54min) 2024-08-06T20:41:39.8902317Z Serial tests (14): 2024-08-06T20:41:39.8903068Z distributed/checkpoint/test_nested_dict 1/1 2024-08-06T20:41:39.8903992Z distributed/_tools/test_mod_tracker 1/1 2024-08-06T20:41:39.8905025Z distributed/tensor/parallel/test_micro_pipeline_tp 1/1 2024-08-06T20:41:39.8906100Z distributed/pipelining/test_transformer 1/1 2024-08-06T20:41:39.8906948Z distributed/_tensor/test_attention 1/1 2024-08-06T20:41:39.8907967Z distributed/pipelining/test_schedule_multiproc 1/1 2024-08-06T20:41:39.8908980Z distributed/_composable/test_contract 1/1 2024-08-06T20:41:39.8909835Z distributed/_tensor/test_dtensor 1/1 2024-08-06T20:41:39.8910793Z distributed/pipelining/test_composability 1/1 2024-08-06T20:41:39.8911792Z distributed/tensor/parallel/test_tp_style 1/1 2024-08-06T20:41:39.8912715Z distributed/pipelining/test_microbatch 1/1 2024-08-06T20:41:39.8913672Z distributed/tensor/parallel/test_tp_examples 1/1 2024-08-06T20:41:39.8914649Z distributed/checkpoint/test_traverse 1/1 2024-08-06T20:41:39.8915499Z distributed/test_device_mesh 1/1 2024-08-06T20:41:39.8916300Z Parallel tests (0): 2024-08-06T20:41:39.8917039Z Name: excluded (est. time: 81.36min) 2024-08-06T20:41:39.8917809Z Serial tests (54): 2024-08-06T20:41:39.8918514Z distributed/test_distributed_spawn 2/10 2024-08-06T20:41:39.8919423Z distributed/test_distributed_spawn 5/10 2024-08-06T20:41:39.8920293Z distributed/test_distributed_spawn 8/10 2024-08-06T20:41:39.8921494Z distributed/test_c10d_gloo 1/1 2024-08-06T20:41:39.8922306Z distributed/fsdp/test_fsdp_core 1/2 2024-08-06T20:41:39.8923171Z distributed/fsdp/test_fsdp_state_dict 2/2 2024-08-06T20:41:39.8924123Z distributed/fsdp/test_fsdp_mixed_precision 1/1 2024-08-06T20:41:39.8925116Z distributed/fsdp/test_fsdp_use_orig_params 1/1 2024-08-06T20:41:39.8926055Z distributed/test_c10d_pypg 1/1 2024-08-06T20:41:39.8926930Z distributed/fsdp/test_fsdp_freezing_weights 1/1 2024-08-06T20:41:39.8927905Z distributed/fsdp/test_fsdp_comm_hooks 1/1 2024-08-06T20:41:39.8928790Z distributed/test_c10d_common 1/1 2024-08-06T20:41:39.8929614Z distributed/fsdp/test_fsdp_checkpoint 1/1 2024-08-06T20:41:39.8930534Z distributed/fsdp/test_fsdp_dtensor_state_dict 1/1 2024-08-06T20:41:39.8931793Z distributed/fsdp/test_fsdp_ignored_modules 1/1 2024-08-06T20:41:39.8932822Z distributed/fsdp/test_fsdp_flatten_params 1/1 2024-08-06T20:41:39.8933762Z distributed/fsdp/test_fsdp_comm 1/1 2024-08-06T20:41:39.8934638Z distributed/fsdp/test_fsdp_overlap 1/1 2024-08-06T20:41:39.8935508Z distributed/fsdp/test_fsdp_grad_acc 1/1 2024-08-06T20:41:39.8936396Z distributed/launcher/test_run 1/1 2024-08-06T20:41:39.8937222Z distributed/fsdp/test_fsdp_meta 1/1 2024-08-06T20:41:39.8938178Z distributed/fsdp/test_fsdp_clip_grad_norm 1/1 2024-08-06T20:41:39.8939620Z distributed/_composable/test_replicate 1/1 2024-08-06T20:41:39.8940635Z distributed/checkpoint/test_dtensor_resharding 1/1 2024-08-06T20:41:39.8941883Z distributed/_composable/test_composability/test_2d_composability 1/1 2024-08-06T20:41:39.8943227Z distributed/_composable/fsdp/test_fully_shard_autograd 1/1 2024-08-06T20:41:39.8944534Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 2024-08-06T20:41:39.8945834Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 2024-08-06T20:41:39.8946890Z distributed/_tensor/test_embedding_ops 1/1 2024-08-06T20:41:39.8947789Z distributed/fsdp/test_fsdp_memory 1/1 2024-08-06T20:41:39.8948818Z distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 2024-08-06T20:41:39.8949871Z distributed/_shard/sharding_spec/test_sharding_spec 1/1 2024-08-06T20:41:39.8950463Z distributed/_shard/sharding_plan/test_sharding_plan 1/1 2024-08-06T20:41:39.8951103Z distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 1/1 2024-08-06T20:41:39.8951756Z distributed/checkpoint/test_fsdp_optim_state 1/1 2024-08-06T20:41:39.8952684Z distributed/fsdp/test_shard_utils 1/1 2024-08-06T20:41:39.8953718Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 1/1 2024-08-06T20:41:39.8954953Z distributed/_shard/sharded_optim/test_sharded_optim 1/1 2024-08-06T20:41:39.8956027Z distributed/test_inductor_collectives 1/1 2024-08-06T20:41:39.8957014Z distributed/elastic/utils/distributed_test 1/1 2024-08-06T20:41:39.8958073Z distributed/fsdp/test_fsdp_multiple_wrapping 1/1 2024-08-06T20:41:39.8959145Z distributed/checkpoint/test_dtensor_checkpoint 1/1 2024-08-06T20:41:39.8960119Z distributed/fsdp/test_fsdp_uneven 1/1 2024-08-06T20:41:39.8961017Z distributed/rpc/test_share_memory 1/1 2024-08-06T20:41:39.8962012Z distributed/tensor/parallel/test_tp_random_state 1/1 2024-08-06T20:41:39.8962975Z distributed/test_launcher 1/1 2024-08-06T20:41:39.8963858Z distributed/fsdp/test_checkpoint_wrapper 1/1 2024-08-06T20:41:39.8964804Z distributed/_tensor/test_op_strategy 1/1 2024-08-06T20:41:39.8965667Z distributed/fsdp/test_utils 1/1 2024-08-06T20:41:39.8966463Z distributed/test_control_collectives 1/1 2024-08-06T20:41:39.8967362Z distributed/_tools/test_memory_tracker 1/1 2024-08-06T20:41:39.8968266Z distributed/elastic/metrics/api_test 1/1 2024-08-06T20:41:39.8969167Z distributed/test_collective_utils 1/1 2024-08-06T20:41:39.8969949Z distributed/rpc/test_faulty_agent 1/1 2024-08-06T20:41:39.8970358Z Parallel tests (0): 2024-08-06T20:41:39.8970844Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-08-06T20:41:39.8996140Z Running distributed/checkpoint/test_nested_dict 1/1 ... [2024-08-06 20:41:39.899192] 2024-08-06T20:41:39.9001816Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_nested_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-06 20:41:39.899642] 2024-08-06T20:41:43.9730363Z 2024-08-06T20:41:43.9731882Z distributed/checkpoint/test_nested_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_nested_dict_1.1_e2fe543bae8dde0d_.log 2024-08-06T20:41:43.9734178Z Running 2 items in this shard: test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_flattening_round_trip, test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_mapping 2024-08-06T20:41:43.9735285Z 2024-08-06T20:41:43.9737773Z Running distributed/_tools/test_mod_tracker 1/1 ... [2024-08-06 20:41:43.973440] 2024-08-06T20:41:43.9742588Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_mod_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-06 20:41:43.973886] 2024-08-06T20:41:47.7958246Z 2024-08-06T20:41:47.7959680Z distributed/_tools/test_mod_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_mod_tracker_1.1_5285fa00ee09f7dd_.log 2024-08-06T20:41:47.7962202Z Running 4 items in this shard: test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_ac, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_bw_detection, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_module_hierarchy, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_user_hooks 2024-08-06T20:41:47.7963962Z 2024-08-06T20:41:47.7964383Z Running distributed/tensor/parallel/test_micro_pipeline_tp 1/1 ... [2024-08-06 20:41:47.796049] 2024-08-06T20:41:47.7968471Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_micro_pipeline_tp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-06 20:41:47.796455] 2024-08-06T20:41:54.8273829Z 2024-08-06T20:41:54.8274926Z distributed/tensor/parallel/test_micro_pipeline_tp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_micro_pipeline_tp_1.1_fedac08571bf0e1e_.log 2024-08-06T20:41:54.8294845Z Running 29 items in this shard: test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_all_gather_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_reduce_scatter_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_get_unexposed_collectives 2024-08-06T20:41:54.8314734Z 2024-08-06T20:41:54.8315104Z Running distributed/pipelining/test_transformer 1/1 ... [2024-08-06 20:41:54.827701] 2024-08-06T20:41:54.8316711Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_transformer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-06 20:41:54.828977] 2024-08-06T20:42:00.0037101Z 2024-08-06T20:42:00.0038391Z distributed/pipelining/test_transformer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_transformer_1.1_df2a49ef5473ba29_.log 2024-08-06T20:42:00.0039896Z Running 1 items in this shard: test/distributed/pipelining/test_transformer.py::TransformerTests::test_ir 2024-08-06T20:42:00.0040524Z 2024-08-06T20:42:00.0042474Z Running distributed/_tensor/test_attention 1/1 ... [2024-08-06 20:42:00.003951] 2024-08-06T20:42:00.0046954Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_attention.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-06 20:42:00.004366] 2024-08-06T20:42:07.8348434Z 2024-08-06T20:42:07.8350141Z distributed/_tensor/test_attention 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_attention_1.1_3b9cfd6aa6e85816_.log 2024-08-06T20:42:07.8356368Z Running 8 items in this shard: test/distributed/_tensor/test_attention.py::RingAttentionTest::test_is_causal_behavior, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_compile_attention_fn0, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_compile_attention_fn1, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_custom_transformer, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_native_transformer_is_causal_False, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_native_transformer_is_causal_True, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa_is_causal_False, test/distributed/_tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa_is_causal_True 2024-08-06T20:42:07.8360984Z 2024-08-06T20:42:07.8361394Z Running distributed/pipelining/test_schedule_multiproc 1/1 ... [2024-08-06 20:42:07.835084] 2024-08-06T20:42:07.8363036Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_schedule_multiproc.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-06 20:42:07.835491] 2024-08-06T20:42:21.1819312Z 2024-08-06T20:42:21.1821900Z distributed/pipelining/test_schedule_multiproc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_schedule_multiproc_1.1_f476203a35882f2b_.log 2024-08-06T20:42:21.1839059Z Running 14 items in this shard: test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_forward_only_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass0_ModelClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass1_ModelClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_weight_update_mlp_e2e_ScheduleClass0 2024-08-06T20:42:21.1856591Z Running 14 items in this shard: test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_forward_only_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass0_ModelClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass1_ModelClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_weight_update_mlp_e2e_ScheduleClass0 2024-08-06T20:42:21.1865454Z 2024-08-06T20:42:21.1865806Z Running distributed/_composable/test_contract 1/1 ... [2024-08-06 20:42:21.182616] 2024-08-06T20:42:21.1867358Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_contract.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-06 20:42:21.183238] 2024-08-06T20:42:25.1056932Z 2024-08-06T20:42:25.1059646Z distributed/_composable/test_contract 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_contract_1.1_60364e3fd68aedca_.log 2024-08-06T20:42:25.1065539Z Running 5 items in this shard: test/distributed/_composable/test_contract.py::TestContract::test_add_hooks, test/distributed/_composable/test_contract.py::TestContract::test_modify_fqn, test/distributed/_composable/test_contract.py::TestContract::test_multi_module_api, test/distributed/_composable/test_contract.py::TestContract::test_registry, test/distributed/_composable/test_contract.py::TestContract::test_state 2024-08-06T20:42:25.1069483Z 2024-08-06T20:42:25.1070121Z Running distributed/_tensor/test_dtensor 1/1 ... [2024-08-06 20:42:25.106177] 2024-08-06T20:42:25.1073068Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-06 20:42:25.106803] 2024-08-06T20:44:42.6253596Z 2024-08-06T20:44:42.6256877Z distributed/_tensor/test_dtensor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_1.1_57b9cc40a325c81d_.log 2024-08-06T20:44:42.6271121Z Running 32 items in this shard: test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_async_output, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_constructor, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_new_empty_strided, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_properties, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_save_load, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_spec_hash, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_spec_read_only_after_set, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_stride, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_negative_dim, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_then_to_local, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_uneven_sharding, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_uneven_sharding_raise_error, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_full_tensor_grad_hint, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_full_tensor_sync, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_meta_dtensor, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_modules_w_meta_dtensor, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_to_local, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_to_local_grad_hint, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_auto_implicit_replication, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_default_value_sub_mesh, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_device_mesh_nd, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_2d_mesh, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_api_device_mesh_context_manager, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_device_mesh_device_conversion, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_spec_local_shard_offset, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_from_local_sub_mesh, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_implicit_replication, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_metadata_consistency_check, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_redistribute_sub_mesh, test/distributed/_tensor/test_dtensor.py::TestDTensorPlacementTypes::test_split_tensor_1D, test/distributed/_tensor/test_dtensor.py::DTensorLogTest::test_dtensor_log 2024-08-06T20:44:42.6284425Z 2024-08-06T20:44:42.6284828Z Running distributed/pipelining/test_composability 1/1 ... [2024-08-06 20:44:42.625669] 2024-08-06T20:44:42.6286466Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_composability.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-06 20:44:42.626168] 2024-08-06T20:44:46.8162807Z 2024-08-06T20:44:46.8164486Z distributed/pipelining/test_composability 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_composability_1.1_7f5aff52a5b472cb_.log 2024-08-06T20:44:46.8165598Z 2024-08-06T20:44:46.8168393Z Running distributed/tensor/parallel/test_tp_style 1/1 ... [2024-08-06 20:44:46.816529] 2024-08-06T20:44:46.8173126Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_style.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-06 20:44:46.816955] 2024-08-06T20:45:32.1782067Z 2024-08-06T20:45:32.1785867Z distributed/tensor/parallel/test_tp_style 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_style_1.1_0b1813c8ccd0c9c0_.log 2024-08-06T20:45:32.1791952Z Running 9 items in this shard: test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_colwise_parallel_embedding, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_colwise_parallel_style, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_input, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_input_multiple_inputs, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_kwargs_input, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_output, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_rowwise_parallel_embedding, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_rowwise_parallel_style, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_sequence_parallel_style 2024-08-06T20:45:32.1801515Z 2024-08-06T20:45:32.1802295Z Running distributed/pipelining/test_microbatch 1/1 ... [2024-08-06 20:45:32.178478] 2024-08-06T20:45:32.1805802Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_microbatch.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-06 20:45:32.178902] 2024-08-06T20:45:36.9529268Z 2024-08-06T20:45:36.9530696Z distributed/pipelining/test_microbatch 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_microbatch_1.1_8d16471a4fa1695a_.log 2024-08-06T20:45:36.9532663Z Running 2 items in this shard: test/distributed/pipelining/test_microbatch.py::MicrobatchTests::test_chunk_spec, test/distributed/pipelining/test_microbatch.py::MicrobatchTests::test_split_and_merge 2024-08-06T20:45:36.9533709Z 2024-08-06T20:45:36.9535970Z Running distributed/tensor/parallel/test_tp_examples 1/1 ... [2024-08-06 20:45:36.953216] 2024-08-06T20:45:36.9541205Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_examples.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-06 20:45:36.953640] 2024-08-06T20:46:28.8741509Z 2024-08-06T20:46:28.8744744Z distributed/tensor/parallel/test_tp_examples 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_examples_1.1_cb1fad4dc3196dc7_.log 2024-08-06T20:46:28.8751954Z Running 9 items in this shard: test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_loss_parallel, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_inference, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_training_is_seq_parallel_False_recompute_activation_False, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_training_is_seq_parallel_True_recompute_activation_False, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_False_float32, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_False_float64, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_True_float32, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_True_float64, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_weight_tying 2024-08-06T20:46:28.8758511Z 2024-08-06T20:46:28.8758862Z Running distributed/checkpoint/test_traverse 1/1 ... [2024-08-06 20:46:28.874380] 2024-08-06T20:46:28.8760411Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_traverse.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-06 20:46:28.874830] 2024-08-06T20:46:32.7970532Z 2024-08-06T20:46:32.7971989Z distributed/checkpoint/test_traverse 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_traverse_1.1_11307c65fb58dc1a_.log 2024-08-06T20:46:32.7976178Z Running 7 items in this shard: test/distributed/checkpoint/test_traverse.py::TestTraverse::test_get_element, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_set_element, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_doesnt_ignore_intermediate_collections, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_nested_dict, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_nested_list, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_shallow, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_with_ordered_dict 2024-08-06T20:46:32.7980159Z 2024-08-06T20:46:32.7980470Z Running distributed/test_device_mesh 1/1 ... [2024-08-06 20:46:32.797291] 2024-08-06T20:46:32.7981920Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_device_mesh.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-06 20:46:32.797703] 2024-08-06T20:49:07.5730209Z 2024-08-06T20:49:07.5733619Z distributed/test_device_mesh 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_device_mesh_1.1_721358dba9515433_.log 2024-08-06T20:49:07.5751167Z Running 39 items in this shard: test/distributed/test_device_mesh.py::DeviceMeshTest::test_assert_invalid_mesh_tensor, test/distributed/test_device_mesh.py::DeviceMeshTest::test_device_mesh_2d, test/distributed/test_device_mesh.py::DeviceMeshTest::test_device_mesh_init_backend, test/distributed/test_device_mesh.py::DeviceMeshTest::test_fake_pg_device_mesh, test/distributed/test_device_mesh.py::DeviceMeshTest::test_from_group_with_global_pg, test/distributed/test_device_mesh.py::DeviceMeshTest::test_from_group_with_invalid_mesh, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_group_and_get_all_groups, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_local_rank, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_local_rank_raises_exception, test/distributed/test_device_mesh.py::DeviceMeshTest::test_init_process_group, test/distributed/test_device_mesh.py::DeviceMeshTest::test_raises_invalid_device_type, test/distributed/test_device_mesh.py::DeviceMeshTest::test_set_mesh_dim_group_options, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_hash, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_nd, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_parent_child_hash, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_from_group_with_mesh_shape, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_get_local_rank_3d, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_init_device_mesh, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_raises_duplicate_mesh_dim_names, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_raises_mesh_shape_mesh_dim_names_mismatch, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_cache_and_reuse_submesh_slice_result, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_1d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_2d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_3d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_3d_noncontinuous_slicing, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_invalid_mesh_dim_name, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_no_mesh_dim_found, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_mesh_dim_by_name, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_parent_mesh, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_parent_mesh_dim_exist, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_parent_mesh_dim_not_exist, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_all_gather_uneven, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_1d, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_nd, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_reduce_scatter_contiguous, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_reduce_scatter_uneven, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_1d, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_nd, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_uneven 2024-08-06T20:49:07.5767510Z 2024-08-06T20:49:08.5371221Z 2024-08-06T20:49:08.5371962Z real 7m33.528s 2024-08-06T20:49:08.5372341Z user 25m38.929s 2024-08-06T20:49:08.5372637Z sys 7m35.604s 2024-08-06T20:49:08.5372951Z + assert_git_not_dirty 2024-08-06T20:49:08.5373365Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-08-06T20:49:08.5373864Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *xla* ]] 2024-08-06T20:49:08.5377201Z ++ git status --porcelain 2024-08-06T20:49:08.5377603Z ++ grep -v '?? third_party' 2024-08-06T20:49:11.4232700Z ++ true 2024-08-06T20:49:11.4234496Z + git_status= 2024-08-06T20:49:11.4234835Z + [[ -n '' ]] 2024-08-06T20:49:11.4235218Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-08-06T20:49:11.4235675Z + [[ 2 == 1 ]] 2024-08-06T20:49:11.4236237Z + [[ 2 == 1 ]] 2024-08-06T20:49:11.4236779Z + cleanup_workspace 2024-08-06T20:49:11.4237691Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-06T20:49:11.4239627Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-06T20:49:11.4241144Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-06T20:49:11.4242350Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-06T20:49:11.4243708Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-06T20:49:11.4245314Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-06T20:49:11.4246590Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-06T20:49:11.9764381Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-06T20:49:11.9764894Z cat test/**/*_toprint.log || true 2024-08-06T20:49:11.9772866Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:49:11.9773352Z env: 2024-08-06T20:49:11.9773627Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:49:11.9774059Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:49:11.9774784Z DOCKER_CONTAINER_ID: 0e2f9f76c86bfdf6b10ad25e0546516e8d36c6bdfb223c138546808c0d6a4ac6 2024-08-06T20:49:11.9775390Z ##[endgroup] 2024-08-06T20:49:11.9846489Z cat: test/**/*_toprint.log: No such file or directory 2024-08-06T20:49:11.9886505Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-06T20:49:11.9887011Z kill "$MONITOR_SCRIPT_PID" 2024-08-06T20:49:11.9894444Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:49:11.9894939Z env: 2024-08-06T20:49:11.9895236Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:49:11.9895659Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:49:11.9896365Z DOCKER_CONTAINER_ID: 0e2f9f76c86bfdf6b10ad25e0546516e8d36c6bdfb223c138546808c0d6a4ac6 2024-08-06T20:49:11.9897006Z MONITOR_SCRIPT_PID: 66923 2024-08-06T20:49:11.9897346Z ##[endgroup] 2024-08-06T20:49:12.0066081Z Prepare all required actions 2024-08-06T20:49:12.0066631Z Getting action download info 2024-08-06T20:49:12.2260837Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-06T20:49:12.4040106Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-06T20:49:12.4040572Z with: 2024-08-06T20:49:12.4041029Z file-suffix: test-distributed-2-3-linux.8xlarge.nvidia.gpu_28428107750 2024-08-06T20:49:12.4041614Z s3-bucket: gha-artifacts 2024-08-06T20:49:12.4041965Z env: 2024-08-06T20:49:12.4042235Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:49:12.4042667Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:49:12.4043369Z DOCKER_CONTAINER_ID: 0e2f9f76c86bfdf6b10ad25e0546516e8d36c6bdfb223c138546808c0d6a4ac6 2024-08-06T20:49:12.4043975Z ##[endgroup] 2024-08-06T20:49:12.4083089Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-06T20:49:12.4083737Z # Remove any previous test jsons if they exist 2024-08-06T20:49:12.4084204Z rm -f test-jsons-*.zip 2024-08-06T20:49:12.4084692Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-06T20:49:12.4092671Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:49:12.4093137Z env: 2024-08-06T20:49:12.4093433Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:49:12.4093870Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:49:12.4094552Z DOCKER_CONTAINER_ID: 0e2f9f76c86bfdf6b10ad25e0546516e8d36c6bdfb223c138546808c0d6a4ac6 2024-08-06T20:49:12.4095354Z FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_28428107750 2024-08-06T20:49:12.4095914Z ##[endgroup] 2024-08-06T20:49:12.4339202Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-06T20:49:12.4374065Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-06T20:49:12.4374715Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-06T20:49:12.4382457Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-06T20:49:12.4384864Z adding: test/test-reports/td_exclusions-0e3121455e44aca204d1.json (deflated 86%) 2024-08-06T20:49:12.4385569Z adding: test/.pytorch-slow-tests.json (deflated 66%) 2024-08-06T20:49:12.4401157Z adding: test/.pytorch-disabled-tests.json (deflated 89%) 2024-08-06T20:49:12.4437661Z ##[group]Run # Remove any previous test reports if they exist 2024-08-06T20:49:12.4438261Z # Remove any previous test reports if they exist 2024-08-06T20:49:12.4438741Z rm -f test-reports-*.zip 2024-08-06T20:49:12.4439293Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-06T20:49:12.4447119Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:49:12.4447610Z env: 2024-08-06T20:49:12.4447910Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:49:12.4448332Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:49:12.4449048Z DOCKER_CONTAINER_ID: 0e2f9f76c86bfdf6b10ad25e0546516e8d36c6bdfb223c138546808c0d6a4ac6 2024-08-06T20:49:12.4449855Z FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_28428107750 2024-08-06T20:49:12.4450413Z ##[endgroup] 2024-08-06T20:49:12.4665720Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_nested_dict/distributed.checkpoint.test_nested_dict-e2a39c348acc17c7.xml (deflated 51%) 2024-08-06T20:49:12.4667291Z adding: test/test-reports/python-pytest/distributed._tools.test_mod_tracker/distributed._tools.test_mod_tracker-6455bccfd5e3d267.xml (deflated 64%) 2024-08-06T20:49:12.4668938Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_micro_pipeline_tp/distributed.tensor.parallel.test_micro_pipeline_tp-e5ad0167f321e67b.xml (deflated 95%) 2024-08-06T20:49:12.4670675Z adding: test/test-reports/python-pytest/distributed.pipelining.test_transformer/distributed.pipelining.test_transformer-11adf80140722ad3.xml (deflated 40%) 2024-08-06T20:49:12.4672494Z adding: test/test-reports/python-pytest/distributed._tensor.test_attention/distributed._tensor.test_attention-7c66cf05785ae04f.xml (deflated 86%) 2024-08-06T20:49:12.4674153Z adding: test/test-reports/python-pytest/distributed.pipelining.test_schedule_multiproc/distributed.pipelining.test_schedule_multiproc-d539c5d197b972d5.xml (deflated 82%) 2024-08-06T20:49:12.4675925Z adding: test/test-reports/python-pytest/distributed.pipelining.test_schedule_multiproc/distributed.pipelining.test_schedule_multiproc-84931e5179a0a0b4.xml (deflated 82%) 2024-08-06T20:49:12.4677590Z adding: test/test-reports/python-pytest/distributed._composable.test_contract/distributed._composable.test_contract-cc75920f7b7c09de.xml (deflated 69%) 2024-08-06T20:49:12.4679310Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor/distributed._tensor.test_dtensor-8abab9ab6b2688c7.xml (deflated 82%) 2024-08-06T20:49:12.4680897Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_style/distributed.tensor.parallel.test_tp_style-18be9dec8f580820.xml (deflated 77%) 2024-08-06T20:49:12.4702479Z adding: test/test-reports/python-pytest/distributed.pipelining.test_microbatch/distributed.pipelining.test_microbatch-533b78b947d8a940.xml (deflated 54%) 2024-08-06T20:49:12.4704411Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_examples/distributed.tensor.parallel.test_tp_examples-0363f33a5043a524.xml (deflated 78%) 2024-08-06T20:49:12.4706049Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_traverse/distributed.checkpoint.test_traverse-42191b795ede9304.xml (deflated 75%) 2024-08-06T20:49:12.4707507Z adding: test/test-reports/python-pytest/distributed.test_device_mesh/distributed.test_device_mesh-f2639127fbda6406.xml (deflated 85%) 2024-08-06T20:49:12.4741180Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-06T20:49:12.4741763Z # Remove any previous usage logs if they exist 2024-08-06T20:49:12.4742238Z rm -f logs-*.zip 2024-08-06T20:49:12.4742827Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-06T20:49:12.4743507Z # so check to see if the file exists first 2024-08-06T20:49:12.4743982Z if [ -f 'usage_log.txt' ]; then 2024-08-06T20:49:12.4744449Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-06T20:49:12.4744909Z fi 2024-08-06T20:49:12.4745263Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-06T20:49:12.4745767Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-06T20:49:12.4746224Z fi 2024-08-06T20:49:12.4753310Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:49:12.4753782Z env: 2024-08-06T20:49:12.4754083Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:49:12.4754531Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:49:12.4755209Z DOCKER_CONTAINER_ID: 0e2f9f76c86bfdf6b10ad25e0546516e8d36c6bdfb223c138546808c0d6a4ac6 2024-08-06T20:49:12.4756018Z FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_28428107750 2024-08-06T20:49:12.4756583Z ##[endgroup] 2024-08-06T20:49:12.4801575Z adding: usage_log.txt (deflated 92%) 2024-08-06T20:49:12.5025820Z adding: test/test-reports/distributed.checkpoint.test_nested_dict_1.1_e2fe543bae8dde0d_.log (deflated 59%) 2024-08-06T20:49:12.5026858Z adding: test/test-reports/distributed._tools.test_mod_tracker_1.1_5285fa00ee09f7dd_.log (deflated 64%) 2024-08-06T20:49:12.5027987Z adding: test/test-reports/distributed.tensor.parallel.test_micro_pipeline_tp_1.1_fedac08571bf0e1e_.log (deflated 90%) 2024-08-06T20:49:12.5029122Z adding: test/test-reports/distributed.pipelining.test_transformer_1.1_df2a49ef5473ba29_.log (deflated 55%) 2024-08-06T20:49:12.5030157Z adding: test/test-reports/distributed._tensor.test_attention_1.1_3b9cfd6aa6e85816_.log (deflated 76%) 2024-08-06T20:49:12.5031409Z adding: test/test-reports/distributed.pipelining.test_schedule_multiproc_1.1_f476203a35882f2b_.log (deflated 89%) 2024-08-06T20:49:12.5032698Z adding: test/test-reports/distributed._composable.test_contract_1.1_60364e3fd68aedca_.log (deflated 67%) 2024-08-06T20:49:12.5035282Z adding: test/test-reports/distributed._tensor.test_dtensor_1.1_57b9cc40a325c81d_.log (deflated 85%) 2024-08-06T20:49:12.5036326Z adding: test/test-reports/distributed.pipelining.test_composability_1.1_7f5aff52a5b472cb_.log (deflated 3%) 2024-08-06T20:49:12.5037394Z adding: test/test-reports/distributed.tensor.parallel.test_tp_style_1.1_0b1813c8ccd0c9c0_.log (deflated 76%) 2024-08-06T20:49:12.5038454Z adding: test/test-reports/distributed.pipelining.test_microbatch_1.1_8d16471a4fa1695a_.log (deflated 58%) 2024-08-06T20:49:12.5039539Z adding: test/test-reports/distributed.tensor.parallel.test_tp_examples_1.1_cb1fad4dc3196dc7_.log (deflated 78%) 2024-08-06T20:49:12.5040622Z adding: test/test-reports/distributed.checkpoint.test_traverse_1.1_11307c65fb58dc1a_.log (deflated 71%) 2024-08-06T20:49:12.5041593Z adding: test/test-reports/distributed.test_device_mesh_1.1_721358dba9515433_.log (deflated 83%) 2024-08-06T20:49:12.5081230Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-06T20:49:12.5082174Z # Remove any previous debugging artifacts if they exist 2024-08-06T20:49:12.5082671Z rm -f debug-*.zip 2024-08-06T20:49:12.5083044Z if [ -d 'test/debug' ]; then 2024-08-06T20:49:12.5083498Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-06T20:49:12.5083926Z fi 2024-08-06T20:49:12.5091043Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:49:12.5091644Z env: 2024-08-06T20:49:12.5091952Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:49:12.5092383Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:49:12.5093083Z DOCKER_CONTAINER_ID: 0e2f9f76c86bfdf6b10ad25e0546516e8d36c6bdfb223c138546808c0d6a4ac6 2024-08-06T20:49:12.5093907Z FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_28428107750 2024-08-06T20:49:12.5094442Z ##[endgroup] 2024-08-06T20:49:12.5186300Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-06T20:49:12.5186733Z with: 2024-08-06T20:49:12.5187026Z s3-bucket: gha-artifacts 2024-08-06T20:49:12.5187436Z s3-prefix: pytorch/pytorch/10273124344/1/artifact 2024-08-06T20:49:12.5187903Z retention-days: 14 2024-08-06T20:49:12.5188227Z if-no-files-found: warn 2024-08-06T20:49:12.5188588Z path: test-jsons-*.zip 2024-08-06T20:49:12.5188937Z name: artifact 2024-08-06T20:49:12.5189286Z region: us-east-1 2024-08-06T20:49:12.5189605Z env: 2024-08-06T20:49:12.5189891Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:49:12.5190317Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:49:12.5191054Z DOCKER_CONTAINER_ID: 0e2f9f76c86bfdf6b10ad25e0546516e8d36c6bdfb223c138546808c0d6a4ac6 2024-08-06T20:49:12.5191720Z ##[endgroup] 2024-08-06T20:49:12.9614375Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-06T20:49:12.9615386Z With the provided path, there will be 1 file uploaded 2024-08-06T20:49:12.9615999Z Uploading to s3 prefix: pytorch/pytorch/10273124344/1/artifact 2024-08-06T20:49:12.9663186Z Starting upload of test-jsons-test-distributed-2-3-linux.8xlarge.nvidia.gpu_28428107750.zip 2024-08-06T20:49:13.0807551Z Finished upload of test-jsons-test-distributed-2-3-linux.8xlarge.nvidia.gpu_28428107750.zip 2024-08-06T20:49:13.0970012Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-06T20:49:13.0970447Z with: 2024-08-06T20:49:13.0970728Z s3-bucket: gha-artifacts 2024-08-06T20:49:13.0971147Z s3-prefix: pytorch/pytorch/10273124344/1/artifact 2024-08-06T20:49:13.0971608Z retention-days: 14 2024-08-06T20:49:13.0971934Z if-no-files-found: error 2024-08-06T20:49:13.0972304Z path: test-reports-*.zip 2024-08-06T20:49:13.0972657Z name: artifact 2024-08-06T20:49:13.0972949Z region: us-east-1 2024-08-06T20:49:13.0973259Z env: 2024-08-06T20:49:13.0973553Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:49:13.0973974Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:49:13.0974882Z DOCKER_CONTAINER_ID: 0e2f9f76c86bfdf6b10ad25e0546516e8d36c6bdfb223c138546808c0d6a4ac6 2024-08-06T20:49:13.0975541Z ##[endgroup] 2024-08-06T20:49:13.5385608Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-06T20:49:13.5386722Z With the provided path, there will be 1 file uploaded 2024-08-06T20:49:13.5387855Z Uploading to s3 prefix: pytorch/pytorch/10273124344/1/artifact 2024-08-06T20:49:13.5437492Z Starting upload of test-reports-test-distributed-2-3-linux.8xlarge.nvidia.gpu_28428107750.zip 2024-08-06T20:49:13.6664328Z Finished upload of test-reports-test-distributed-2-3-linux.8xlarge.nvidia.gpu_28428107750.zip 2024-08-06T20:49:13.6827867Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-06T20:49:13.6828297Z with: 2024-08-06T20:49:13.6828603Z s3-bucket: gha-artifacts 2024-08-06T20:49:13.6829026Z s3-prefix: pytorch/pytorch/10273124344/1/artifact 2024-08-06T20:49:13.6829463Z retention-days: 14 2024-08-06T20:49:13.6829804Z if-no-files-found: ignore 2024-08-06T20:49:13.6830182Z path: logs-*.zip 2024-08-06T20:49:13.6830484Z name: artifact 2024-08-06T20:49:13.6830798Z region: us-east-1 2024-08-06T20:49:13.6831112Z env: 2024-08-06T20:49:13.6831590Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:49:13.6832032Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:49:13.6832726Z DOCKER_CONTAINER_ID: 0e2f9f76c86bfdf6b10ad25e0546516e8d36c6bdfb223c138546808c0d6a4ac6 2024-08-06T20:49:13.6833333Z ##[endgroup] 2024-08-06T20:49:14.1258658Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-06T20:49:14.1259782Z With the provided path, there will be 1 file uploaded 2024-08-06T20:49:14.1260381Z Uploading to s3 prefix: pytorch/pytorch/10273124344/1/artifact 2024-08-06T20:49:14.1307257Z Starting upload of logs-test-distributed-2-3-linux.8xlarge.nvidia.gpu_28428107750.zip 2024-08-06T20:49:14.2602832Z Finished upload of logs-test-distributed-2-3-linux.8xlarge.nvidia.gpu_28428107750.zip 2024-08-06T20:49:14.2768484Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-06T20:49:14.2768924Z with: 2024-08-06T20:49:14.2769223Z s3-bucket: gha-artifacts 2024-08-06T20:49:14.2769648Z s3-prefix: pytorch/pytorch/10273124344/1/artifact 2024-08-06T20:49:14.2770106Z retention-days: 14 2024-08-06T20:49:14.2770448Z if-no-files-found: ignore 2024-08-06T20:49:14.2770813Z path: debug-*.zip 2024-08-06T20:49:14.2771112Z name: artifact 2024-08-06T20:49:14.2771418Z region: us-east-1 2024-08-06T20:49:14.2771726Z env: 2024-08-06T20:49:14.2771997Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:49:14.2772431Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:49:14.2773127Z DOCKER_CONTAINER_ID: 0e2f9f76c86bfdf6b10ad25e0546516e8d36c6bdfb223c138546808c0d6a4ac6 2024-08-06T20:49:14.2773733Z ##[endgroup] 2024-08-06T20:49:14.7216464Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-06T20:49:14.7382440Z ##[group]Run # shellcheck disable=SC2156 2024-08-06T20:49:14.7382961Z # shellcheck disable=SC2156 2024-08-06T20:49:14.7383693Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-06T20:49:14.7391916Z shell: /usr/bin/bash -e {0} 2024-08-06T20:49:14.7392284Z env: 2024-08-06T20:49:14.7392562Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:49:14.7393001Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:49:14.7393701Z DOCKER_CONTAINER_ID: 0e2f9f76c86bfdf6b10ad25e0546516e8d36c6bdfb223c138546808c0d6a4ac6 2024-08-06T20:49:14.7394325Z ##[endgroup] 2024-08-06T20:49:15.0387066Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-06T20:49:15.0387645Z with: 2024-08-06T20:49:15.0387907Z env: 2024-08-06T20:49:15.0388195Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:49:15.0388634Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:49:15.0389314Z DOCKER_CONTAINER_ID: 0e2f9f76c86bfdf6b10ad25e0546516e8d36c6bdfb223c138546808c0d6a4ac6 2024-08-06T20:49:15.0390003Z ##[endgroup] 2024-08-06T20:49:15.0416105Z ##[group]Run set -eou pipefail 2024-08-06T20:49:15.0416517Z set -eou pipefail 2024-08-06T20:49:15.0416852Z  2024-08-06T20:49:15.0417339Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-06T20:49:15.0417927Z for _ in $(seq 1440); do 2024-08-06T20:49:15.0418658Z  # Break if no ssh session exists anymore 2024-08-06T20:49:15.0419104Z  if [ "$(who)" = "" ]; then 2024-08-06T20:49:15.0419515Z  break 2024-08-06T20:49:15.0419887Z  fi 2024-08-06T20:49:15.0420180Z  echo "." 2024-08-06T20:49:15.0420508Z  sleep 5 2024-08-06T20:49:15.0420829Z done 2024-08-06T20:49:15.0428269Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:49:15.0428758Z env: 2024-08-06T20:49:15.0429037Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:49:15.0429477Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:49:15.0430176Z DOCKER_CONTAINER_ID: 0e2f9f76c86bfdf6b10ad25e0546516e8d36c6bdfb223c138546808c0d6a4ac6 2024-08-06T20:49:15.0430797Z ##[endgroup] 2024-08-06T20:49:15.0453746Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-06T20:49:15.0507657Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-06T20:49:15.0508350Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-06T20:49:15.0508876Z # shellcheck disable=SC2046 2024-08-06T20:49:15.0509322Z docker stop $(docker ps -q) || true 2024-08-06T20:49:15.0509771Z # Prune all of the docker images 2024-08-06T20:49:15.0510179Z docker system prune -af 2024-08-06T20:49:15.0517426Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:49:15.0517908Z env: 2024-08-06T20:49:15.0518184Z GIT_DEFAULT_BRANCH: main 2024-08-06T20:49:15.0518622Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-06T20:49:15.0519321Z DOCKER_CONTAINER_ID: 0e2f9f76c86bfdf6b10ad25e0546516e8d36c6bdfb223c138546808c0d6a4ac6 2024-08-06T20:49:15.0519925Z ##[endgroup] 2024-08-06T20:49:15.3983860Z 0e2f9f76c86b 2024-08-06T20:49:15.9766625Z Deleted Containers: 2024-08-06T20:49:15.9767368Z 0e2f9f76c86bfdf6b10ad25e0546516e8d36c6bdfb223c138546808c0d6a4ac6 2024-08-06T20:49:15.9767856Z 2024-08-06T20:49:22.6447525Z Deleted Images: 2024-08-06T20:49:22.6448857Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-06T20:49:22.6450668Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9@sha256:66181a134b3ebc35388dabcc46dc11e767152fd72514dad76711590f50b650f7 2024-08-06T20:49:22.6452097Z deleted: sha256:8e4e19ae0377c273bc603b24bc49d406d6f755072d4534d040c86afbe761a129 2024-08-06T20:49:22.6453072Z deleted: sha256:bee5df81b6fe1ee0aae7e536a9bda16351d0515be5b789ad50bf0ac60fd41e33 2024-08-06T20:49:22.6471838Z deleted: sha256:8ee1ee099b65b2d73d66dc3330f82f8be2dd6cbdccf3d718b448a5e3419c214e 2024-08-06T20:49:22.6472948Z deleted: sha256:aed7424cdc4428e32e398a33e98e6e252075e6280e7068e20df7626451fa312a 2024-08-06T20:49:22.6473738Z deleted: sha256:61b7d2994ed16481c3de07839b5a3bb779dc2f3f716ad49acb91537042c3b7d1 2024-08-06T20:49:22.6474532Z deleted: sha256:79999dfec9b5df258486b014aa99c30979900c4131effd2a7bc18251cd5fc372 2024-08-06T20:49:22.6475352Z deleted: sha256:9020e3bab07b431af26c788ac2d12cbca9a76ffb1c923ea45a38e872942e1f34 2024-08-06T20:49:22.6476376Z deleted: sha256:387e2f2f359df5f69c500ae9589c224d8283ae5418520509b3c9ef0a45d116fa 2024-08-06T20:49:22.6477213Z deleted: sha256:5ddfc73c14e2e0a762c51ddecfb2710258366e1d048a8bf56d70268486907b99 2024-08-06T20:49:22.6478015Z deleted: sha256:4dffb251634d1c8105e68310ef1b2ffb923dac4c4139f494be0938ad032a1bca 2024-08-06T20:49:22.6478787Z deleted: sha256:e0df7617281f0e4396253d2f0c40d1418ad604dc0912cae13f153fe22e316fa0 2024-08-06T20:49:22.6479583Z deleted: sha256:d239b3ccf05c2850364c3f73acbc6e0017a25f8faa4c3abaa8524e63dc9f9c29 2024-08-06T20:49:22.6480387Z deleted: sha256:918d0912faf4251f3be58f9a5aaa80f59a180ace4771b9ce903cedf94a57723a 2024-08-06T20:49:22.6481151Z deleted: sha256:09d5696392736fa88c52a0656d1d831d2bcb4015630810412565a5a086eb98ff 2024-08-06T20:49:22.6481933Z deleted: sha256:67542af70388f69b345fc7f0965e1fce3c552ee987689f804c8406536f16534f 2024-08-06T20:49:22.6482721Z deleted: sha256:350337bafbf2976e8d4e0b7ce3cc66b2cb43ebb8622fa6e538b02f561f929393 2024-08-06T20:49:22.6483599Z deleted: sha256:00fe9fe12fefe10d5dbfb2f855aa385e840f360deeccbc9ee92cae51c6eea4ba 2024-08-06T20:49:22.6484414Z deleted: sha256:9895af0586ff7d94936462a92d7cec0d8ec3b01df27b664f7e4ac277ce7bbe9e 2024-08-06T20:49:22.6485190Z deleted: sha256:7dd7dbc3ec792cf9d043ff867afe169d15fe414138097ad9247a5f388d3eeccf 2024-08-06T20:49:22.6485991Z deleted: sha256:e0a3ace1b87c67d71f5db76d211740431660a7fbcd80415333c9e7b7b9a2e89b 2024-08-06T20:49:22.6486776Z deleted: sha256:b9c56b5040d5d5998961b788563b85e5be12f678bc6b3976d089a81f1c93b7e8 2024-08-06T20:49:22.6487543Z deleted: sha256:9232fe8591973938b7bdb0a7a31dcdbc79f9667d6e011a2facf4e17fb77d4922 2024-08-06T20:49:22.6488668Z deleted: sha256:52f5fbe778d735123327825c6a16dcec9bc84c2623087bb4c7c094ac99764ae7 2024-08-06T20:49:22.6489686Z deleted: sha256:4b7c4121b76b04d5c7e4a0b77add435c0caa99cccff506ea704eba27fd2b2166 2024-08-06T20:49:22.6490525Z deleted: sha256:795dd765ace5b617808fc40cd576b449ae01b8671290e6df6b49d86ea79f4145 2024-08-06T20:49:22.6491328Z deleted: sha256:51cc651f1ffbeef49ad010ab2d4cf2d5ce5f8f5767c2133346a959c773d837bb 2024-08-06T20:49:22.6492122Z deleted: sha256:4c3019549b40f7a268d344eae162d7a7cc78855880db865944da136e01c190e9 2024-08-06T20:49:22.6492949Z deleted: sha256:855b789ec056edaab8cc8dad50a82e9dbfb9b20bcf39c70d9c3bd40080b2b65f 2024-08-06T20:49:22.6493756Z deleted: sha256:cef853cb7e0f5d6351692a4b9a0413a2496ea3c7bfe5d8063528ac2d6c17696f 2024-08-06T20:49:22.6494545Z deleted: sha256:4f7b163ef5ff693ce9f7380bdf7b526131f2567489342f0630949c9341e1c63f 2024-08-06T20:49:22.6495332Z deleted: sha256:09f12e5e102232b4ec9fd5168f4aa4b20ee0dfc755b053ac57690657e72e3fe3 2024-08-06T20:49:22.6496109Z deleted: sha256:c9962be393e03353af0d6a26f5478a94cacb5e528ee3f97609fac594c9a5d387 2024-08-06T20:49:22.6496920Z deleted: sha256:ac7aa4ec25b0d9a035c438effaa309071729de3f3a0f96e8e481b9839e7460b8 2024-08-06T20:49:22.6497714Z deleted: sha256:63ff2106018988888d022ee38d6ce1805aa29d8368a18dec1ab8eb587799ab44 2024-08-06T20:49:22.6498805Z deleted: sha256:3298c1acdeb701f11351366cfe680a84b8290666fdcd76af5ac79f226422fc0d 2024-08-06T20:49:22.6499612Z deleted: sha256:f076cc29eff5dae969bc7c05003d680b6e683eaef971c0a3f67a8a4d12693ee4 2024-08-06T20:49:22.6500400Z deleted: sha256:2604655e98c557e6de2e650d21cd0306c01f12204f3538386a9f374d003a4d70 2024-08-06T20:49:22.6501159Z deleted: sha256:fb942194d6d4fc6f9c8a4c8cba17330563877c92369892edc9b005b8b491f204 2024-08-06T20:49:22.6501950Z deleted: sha256:d349cc387708a540eee839ccd78494458dc5f1ccf989db608d25c5f798ec8c76 2024-08-06T20:49:22.6502761Z deleted: sha256:269864865f9b12ee399ca9ee3e15a16cedb86fd4889e01a83567afb357a774e6 2024-08-06T20:49:22.6503536Z deleted: sha256:236a91ce743cd500427338a78915ff834ae80331fcc06afccfca82a35bffb63c 2024-08-06T20:49:22.6504350Z deleted: sha256:de968bbc1d8bc2ad420d6a1c78a9a6cf8a763bb855ca3aefe7de6c5c52201f7a 2024-08-06T20:49:22.6505179Z deleted: sha256:0f15ecc513ffdac89965bd76cd9f8c7f4e37bbe7be0bfa086353133e2dd4f086 2024-08-06T20:49:22.6505981Z deleted: sha256:754bb9b5a5af40b6ad88f0ab14be37b9e512e25edccbfc9fd5e0c5a88ca09c0f 2024-08-06T20:49:22.6506951Z deleted: sha256:427104437dfa1883feec2c441eb4e4f825481883c6d3d31759361cbc924a27fe 2024-08-06T20:49:22.6507768Z deleted: sha256:0b404434cf2c3087ef28ea3cdc5813cc555f6e261c2514a7ab5bf5d484b5b575 2024-08-06T20:49:22.6508573Z deleted: sha256:bfb91e062bab9dafb63d0d4ab366026692fbd63e642da9ab28f04c8bba9b56a3 2024-08-06T20:49:22.6509353Z deleted: sha256:54617b36a396e9e82f2092257d2f621c90de460e0e54b1cf8df065573fbaa0ba 2024-08-06T20:49:22.6510151Z deleted: sha256:d8bdc337db1750bb3ebc7c8e4e1bd48e722218f261cb35ec2056ceaa6d3c4e9c 2024-08-06T20:49:22.6510987Z deleted: sha256:25cbacf4be7b2bbcbeaf7375d0eff0eaa33e94c5bef20ff0993bf074fdaa54eb 2024-08-06T20:49:22.6511789Z deleted: sha256:43744f05dcf6dc44c2f5e06533b13e085319a78b6b5e17eba5e7bd4419f5c905 2024-08-06T20:49:22.6512597Z deleted: sha256:2ea7d74f63d16a8d6e14a87e0e916f67f99536b21ac53e80b27786dcd1ce36fd 2024-08-06T20:49:22.6513396Z deleted: sha256:dbfb6031f18e50c89ff1a268c1e80e2fa88e727c48656718b326b60e8ef0d383 2024-08-06T20:49:22.6514191Z deleted: sha256:3eb3bb1755fcc3d4aba9abe2d2270ed2a55ec6254b3e87ca7688370e67e707a3 2024-08-06T20:49:22.6515004Z deleted: sha256:04605aba2d51924a85499584edda24c28390eb777c1c88eff2f4191b48e3da8c 2024-08-06T20:49:22.6515800Z deleted: sha256:117ff9e5eddba4582a39d416380eadf1c245237612c5ae0d6781d36c36bcb87f 2024-08-06T20:49:22.6516586Z deleted: sha256:ebf86c84eafa12a81b8eb66541a56c8056f4f9bc30e4be761771e53c6e1342aa 2024-08-06T20:49:22.6517392Z deleted: sha256:23d7f7e0c9d7e60acb59e60da9b4ee4e1840c1c406bf2082822c49430975529d 2024-08-06T20:49:22.6518178Z deleted: sha256:3e7e5583645195af5eeb60fe49024a4aa2f0eb3be28ff95c9859bb6233c29c4b 2024-08-06T20:49:22.6518981Z deleted: sha256:45b8f0dd78e72f0156b1fbc4dcb14200c61954f37a75129c753275e5a45ccd9b 2024-08-06T20:49:22.6519904Z deleted: sha256:f6b7a042384e2f2bb3401da2b9f4e9b9f942adb70be7e7f89c46fa81a42774d8 2024-08-06T20:49:22.6520716Z deleted: sha256:eb547a7b8fe732bffe8cc9a66aa19f2b41f6e0616a8a27040307c022d00fbc0d 2024-08-06T20:49:22.6521518Z deleted: sha256:20f729283bb4f2063f3a8baa0b2afc17911da345bf367e1918d8cc09ddd6d964 2024-08-06T20:49:22.6522306Z deleted: sha256:8ce8da9ab0319625a66c35d24ddadf8d2688d16c5024ba2345099431e0aa7ff8 2024-08-06T20:49:22.6523117Z deleted: sha256:e7a91181c74f879360530f0bbe957b8ff96a1ec19adde8a41d9e3fa2038d0d16 2024-08-06T20:49:22.6523906Z deleted: sha256:212b7a0fad7208624a2bc56768d8c489d2690142d0db0fa20f69eb648234e8d1 2024-08-06T20:49:22.6524679Z deleted: sha256:72315e37f51ba8bcc88fc5b56f92e5bb6264456d9a755afa01fd19f9128c1df7 2024-08-06T20:49:22.6525489Z deleted: sha256:bfc5ab07cfab1698147152988eabdc7ebbdeb35b6d9aefd9360dd64104cd3888 2024-08-06T20:49:22.6526292Z deleted: sha256:89edf8c906678c217b9a88160e1561991f53909c3fd27a4a5e1f0ca0f7ff7888 2024-08-06T20:49:22.6527096Z deleted: sha256:d5ba67f8f9b0c2b6da06d2d5d9d5fedc9f17cbe1a5e13a1a4033a4619f4528ba 2024-08-06T20:49:22.6527907Z deleted: sha256:5d61311e1a293c0c9642e34871719525dfeec7028e4b45e08344a469e3fe24bf 2024-08-06T20:49:22.6528699Z deleted: sha256:b89bb70f745a94c0edd5e974aee294e160735f76fd666a1a06a0b9b1be3caf63 2024-08-06T20:49:22.6529488Z deleted: sha256:9e5477f1f21a38c3dfe21a56a7d77d1b1a42453aed3e11729afcbb9d5731770b 2024-08-06T20:49:22.6530297Z deleted: sha256:b76f4af91cffe2bb7b3f09960f5bdbb26f741419defda65c451c4407fc5b0a92 2024-08-06T20:49:22.6531110Z deleted: sha256:3f8748a00ebd394c0e9d9d481972d764dff58896841ef0150b3f7efaac0f4a4a 2024-08-06T20:49:22.6531890Z deleted: sha256:13989c771e3ba4d40f618344e55e5cb7062a35041b5efd69e2f740ba819f46ee 2024-08-06T20:49:22.6532671Z deleted: sha256:9ebb27837f6c67ad8c17e9af4e63c008faae03fa226c3641dd8bbbfeefb106c4 2024-08-06T20:49:22.6533467Z deleted: sha256:8c16907d735950f8093582bc3cf392b70e002456bf9c4224de5a5b86b78b3ea2 2024-08-06T20:49:22.6534249Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-08-06T20:49:22.6534738Z 2024-08-06T20:49:22.6534897Z Total reclaimed space: 26.03GB 2024-08-06T20:49:22.6603227Z Post job cleanup. 2024-08-06T20:49:22.6664607Z Post job cleanup. 2024-08-06T20:49:22.7836680Z [command]/usr/bin/git version 2024-08-06T20:49:22.7890526Z git version 2.40.1 2024-08-06T20:49:22.7955877Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/453fb883-8314-4ad9-869a-a9b6e096b1d4' before making global git config changes 2024-08-06T20:49:22.7957366Z Adding repository directory to the temporary git global config as a safe directory 2024-08-06T20:49:22.7964469Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-06T20:49:22.8008525Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-06T20:49:22.8046713Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-06T20:49:22.8365671Z Entering 'android/libs/fbjni' 2024-08-06T20:49:22.8417102Z Entering 'third_party/FP16' 2024-08-06T20:49:22.8469576Z Entering 'third_party/FXdiv' 2024-08-06T20:49:22.8521129Z Entering 'third_party/NNPACK' 2024-08-06T20:49:22.8572476Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-06T20:49:22.8624579Z Entering 'third_party/XNNPACK' 2024-08-06T20:49:22.8695397Z Entering 'third_party/benchmark' 2024-08-06T20:49:22.8747619Z Entering 'third_party/cpp-httplib' 2024-08-06T20:49:22.8798151Z Entering 'third_party/cpuinfo' 2024-08-06T20:49:22.8849802Z Entering 'third_party/cudnn_frontend' 2024-08-06T20:49:22.8903042Z Entering 'third_party/cutlass' 2024-08-06T20:49:22.8962842Z Entering 'third_party/eigen' 2024-08-06T20:49:22.9016386Z Entering 'third_party/fbgemm' 2024-08-06T20:49:22.9068909Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-06T20:49:22.9117799Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-06T20:49:22.9169083Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-06T20:49:22.9226533Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-06T20:49:22.9275622Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-06T20:49:22.9326844Z Entering 'third_party/flatbuffers' 2024-08-06T20:49:22.9381587Z Entering 'third_party/fmt' 2024-08-06T20:49:22.9432909Z Entering 'third_party/foxi' 2024-08-06T20:49:22.9484406Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-06T20:49:22.9536046Z Entering 'third_party/gloo' 2024-08-06T20:49:22.9589330Z Entering 'third_party/googletest' 2024-08-06T20:49:22.9640315Z Entering 'third_party/ideep' 2024-08-06T20:49:22.9690150Z Entering 'third_party/ideep/mkl-dnn' 2024-08-06T20:49:22.9749309Z Entering 'third_party/ittapi' 2024-08-06T20:49:22.9800217Z Entering 'third_party/kineto' 2024-08-06T20:49:22.9851255Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-06T20:49:22.9902095Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-06T20:49:22.9955586Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-06T20:49:23.0006889Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-06T20:49:23.0058769Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-06T20:49:23.0110525Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-06T20:49:23.0165097Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-06T20:49:23.0217073Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-06T20:49:23.0268207Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-06T20:49:23.0322551Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-06T20:49:23.0374705Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-06T20:49:23.0426249Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-06T20:49:23.0478231Z Entering 'third_party/mimalloc' 2024-08-06T20:49:23.0532377Z Entering 'third_party/nccl/nccl' 2024-08-06T20:49:23.0585053Z Entering 'third_party/nlohmann' 2024-08-06T20:49:23.0638582Z Entering 'third_party/onnx' 2024-08-06T20:49:23.0709672Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-06T20:49:23.0762640Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-06T20:49:23.0817192Z Entering 'third_party/opentelemetry-cpp' 2024-08-06T20:49:23.0870345Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-06T20:49:23.0921247Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-06T20:49:23.0973028Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-06T20:49:23.1023899Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-06T20:49:23.1076789Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-06T20:49:23.1126628Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-06T20:49:23.1176846Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-06T20:49:23.1226876Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-06T20:49:23.1278775Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-06T20:49:23.1331791Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-06T20:49:23.1405318Z Entering 'third_party/pocketfft' 2024-08-06T20:49:23.1456629Z Entering 'third_party/protobuf' 2024-08-06T20:49:23.1511787Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-06T20:49:23.1561996Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-06T20:49:23.1614623Z Entering 'third_party/psimd' 2024-08-06T20:49:23.1665648Z Entering 'third_party/pthreadpool' 2024-08-06T20:49:23.1717771Z Entering 'third_party/pybind11' 2024-08-06T20:49:23.1769538Z Entering 'third_party/python-peachpy' 2024-08-06T20:49:23.1821355Z Entering 'third_party/sleef' 2024-08-06T20:49:23.1872582Z Entering 'third_party/tensorpipe' 2024-08-06T20:49:23.1923217Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-06T20:49:23.1973914Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-06T20:49:23.2024529Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-06T20:49:23.2074085Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-06T20:49:23.2123714Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-06T20:49:23.2197185Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-06T20:49:23.2223441Z http.https://github.com/.extraheader 2024-08-06T20:49:23.2235823Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-06T20:49:23.2272732Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-06T20:49:23.2581140Z Entering 'android/libs/fbjni' 2024-08-06T20:49:23.2614430Z http.https://github.com/.extraheader 2024-08-06T20:49:23.2648570Z Entering 'third_party/FP16' 2024-08-06T20:49:23.2682142Z http.https://github.com/.extraheader 2024-08-06T20:49:23.2716415Z Entering 'third_party/FXdiv' 2024-08-06T20:49:23.2749568Z http.https://github.com/.extraheader 2024-08-06T20:49:23.2783029Z Entering 'third_party/NNPACK' 2024-08-06T20:49:23.2816836Z http.https://github.com/.extraheader 2024-08-06T20:49:23.2851030Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-06T20:49:23.2884376Z http.https://github.com/.extraheader 2024-08-06T20:49:23.2918708Z Entering 'third_party/XNNPACK' 2024-08-06T20:49:23.2952502Z http.https://github.com/.extraheader 2024-08-06T20:49:23.3004601Z Entering 'third_party/benchmark' 2024-08-06T20:49:23.3036809Z http.https://github.com/.extraheader 2024-08-06T20:49:23.3070129Z Entering 'third_party/cpp-httplib' 2024-08-06T20:49:23.3103394Z http.https://github.com/.extraheader 2024-08-06T20:49:23.3136348Z Entering 'third_party/cpuinfo' 2024-08-06T20:49:23.3169580Z http.https://github.com/.extraheader 2024-08-06T20:49:23.3203560Z Entering 'third_party/cudnn_frontend' 2024-08-06T20:49:23.3236097Z http.https://github.com/.extraheader 2024-08-06T20:49:23.3270681Z Entering 'third_party/cutlass' 2024-08-06T20:49:23.3303648Z http.https://github.com/.extraheader 2024-08-06T20:49:23.3346596Z Entering 'third_party/eigen' 2024-08-06T20:49:23.3381706Z http.https://github.com/.extraheader 2024-08-06T20:49:23.3417265Z Entering 'third_party/fbgemm' 2024-08-06T20:49:23.3450690Z http.https://github.com/.extraheader 2024-08-06T20:49:23.3485679Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-06T20:49:23.3518271Z http.https://github.com/.extraheader 2024-08-06T20:49:23.3552205Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-06T20:49:23.3584966Z http.https://github.com/.extraheader 2024-08-06T20:49:23.3620696Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-06T20:49:23.3651098Z http.https://github.com/.extraheader 2024-08-06T20:49:23.3693058Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-06T20:49:23.3725571Z http.https://github.com/.extraheader 2024-08-06T20:49:23.3760044Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-06T20:49:23.3792656Z http.https://github.com/.extraheader 2024-08-06T20:49:23.3827237Z Entering 'third_party/flatbuffers' 2024-08-06T20:49:23.3859664Z http.https://github.com/.extraheader 2024-08-06T20:49:23.3896914Z Entering 'third_party/fmt' 2024-08-06T20:49:23.3930310Z http.https://github.com/.extraheader 2024-08-06T20:49:23.3964295Z Entering 'third_party/foxi' 2024-08-06T20:49:23.3997534Z http.https://github.com/.extraheader 2024-08-06T20:49:23.4031903Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-06T20:49:23.4064302Z http.https://github.com/.extraheader 2024-08-06T20:49:23.4097718Z Entering 'third_party/gloo' 2024-08-06T20:49:23.4130507Z http.https://github.com/.extraheader 2024-08-06T20:49:23.4165381Z Entering 'third_party/googletest' 2024-08-06T20:49:23.4198194Z http.https://github.com/.extraheader 2024-08-06T20:49:23.4231841Z Entering 'third_party/ideep' 2024-08-06T20:49:23.4265001Z http.https://github.com/.extraheader 2024-08-06T20:49:23.4297234Z Entering 'third_party/ideep/mkl-dnn' 2024-08-06T20:49:23.4330036Z http.https://github.com/.extraheader 2024-08-06T20:49:23.4372827Z Entering 'third_party/ittapi' 2024-08-06T20:49:23.4406863Z http.https://github.com/.extraheader 2024-08-06T20:49:23.4440630Z Entering 'third_party/kineto' 2024-08-06T20:49:23.4474640Z http.https://github.com/.extraheader 2024-08-06T20:49:23.4508619Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-06T20:49:23.4542050Z http.https://github.com/.extraheader 2024-08-06T20:49:23.4575521Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-06T20:49:23.4608819Z http.https://github.com/.extraheader 2024-08-06T20:49:23.4644598Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-06T20:49:23.4678248Z http.https://github.com/.extraheader 2024-08-06T20:49:23.4713317Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-06T20:49:23.4745494Z http.https://github.com/.extraheader 2024-08-06T20:49:23.4780380Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-06T20:49:23.4812746Z http.https://github.com/.extraheader 2024-08-06T20:49:23.4846218Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-06T20:49:23.4878736Z http.https://github.com/.extraheader 2024-08-06T20:49:23.4914841Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-06T20:49:23.4948176Z http.https://github.com/.extraheader 2024-08-06T20:49:23.4983156Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-06T20:49:23.5015223Z http.https://github.com/.extraheader 2024-08-06T20:49:23.5049302Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-06T20:49:23.5082058Z http.https://github.com/.extraheader 2024-08-06T20:49:23.5117957Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-06T20:49:23.5150083Z http.https://github.com/.extraheader 2024-08-06T20:49:23.5185708Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-06T20:49:23.5219242Z http.https://github.com/.extraheader 2024-08-06T20:49:23.5252565Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-06T20:49:23.5284956Z http.https://github.com/.extraheader 2024-08-06T20:49:23.5320153Z Entering 'third_party/mimalloc' 2024-08-06T20:49:23.5354779Z http.https://github.com/.extraheader 2024-08-06T20:49:23.5389207Z Entering 'third_party/nccl/nccl' 2024-08-06T20:49:23.5422474Z http.https://github.com/.extraheader 2024-08-06T20:49:23.5456282Z Entering 'third_party/nlohmann' 2024-08-06T20:49:23.5490385Z http.https://github.com/.extraheader 2024-08-06T20:49:23.5525559Z Entering 'third_party/onnx' 2024-08-06T20:49:23.5559959Z http.https://github.com/.extraheader 2024-08-06T20:49:23.5611443Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-06T20:49:23.5644458Z http.https://github.com/.extraheader 2024-08-06T20:49:23.5678784Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-06T20:49:23.5712147Z http.https://github.com/.extraheader 2024-08-06T20:49:23.5748869Z Entering 'third_party/opentelemetry-cpp' 2024-08-06T20:49:23.5782418Z http.https://github.com/.extraheader 2024-08-06T20:49:23.5818056Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-06T20:49:23.5851095Z http.https://github.com/.extraheader 2024-08-06T20:49:23.5885031Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-06T20:49:23.5917104Z http.https://github.com/.extraheader 2024-08-06T20:49:23.5950966Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-06T20:49:23.5983171Z http.https://github.com/.extraheader 2024-08-06T20:49:23.6016613Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-06T20:49:23.6048885Z http.https://github.com/.extraheader 2024-08-06T20:49:23.6087555Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-06T20:49:23.6119263Z http.https://github.com/.extraheader 2024-08-06T20:49:23.6153234Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-06T20:49:23.6186329Z http.https://github.com/.extraheader 2024-08-06T20:49:23.6219407Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-06T20:49:23.6251525Z http.https://github.com/.extraheader 2024-08-06T20:49:23.6284688Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-06T20:49:23.6317343Z http.https://github.com/.extraheader 2024-08-06T20:49:23.6353035Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-06T20:49:23.6386494Z http.https://github.com/.extraheader 2024-08-06T20:49:23.6421993Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-06T20:49:23.6453611Z http.https://github.com/.extraheader 2024-08-06T20:49:23.6513276Z Entering 'third_party/pocketfft' 2024-08-06T20:49:23.6546829Z http.https://github.com/.extraheader 2024-08-06T20:49:23.6580647Z Entering 'third_party/protobuf' 2024-08-06T20:49:23.6613956Z http.https://github.com/.extraheader 2024-08-06T20:49:23.6650950Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-06T20:49:23.6683941Z http.https://github.com/.extraheader 2024-08-06T20:49:23.6718276Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-06T20:49:23.6751689Z http.https://github.com/.extraheader 2024-08-06T20:49:23.6787281Z Entering 'third_party/psimd' 2024-08-06T20:49:23.6820567Z http.https://github.com/.extraheader 2024-08-06T20:49:23.6854134Z Entering 'third_party/pthreadpool' 2024-08-06T20:49:23.6887140Z http.https://github.com/.extraheader 2024-08-06T20:49:23.6921182Z Entering 'third_party/pybind11' 2024-08-06T20:49:23.6954764Z http.https://github.com/.extraheader 2024-08-06T20:49:23.6989310Z Entering 'third_party/python-peachpy' 2024-08-06T20:49:23.7022478Z http.https://github.com/.extraheader 2024-08-06T20:49:23.7055199Z Entering 'third_party/sleef' 2024-08-06T20:49:23.7087966Z http.https://github.com/.extraheader 2024-08-06T20:49:23.7122540Z Entering 'third_party/tensorpipe' 2024-08-06T20:49:23.7155800Z http.https://github.com/.extraheader 2024-08-06T20:49:23.7190055Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-06T20:49:23.7222896Z http.https://github.com/.extraheader 2024-08-06T20:49:23.7255863Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-06T20:49:23.7287550Z http.https://github.com/.extraheader 2024-08-06T20:49:23.7321377Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-06T20:49:23.7353901Z http.https://github.com/.extraheader 2024-08-06T20:49:23.7388148Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-06T20:49:23.7420552Z http.https://github.com/.extraheader 2024-08-06T20:49:23.7452757Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-06T20:49:23.7485681Z http.https://github.com/.extraheader 2024-08-06T20:49:23.7622966Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-06T20:49:23.7651272Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-06T20:49:23.7659112Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-06T20:49:23.7659622Z ##[endgroup] 2024-08-06T20:49:29.5964396Z Cleaning up orphan processes