2024-08-20T21:47:47.8543073Z Current runner version: '2.319.1' 2024-08-20T21:47:47.8551533Z Runner name: 'i-08a502d1ce8446e65' 2024-08-20T21:47:47.8552534Z Runner group name: 'Default' 2024-08-20T21:47:47.8553685Z Machine name: 'ip-10-0-36-3' 2024-08-20T21:47:47.8575935Z Testing runner upgrade compatibility 2024-08-20T21:47:48.0323396Z ##[group]GITHUB_TOKEN Permissions 2024-08-20T21:47:48.0325947Z Actions: read 2024-08-20T21:47:48.0326678Z Attestations: read 2024-08-20T21:47:48.0327343Z Checks: read 2024-08-20T21:47:48.0327993Z Contents: read 2024-08-20T21:47:48.0328622Z Deployments: read 2024-08-20T21:47:48.0329275Z Discussions: read 2024-08-20T21:47:48.0329875Z Issues: read 2024-08-20T21:47:48.0330470Z Metadata: read 2024-08-20T21:47:48.0331070Z Packages: read 2024-08-20T21:47:48.0331666Z Pages: read 2024-08-20T21:47:48.0332262Z PullRequests: read 2024-08-20T21:47:48.0332932Z RepositoryProjects: read 2024-08-20T21:47:48.0333636Z SecurityEvents: read 2024-08-20T21:47:48.0334291Z Statuses: read 2024-08-20T21:47:48.0335198Z ##[endgroup] 2024-08-20T21:47:48.0339729Z Secret source: Actions 2024-08-20T21:47:48.0340625Z Prepare workflow directory 2024-08-20T21:47:48.5210102Z Prepare all required actions 2024-08-20T21:47:48.5439695Z Getting action download info 2024-08-20T21:47:48.7246708Z Download action repository 'pytorch/test-infra@main' (SHA:0c3a2634aaa2f638c8f640e743f03d696ce1191f) 2024-08-20T21:47:49.1759326Z Download action repository 'pytorch/pytorch@main' (SHA:2e1830c7c8dfc1141c89232e0356671ac07aa979) 2024-08-20T21:47:53.3168353Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-20T21:47:53.4722704Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-20T21:47:53.7949570Z Getting action download info 2024-08-20T21:47:53.8798220Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-20T21:47:54.0981541Z Getting action download info 2024-08-20T21:47:54.1994934Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-20T21:47:54.3428940Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/133712/merge (f2fb9405c2fa9f9502a76363091cce6fd8179736) 2024-08-20T21:47:54.3432464Z ##[group] Inputs 2024-08-20T21:47:54.3433064Z build-environment: linux-focal-cuda11.8-py3.10-gcc9 2024-08-20T21:47:54.3435435Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}]} 2024-08-20T21:47:54.3438505Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:47:54.3439916Z sync-tag: 2024-08-20T21:47:54.3441095Z timeout-minutes: 360 2024-08-20T21:47:54.3441552Z use-gha: 2024-08-20T21:47:54.3441924Z dashboard-tag: 2024-08-20T21:47:54.3442321Z s3-bucket: gha-artifacts 2024-08-20T21:47:54.3442789Z aws-role-to-assume: 2024-08-20T21:47:54.3443236Z ##[endgroup] 2024-08-20T21:47:54.3444566Z Complete job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, amz2023.linux.8xlarge.nvidia.gpu) 2024-08-20T21:47:54.4425768Z A job started hook has been configured by the self-hosted runner administrator 2024-08-20T21:47:54.4624060Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-20T21:47:54.4635690Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:47:54.4636367Z ##[endgroup] 2024-08-20T21:47:56.3890731Z Runner Type: amz2023.linux.8xlarge.nvidia.gpu 2024-08-20T21:47:56.3891424Z Instance Type: g3.8xlarge 2024-08-20T21:47:56.3892265Z AMI Name: al2023-ami-2023.5.20240701.0-kernel-6.1-x86_64 2024-08-20T21:47:56.3893537Z AMI ID: ami-06c68f701d8090592 2024-08-20T21:48:03.5771486Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-20T21:48:03.5772258Z with: 2024-08-20T21:48:03.5773106Z github-secret: *** 2024-08-20T21:48:03.5774327Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T21:48:03.5775602Z activate-with-label: false 2024-08-20T21:48:03.5776048Z label: with-ssh 2024-08-20T21:48:03.5776426Z remove-existing-keys: true 2024-08-20T21:48:03.5776874Z fail-silently: true 2024-08-20T21:48:03.5777254Z env: 2024-08-20T21:48:03.5777571Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:48:03.5777996Z ##[endgroup] 2024-08-20T21:48:03.6944965Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-20T21:48:04.0015035Z Grabbing public ssh keys from https://github.com/pytorchmergebot.keys 2024-08-20T21:48:04.0644066Z No SSH keys found for user pytorchmergebot 2024-08-20T21:48:04.0644935Z Grabbing public ssh keys from https://github.com/XuehaiPan.keys 2024-08-20T21:48:04.1294323Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-20T21:48:04.1317685Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-20T21:48:04.1359786Z Login using: ssh ec2-user@ec2-107-23-6-98.compute-1.amazonaws.com 2024-08-20T21:48:04.1360920Z All testing is done inside the container, to start an interactive session run: 2024-08-20T21:48:04.1361953Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T21:48:04.1500865Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-20T21:48:04.1501568Z with: 2024-08-20T21:48:04.1501913Z submodules: recursive 2024-08-20T21:48:04.1502326Z fetch-depth: 0 2024-08-20T21:48:04.1502665Z env: 2024-08-20T21:48:04.1502995Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:48:04.1503421Z ##[endgroup] 2024-08-20T21:48:04.1759679Z ##[group]Run retry () { 2024-08-20T21:48:04.1760158Z retry () { 2024-08-20T21:48:04.1760784Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-20T21:48:04.1761470Z } 2024-08-20T21:48:04.1761827Z echo "${GITHUB_WORKSPACE}" 2024-08-20T21:48:04.1762341Z if [ -z "${NO_SUDO}" ]; then 2024-08-20T21:48:04.1762935Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T21:48:04.1763479Z else 2024-08-20T21:48:04.1763901Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T21:48:04.1764426Z fi 2024-08-20T21:48:04.1764843Z mkdir "${GITHUB_WORKSPACE}" 2024-08-20T21:48:04.1772258Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:48:04.1772891Z env: 2024-08-20T21:48:04.1773224Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:48:04.1773625Z NO_SUDO: 2024-08-20T21:48:04.1773958Z ##[endgroup] 2024-08-20T21:48:04.1805438Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T21:48:07.6413109Z ##[group]Run malfet/checkout@silent-checkout 2024-08-20T21:48:07.6413650Z with: 2024-08-20T21:48:07.6414031Z ref: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:48:07.6414576Z fetch-depth: 0 2024-08-20T21:48:07.6414954Z submodules: recursive 2024-08-20T21:48:07.6415348Z quiet-checkout: true 2024-08-20T21:48:07.6415769Z repository: pytorch/pytorch 2024-08-20T21:48:07.6416377Z token: *** 2024-08-20T21:48:07.6416715Z ssh-strict: true 2024-08-20T21:48:07.6417112Z persist-credentials: true 2024-08-20T21:48:07.6417546Z clean: true 2024-08-20T21:48:07.6417930Z sparse-checkout-cone-mode: true 2024-08-20T21:48:07.6418403Z lfs: false 2024-08-20T21:48:07.6418762Z set-safe-directory: true 2024-08-20T21:48:07.6419154Z env: 2024-08-20T21:48:07.6419476Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:48:07.6419894Z ##[endgroup] 2024-08-20T21:48:07.7718109Z Syncing repository: pytorch/pytorch 2024-08-20T21:48:07.7720477Z ##[group]Getting Git version info 2024-08-20T21:48:07.7721430Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T21:48:07.7722848Z [command]/usr/bin/git version 2024-08-20T21:48:07.7723295Z git version 2.40.1 2024-08-20T21:48:07.7727665Z ##[endgroup] 2024-08-20T21:48:07.7749356Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/6210b777-20d1-4d22-a635-74c1128d5c02' before making global git config changes 2024-08-20T21:48:07.7750928Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T21:48:07.7756468Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T21:48:07.7804519Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T21:48:07.7810258Z ##[group]Initializing the repository 2024-08-20T21:48:07.7814192Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T21:48:07.7849660Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-20T21:48:07.7850734Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-20T21:48:07.7851705Z hint: of your new repositories, which will suppress this warning, call: 2024-08-20T21:48:07.7852398Z hint: 2024-08-20T21:48:07.7852946Z hint: git config --global init.defaultBranch 2024-08-20T21:48:07.7853503Z hint: 2024-08-20T21:48:07.7854161Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-20T21:48:07.7855236Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-20T21:48:07.7856172Z hint: 2024-08-20T21:48:07.7856610Z hint: git branch -m 2024-08-20T21:48:07.7857548Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-20T21:48:07.7864881Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-20T21:48:07.7902825Z ##[endgroup] 2024-08-20T21:48:07.7903503Z ##[group]Disabling automatic garbage collection 2024-08-20T21:48:07.7907088Z [command]/usr/bin/git config --local gc.auto 0 2024-08-20T21:48:07.7942473Z ##[endgroup] 2024-08-20T21:48:07.7943126Z ##[group]Setting up auth 2024-08-20T21:48:07.7951614Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T21:48:07.7989385Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T21:48:07.8309008Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T21:48:07.8343289Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T21:48:07.8667620Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T21:48:07.8721711Z ##[endgroup] 2024-08-20T21:48:07.8722380Z ##[group]Fetching the repository 2024-08-20T21:48:07.8730448Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-20T21:48:11.2824933Z remote: Enumerating objects: 1008519 2024-08-20T21:48:11.2969473Z remote: Enumerating objects: 1008524 2024-08-20T21:48:11.2970320Z remote: Enumerating objects: 1010161, done. 2024-08-20T21:48:11.2974896Z remote: Counting objects: 0% (1/1642) 2024-08-20T21:48:11.2975670Z remote: Counting objects: 1% (17/1642) 2024-08-20T21:48:11.2976383Z remote: Counting objects: 2% (33/1642) 2024-08-20T21:48:11.2977409Z remote: Counting objects: 3% (50/1642) 2024-08-20T21:48:11.2978035Z remote: Counting objects: 4% (66/1642) 2024-08-20T21:48:11.2978649Z remote: Counting objects: 5% (83/1642) 2024-08-20T21:48:11.2979262Z remote: Counting objects: 6% (99/1642) 2024-08-20T21:48:11.2980678Z remote: Counting objects: 7% (115/1642) 2024-08-20T21:48:11.2981294Z remote: Counting objects: 8% (132/1642) 2024-08-20T21:48:11.2981914Z remote: Counting objects: 9% (148/1642) 2024-08-20T21:48:11.2982732Z remote: Counting objects: 10% (165/1642) 2024-08-20T21:48:11.2983772Z remote: Counting objects: 11% (181/1642) 2024-08-20T21:48:11.2984790Z remote: Counting objects: 12% (198/1642) 2024-08-20T21:48:11.2985491Z remote: Counting objects: 13% (214/1642) 2024-08-20T21:48:11.2986115Z remote: Counting objects: 14% (230/1642) 2024-08-20T21:48:11.2986711Z remote: Counting objects: 15% (247/1642) 2024-08-20T21:48:11.2987332Z remote: Counting objects: 16% (263/1642) 2024-08-20T21:48:11.2988331Z remote: Counting objects: 17% (280/1642) 2024-08-20T21:48:11.2988931Z remote: Counting objects: 18% (296/1642) 2024-08-20T21:48:11.2989545Z remote: Counting objects: 19% (312/1642) 2024-08-20T21:48:11.2990487Z remote: Counting objects: 20% (329/1642) 2024-08-20T21:48:11.2991441Z remote: Counting objects: 21% (345/1642) 2024-08-20T21:48:11.2992047Z remote: Counting objects: 22% (362/1642) 2024-08-20T21:48:11.2993247Z remote: Counting objects: 23% (378/1642) 2024-08-20T21:48:11.2994188Z remote: Counting objects: 24% (395/1642) 2024-08-20T21:48:11.2994787Z remote: Counting objects: 25% (411/1642) 2024-08-20T21:48:11.2995621Z remote: Counting objects: 26% (427/1642) 2024-08-20T21:48:11.2996830Z remote: Counting objects: 27% (444/1642) 2024-08-20T21:48:11.2997461Z remote: Counting objects: 28% (460/1642) 2024-08-20T21:48:11.2998327Z remote: Counting objects: 29% (477/1642) 2024-08-20T21:48:11.2999674Z remote: Counting objects: 30% (493/1642) 2024-08-20T21:48:11.3000302Z remote: Counting objects: 31% (510/1642) 2024-08-20T21:48:11.3001370Z remote: Counting objects: 32% (526/1642) 2024-08-20T21:48:11.3002219Z remote: Counting objects: 33% (542/1642) 2024-08-20T21:48:11.3002840Z remote: Counting objects: 34% (559/1642) 2024-08-20T21:48:11.3003644Z remote: Counting objects: 35% (575/1642) 2024-08-20T21:48:11.3004609Z remote: Counting objects: 36% (592/1642) 2024-08-20T21:48:11.3005224Z remote: Counting objects: 37% (608/1642) 2024-08-20T21:48:11.3006052Z remote: Counting objects: 38% (624/1642) 2024-08-20T21:48:11.3006674Z remote: Counting objects: 39% (641/1642) 2024-08-20T21:48:11.3007295Z remote: Counting objects: 40% (657/1642) 2024-08-20T21:48:11.3007889Z remote: Counting objects: 41% (674/1642) 2024-08-20T21:48:11.3008550Z remote: Counting objects: 42% (690/1642) 2024-08-20T21:48:11.3009159Z remote: Counting objects: 43% (707/1642) 2024-08-20T21:48:11.3009751Z remote: Counting objects: 44% (723/1642) 2024-08-20T21:48:11.3010364Z remote: Counting objects: 45% (739/1642) 2024-08-20T21:48:11.3010988Z remote: Counting objects: 46% (756/1642) 2024-08-20T21:48:11.3011582Z remote: Counting objects: 47% (772/1642) 2024-08-20T21:48:11.3012310Z remote: Counting objects: 48% (789/1642) 2024-08-20T21:48:11.3012923Z remote: Counting objects: 49% (805/1642) 2024-08-20T21:48:11.3013534Z remote: Counting objects: 50% (821/1642) 2024-08-20T21:48:11.3014122Z remote: Counting objects: 51% (838/1642) 2024-08-20T21:48:11.3014735Z remote: Counting objects: 52% (854/1642) 2024-08-20T21:48:11.3015354Z remote: Counting objects: 53% (871/1642) 2024-08-20T21:48:11.3015952Z remote: Counting objects: 54% (887/1642) 2024-08-20T21:48:11.3016562Z remote: Counting objects: 55% (904/1642) 2024-08-20T21:48:11.3017172Z remote: Counting objects: 56% (920/1642) 2024-08-20T21:48:11.3017765Z remote: Counting objects: 57% (936/1642) 2024-08-20T21:48:11.3018591Z remote: Counting objects: 58% (953/1642) 2024-08-20T21:48:11.3019207Z remote: Counting objects: 59% (969/1642) 2024-08-20T21:48:11.3019801Z remote: Counting objects: 60% (986/1642) 2024-08-20T21:48:11.3020422Z remote: Counting objects: 61% (1002/1642) 2024-08-20T21:48:11.3021066Z remote: Counting objects: 62% (1019/1642) 2024-08-20T21:48:11.3021708Z remote: Counting objects: 63% (1035/1642) 2024-08-20T21:48:11.3022339Z remote: Counting objects: 64% (1051/1642) 2024-08-20T21:48:11.3022982Z remote: Counting objects: 65% (1068/1642) 2024-08-20T21:48:11.3023623Z remote: Counting objects: 66% (1084/1642) 2024-08-20T21:48:11.3024241Z remote: Counting objects: 67% (1101/1642) 2024-08-20T21:48:11.3024875Z remote: Counting objects: 68% (1117/1642) 2024-08-20T21:48:11.3025509Z remote: Counting objects: 69% (1133/1642) 2024-08-20T21:48:11.3026126Z remote: Counting objects: 70% (1150/1642) 2024-08-20T21:48:11.3026766Z remote: Counting objects: 71% (1166/1642) 2024-08-20T21:48:11.3027398Z remote: Counting objects: 72% (1183/1642) 2024-08-20T21:48:11.3028012Z remote: Counting objects: 73% (1199/1642) 2024-08-20T21:48:11.3028642Z remote: Counting objects: 74% (1216/1642) 2024-08-20T21:48:11.3029275Z remote: Counting objects: 75% (1232/1642) 2024-08-20T21:48:11.3029893Z remote: Counting objects: 76% (1248/1642) 2024-08-20T21:48:11.3030523Z remote: Counting objects: 77% (1265/1642) 2024-08-20T21:48:11.3031251Z remote: Counting objects: 78% (1281/1642) 2024-08-20T21:48:11.3031887Z remote: Counting objects: 79% (1298/1642) 2024-08-20T21:48:11.3032525Z remote: Counting objects: 80% (1314/1642) 2024-08-20T21:48:11.3033158Z remote: Counting objects: 81% (1331/1642) 2024-08-20T21:48:11.3033795Z remote: Counting objects: 82% (1347/1642) 2024-08-20T21:48:11.3034413Z remote: Counting objects: 83% (1363/1642) 2024-08-20T21:48:11.3035052Z remote: Counting objects: 84% (1380/1642) 2024-08-20T21:48:11.3035682Z remote: Counting objects: 85% (1396/1642) 2024-08-20T21:48:11.3036305Z remote: Counting objects: 86% (1413/1642) 2024-08-20T21:48:11.3036938Z remote: Counting objects: 87% (1429/1642) 2024-08-20T21:48:11.3037571Z remote: Counting objects: 88% (1445/1642) 2024-08-20T21:48:11.3038188Z remote: Counting objects: 89% (1462/1642) 2024-08-20T21:48:11.3038820Z remote: Counting objects: 90% (1478/1642) 2024-08-20T21:48:11.3039454Z remote: Counting objects: 91% (1495/1642) 2024-08-20T21:48:11.3040072Z remote: Counting objects: 92% (1511/1642) 2024-08-20T21:48:11.3040701Z remote: Counting objects: 93% (1528/1642) 2024-08-20T21:48:11.3041344Z remote: Counting objects: 94% (1544/1642) 2024-08-20T21:48:11.3041977Z remote: Counting objects: 95% (1560/1642) 2024-08-20T21:48:11.3042594Z remote: Counting objects: 96% (1577/1642) 2024-08-20T21:48:11.3043230Z remote: Counting objects: 97% (1593/1642) 2024-08-20T21:48:11.3043859Z remote: Counting objects: 98% (1610/1642) 2024-08-20T21:48:11.3044472Z remote: Counting objects: 99% (1626/1642) 2024-08-20T21:48:11.3045099Z remote: Counting objects: 100% (1642/1642) 2024-08-20T21:48:11.3045775Z remote: Counting objects: 100% (1642/1642), done. 2024-08-20T21:48:11.3175684Z remote: Compressing objects: 0% (1/857) 2024-08-20T21:48:11.3615157Z remote: Compressing objects: 1% (9/857) 2024-08-20T21:48:11.3945651Z remote: Compressing objects: 2% (18/857) 2024-08-20T21:48:11.4866587Z remote: Compressing objects: 3% (26/857) 2024-08-20T21:48:11.5438671Z remote: Compressing objects: 4% (35/857) 2024-08-20T21:48:11.6004593Z remote: Compressing objects: 5% (43/857) 2024-08-20T21:48:11.6276855Z remote: Compressing objects: 6% (52/857) 2024-08-20T21:48:11.6682379Z remote: Compressing objects: 7% (60/857) 2024-08-20T21:48:11.6974970Z remote: Compressing objects: 8% (69/857) 2024-08-20T21:48:11.7248831Z remote: Compressing objects: 9% (78/857) 2024-08-20T21:48:11.7526729Z remote: Compressing objects: 10% (86/857) 2024-08-20T21:48:11.7713625Z remote: Compressing objects: 11% (95/857) 2024-08-20T21:48:11.7977267Z remote: Compressing objects: 12% (103/857) 2024-08-20T21:48:11.8115821Z remote: Compressing objects: 13% (112/857) 2024-08-20T21:48:11.8238946Z remote: Compressing objects: 14% (120/857) 2024-08-20T21:48:11.8328546Z remote: Compressing objects: 15% (129/857) 2024-08-20T21:48:11.8408967Z remote: Compressing objects: 16% (138/857) 2024-08-20T21:48:11.8486673Z remote: Compressing objects: 17% (146/857) 2024-08-20T21:48:11.8549585Z remote: Compressing objects: 18% (155/857) 2024-08-20T21:48:11.8592734Z remote: Compressing objects: 19% (163/857) 2024-08-20T21:48:11.8619237Z remote: Compressing objects: 20% (172/857) 2024-08-20T21:48:11.8651750Z remote: Compressing objects: 21% (180/857) 2024-08-20T21:48:11.8656531Z remote: Compressing objects: 22% (189/857) 2024-08-20T21:48:11.8662005Z remote: Compressing objects: 23% (198/857) 2024-08-20T21:48:11.8674017Z remote: Compressing objects: 24% (206/857) 2024-08-20T21:48:11.8676434Z remote: Compressing objects: 25% (215/857) 2024-08-20T21:48:11.8683867Z remote: Compressing objects: 26% (223/857) 2024-08-20T21:48:11.8691563Z remote: Compressing objects: 27% (232/857) 2024-08-20T21:48:11.8710280Z remote: Compressing objects: 28% (240/857) 2024-08-20T21:48:11.8725367Z remote: Compressing objects: 29% (249/857) 2024-08-20T21:48:11.8741874Z remote: Compressing objects: 30% (258/857) 2024-08-20T21:48:11.8748741Z remote: Compressing objects: 31% (266/857) 2024-08-20T21:48:11.8759607Z remote: Compressing objects: 32% (275/857) 2024-08-20T21:48:11.8771474Z remote: Compressing objects: 33% (283/857) 2024-08-20T21:48:11.8774346Z remote: Compressing objects: 34% (292/857) 2024-08-20T21:48:11.8784171Z remote: Compressing objects: 35% (300/857) 2024-08-20T21:48:11.8790748Z remote: Compressing objects: 36% (309/857) 2024-08-20T21:48:11.8801460Z remote: Compressing objects: 37% (318/857) 2024-08-20T21:48:11.8807679Z remote: Compressing objects: 38% (326/857) 2024-08-20T21:48:11.8813052Z remote: Compressing objects: 39% (335/857) 2024-08-20T21:48:11.8818673Z remote: Compressing objects: 40% (343/857) 2024-08-20T21:48:11.8826706Z remote: Compressing objects: 41% (352/857) 2024-08-20T21:48:11.8832424Z remote: Compressing objects: 42% (360/857) 2024-08-20T21:48:11.8837811Z remote: Compressing objects: 43% (369/857) 2024-08-20T21:48:11.8844883Z remote: Compressing objects: 44% (378/857) 2024-08-20T21:48:11.8850997Z remote: Compressing objects: 45% (386/857) 2024-08-20T21:48:11.8854413Z remote: Compressing objects: 46% (395/857) 2024-08-20T21:48:11.8860018Z remote: Compressing objects: 47% (403/857) 2024-08-20T21:48:11.8862190Z remote: Compressing objects: 48% (412/857) 2024-08-20T21:48:11.8866011Z remote: Compressing objects: 49% (420/857) 2024-08-20T21:48:11.8868289Z remote: Compressing objects: 50% (429/857) 2024-08-20T21:48:11.8870764Z remote: Compressing objects: 51% (438/857) 2024-08-20T21:48:11.8874329Z remote: Compressing objects: 52% (446/857) 2024-08-20T21:48:11.8876496Z remote: Compressing objects: 53% (455/857) 2024-08-20T21:48:11.8878727Z remote: Compressing objects: 54% (463/857) 2024-08-20T21:48:11.8880364Z remote: Compressing objects: 55% (472/857) 2024-08-20T21:48:11.8882020Z remote: Compressing objects: 56% (480/857) 2024-08-20T21:48:11.8883236Z remote: Compressing objects: 57% (489/857) 2024-08-20T21:48:11.8884704Z remote: Compressing objects: 58% (498/857) 2024-08-20T21:48:11.8885661Z remote: Compressing objects: 59% (506/857) 2024-08-20T21:48:11.8886774Z remote: Compressing objects: 60% (515/857) 2024-08-20T21:48:11.8887439Z remote: Compressing objects: 61% (523/857) 2024-08-20T21:48:11.8888117Z remote: Compressing objects: 62% (532/857) 2024-08-20T21:48:11.8888789Z remote: Compressing objects: 63% (540/857) 2024-08-20T21:48:11.8890764Z remote: Compressing objects: 64% (549/857) 2024-08-20T21:48:11.8898282Z remote: Compressing objects: 65% (558/857) 2024-08-20T21:48:11.8903664Z remote: Compressing objects: 66% (566/857) 2024-08-20T21:48:11.8908718Z remote: Compressing objects: 67% (575/857) 2024-08-20T21:48:11.8913114Z remote: Compressing objects: 68% (583/857) 2024-08-20T21:48:11.8917600Z remote: Compressing objects: 69% (592/857) 2024-08-20T21:48:11.8922569Z remote: Compressing objects: 70% (600/857) 2024-08-20T21:48:11.8926837Z remote: Compressing objects: 71% (609/857) 2024-08-20T21:48:11.8930569Z remote: Compressing objects: 72% (618/857) 2024-08-20T21:48:11.8936003Z remote: Compressing objects: 73% (626/857) 2024-08-20T21:48:11.8938251Z remote: Compressing objects: 74% (635/857) 2024-08-20T21:48:11.8941540Z remote: Compressing objects: 75% (643/857) 2024-08-20T21:48:11.8944537Z remote: Compressing objects: 76% (652/857) 2024-08-20T21:48:11.8948528Z remote: Compressing objects: 77% (660/857) 2024-08-20T21:48:11.8950225Z remote: Compressing objects: 78% (669/857) 2024-08-20T21:48:11.8951844Z remote: Compressing objects: 79% (678/857) 2024-08-20T21:48:11.8954368Z remote: Compressing objects: 80% (686/857) 2024-08-20T21:48:11.8956995Z remote: Compressing objects: 81% (695/857) 2024-08-20T21:48:11.8958084Z remote: Compressing objects: 82% (703/857) 2024-08-20T21:48:11.8960296Z remote: Compressing objects: 83% (712/857) 2024-08-20T21:48:11.8962552Z remote: Compressing objects: 84% (720/857) 2024-08-20T21:48:11.8964113Z remote: Compressing objects: 85% (729/857) 2024-08-20T21:48:11.8965629Z remote: Compressing objects: 86% (738/857) 2024-08-20T21:48:11.8967905Z remote: Compressing objects: 87% (746/857) 2024-08-20T21:48:11.8970191Z remote: Compressing objects: 88% (755/857) 2024-08-20T21:48:11.8972328Z remote: Compressing objects: 89% (763/857) 2024-08-20T21:48:11.8973509Z remote: Compressing objects: 90% (772/857) 2024-08-20T21:48:11.8974711Z remote: Compressing objects: 91% (780/857) 2024-08-20T21:48:11.8976005Z remote: Compressing objects: 92% (789/857) 2024-08-20T21:48:11.8976947Z remote: Compressing objects: 93% (798/857) 2024-08-20T21:48:11.8977790Z remote: Compressing objects: 94% (806/857) 2024-08-20T21:48:11.8978864Z remote: Compressing objects: 95% (815/857) 2024-08-20T21:48:11.8980464Z remote: Compressing objects: 96% (823/857) 2024-08-20T21:48:11.8981687Z remote: Compressing objects: 97% (832/857) 2024-08-20T21:48:11.8983021Z remote: Compressing objects: 98% (840/857) 2024-08-20T21:48:11.8984245Z remote: Compressing objects: 99% (849/857) 2024-08-20T21:48:11.8985585Z remote: Compressing objects: 100% (857/857) 2024-08-20T21:48:11.8986332Z remote: Compressing objects: 100% (857/857), done. 2024-08-20T21:48:38.7333815Z remote: Total 1010161 (delta 1055), reused 1262 (delta 782), pack-reused 1008519 (from 1) 2024-08-20T21:49:00.4547518Z [command]/usr/bin/git rev-parse --verify --quiet 40ec5f6ddd9787aca0449b24128343ff4c4a88b3^{object} 2024-08-20T21:49:00.4579565Z 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:49:00.4584825Z ##[endgroup] 2024-08-20T21:49:00.4587417Z ##[group]Determining the checkout info 2024-08-20T21:49:00.4588240Z ##[endgroup] 2024-08-20T21:49:00.4588868Z ##[group]Checking out the ref 2024-08-20T21:49:00.4591907Z [command]/usr/bin/git checkout --quiet --force 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:49:02.4006992Z ##[endgroup] 2024-08-20T21:49:02.4007770Z ##[group]Setting up auth for fetching submodules 2024-08-20T21:49:02.4012983Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T21:49:02.4073157Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-20T21:49:02.4109653Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-20T21:49:02.4145794Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-20T21:49:02.4218887Z ##[endgroup] 2024-08-20T21:49:02.4219668Z ##[group]Fetching submodules 2024-08-20T21:49:02.4220367Z [command]/usr/bin/git submodule sync --recursive 2024-08-20T21:49:02.4540177Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-20T21:49:02.4882544Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-20T21:49:02.4885575Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-20T21:49:02.4889184Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-20T21:49:02.4893071Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-20T21:49:02.4900102Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-20T21:49:02.4902275Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-20T21:49:02.4905517Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-20T21:49:02.4909759Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-20T21:49:02.4914350Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-20T21:49:02.4918979Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-20T21:49:02.4923637Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-20T21:49:02.4928229Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-20T21:49:02.4933090Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-20T21:49:02.4938090Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-20T21:49:02.4943151Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-20T21:49:02.4948643Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:49:02.4956235Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-20T21:49:02.4961768Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-20T21:49:02.4967105Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-20T21:49:02.4972963Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-20T21:49:02.4978601Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-20T21:49:02.4984583Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-20T21:49:02.4990342Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-20T21:49:02.4997601Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-20T21:49:02.5003646Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-20T21:49:02.5010321Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-20T21:49:02.5016549Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-20T21:49:02.5023068Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-20T21:49:02.5029689Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-20T21:49:02.5036512Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-20T21:49:02.5043203Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-20T21:49:02.5050419Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-20T21:49:02.5057272Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-20T21:49:02.5066459Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-20T21:49:02.5099046Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-20T21:49:02.8732865Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-20T21:49:03.0669428Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-20T21:49:03.2513774Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-20T21:49:03.5645589Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-20T21:49:05.6702900Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-20T21:49:17.4920107Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-20T21:49:17.9452215Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-20T21:49:18.3828598Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-20T21:49:19.0795724Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-20T21:49:20.5374960Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-20T21:49:22.9686038Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-20T21:49:29.8518264Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-20T21:49:31.4312511Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-20T21:49:33.3637275Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-20T21:49:34.7209924Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-20T21:49:35.2383369Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-20T21:49:35.6296705Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-20T21:49:36.8716612Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-20T21:49:37.2551450Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-20T21:49:37.5070119Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-20T21:49:39.3310981Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-20T21:49:40.1337144Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-20T21:49:40.7996107Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-20T21:49:47.5525268Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-20T21:49:49.6748611Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-20T21:49:53.9003944Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-20T21:49:54.1289150Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-20T21:50:04.4013388Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-20T21:50:04.6382284Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-20T21:50:04.8885988Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-20T21:50:05.9102506Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-20T21:50:06.2290844Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-20T21:50:06.9100542Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-20T21:50:07.3527286Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-20T21:50:07.3672779Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-20T21:50:07.3782724Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-20T21:50:07.4103630Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-20T21:50:07.4583539Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-20T21:50:08.7342792Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-20T21:50:08.7644733Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-20T21:50:08.8204681Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-20T21:50:08.9439714Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-20T21:50:08.9876900Z Submodule path 'third_party/cudnn_frontend': checked out '23511ba176243f27b3b275da1fb3814ea805a171' 2024-08-20T21:50:09.6799525Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-20T21:50:10.0079851Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-20T21:50:10.1142266Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-20T21:50:10.1163912Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:50:10.1168140Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:50:10.1171969Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:50:10.1176195Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:50:10.1180921Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:50:10.1212382Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-20T21:50:11.3124665Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-20T21:50:11.9754575Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-20T21:50:14.4035705Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-20T21:50:15.5546514Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-20T21:50:15.9351660Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-20T21:50:16.0586279Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-20T21:50:16.6027555Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-20T21:50:16.6822359Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-20T21:50:16.6970039Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-20T21:50:16.8618654Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-20T21:50:16.9120115Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-20T21:50:16.9612019Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-20T21:50:16.9964198Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-20T21:50:17.0539937Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T21:50:17.0694727Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-20T21:50:17.0715690Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-20T21:50:17.0744447Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-20T21:50:31.6481840Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-20T21:50:31.6703257Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-20T21:50:31.7810002Z Submodule path 'third_party/kineto': checked out 'd9753139d181b9ff42872465aac0e5d3018be415' 2024-08-20T21:50:31.7830397Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:50:31.7833820Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:50:31.7837727Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:50:31.7867996Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-20T21:50:32.3882669Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-20T21:50:33.7098016Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-20T21:50:34.9337490Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-20T21:50:34.9357554Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:50:34.9361480Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:50:34.9365476Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:50:34.9369652Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:50:34.9373820Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:50:34.9378289Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:50:34.9383312Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:50:34.9388179Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:50:34.9420516Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-20T21:50:35.9615636Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-20T21:50:36.4107284Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-20T21:50:37.7198562Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-20T21:50:38.0671782Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-20T21:50:38.6286035Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-20T21:50:39.7889820Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-20T21:50:46.5946037Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-20T21:50:47.0943919Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-20T21:50:47.1181776Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-20T21:50:47.1644553Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-20T21:50:47.1805701Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-20T21:50:47.1824780Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:50:47.1855943Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-20T21:50:47.5085320Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-20T21:50:47.5316016Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-20T21:50:47.5832285Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-20T21:50:47.7226433Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-20T21:50:47.7428103Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-20T21:50:47.7921930Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-20T21:50:47.8647487Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-20T21:50:47.9113015Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-20T21:50:47.9417145Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-20T21:50:48.0788778Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-20T21:50:48.5794458Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-20T21:50:48.5835465Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-20T21:50:48.5838633Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-20T21:50:48.5870063Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-20T21:50:49.0785342Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-20T21:50:50.2398766Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-20T21:50:50.2832724Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-20T21:50:50.3764355Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-20T21:50:50.3786336Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:50:50.3790134Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:50:50.3794194Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:50:50.3798452Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:50:50.3802722Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:50:50.3806810Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:50:50.3811044Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:50:50.3815230Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:50:50.3845599Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-20T21:50:50.8606918Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-20T21:50:51.9947287Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-20T21:50:52.3282452Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-20T21:50:59.1405449Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-20T21:50:59.4454719Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-20T21:50:59.6724758Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-20T21:50:59.9941778Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-20T21:51:07.3617929Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-20T21:51:07.4130362Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-20T21:51:07.4319690Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-20T21:51:07.5695935Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-20T21:51:07.5857103Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-20T21:51:07.6042177Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-20T21:51:07.6246553Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-20T21:51:07.6265663Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:51:07.6269441Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:51:07.6299395Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-20T21:51:09.7032437Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-20T21:51:11.1789370Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-20T21:51:11.2390257Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T21:51:12.0000277Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-20T21:51:12.0138977Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-20T21:51:12.3597228Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-20T21:51:12.3623027Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:51:12.3626442Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-20T21:51:12.3659835Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-20T21:51:12.9488832Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-20T21:51:14.1293286Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-20T21:51:14.2192329Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-20T21:51:14.2299862Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-20T21:51:14.2450293Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-20T21:51:14.2927436Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-20T21:51:14.3279784Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-20T21:51:14.3805877Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-20T21:51:14.4162109Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-20T21:51:14.4181785Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:51:14.4185288Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:51:14.4188683Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:51:14.4193006Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:51:14.4222812Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-20T21:51:15.5628489Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-20T21:51:15.8216186Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-20T21:51:17.1586447Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-20T21:51:18.2564282Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-20T21:51:18.2758807Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-20T21:51:18.3650700Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-20T21:51:18.4016610Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-20T21:51:18.4035686Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:51:18.4065805Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-20T21:51:18.6165981Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-20T21:51:18.6206352Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-20T21:51:18.6547754Z Entering 'android/libs/fbjni' 2024-08-20T21:51:18.6596580Z Entering 'third_party/FP16' 2024-08-20T21:51:18.6643603Z Entering 'third_party/FXdiv' 2024-08-20T21:51:18.6690840Z Entering 'third_party/NNPACK' 2024-08-20T21:51:18.6739719Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T21:51:18.6786955Z Entering 'third_party/XNNPACK' 2024-08-20T21:51:18.6854336Z Entering 'third_party/benchmark' 2024-08-20T21:51:18.6901877Z Entering 'third_party/cpp-httplib' 2024-08-20T21:51:18.6949679Z Entering 'third_party/cpuinfo' 2024-08-20T21:51:18.6999064Z Entering 'third_party/cudnn_frontend' 2024-08-20T21:51:18.7047334Z Entering 'third_party/cutlass' 2024-08-20T21:51:18.7105434Z Entering 'third_party/eigen' 2024-08-20T21:51:18.7155129Z Entering 'third_party/fbgemm' 2024-08-20T21:51:18.7204742Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:51:18.7251930Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:51:18.7300504Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:51:18.7354472Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:51:18.7404306Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:51:18.7452118Z Entering 'third_party/flatbuffers' 2024-08-20T21:51:18.7504905Z Entering 'third_party/fmt' 2024-08-20T21:51:18.7552126Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:51:18.7602036Z Entering 'third_party/gloo' 2024-08-20T21:51:18.7649757Z Entering 'third_party/googletest' 2024-08-20T21:51:18.7700030Z Entering 'third_party/ideep' 2024-08-20T21:51:18.7745996Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T21:51:18.7800324Z Entering 'third_party/ittapi' 2024-08-20T21:51:18.7848049Z Entering 'third_party/kineto' 2024-08-20T21:51:18.7896687Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:51:18.7944368Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:51:18.7993449Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:51:18.8042807Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:51:18.8090056Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:51:18.8138995Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:51:18.8188081Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:51:18.8236137Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:51:18.8284548Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:51:18.8335014Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:51:18.8383410Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:51:18.8430666Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:51:18.8479867Z Entering 'third_party/mimalloc' 2024-08-20T21:51:18.8529279Z Entering 'third_party/nccl/nccl' 2024-08-20T21:51:18.8576927Z Entering 'third_party/nlohmann' 2024-08-20T21:51:18.8625758Z Entering 'third_party/onnx' 2024-08-20T21:51:18.8691330Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T21:51:18.8739107Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T21:51:18.8789229Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T21:51:18.8839087Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:51:18.8886295Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:51:18.8934844Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:51:18.8981408Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:51:18.9031213Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:51:18.9077029Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:51:18.9124783Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:51:18.9171303Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:51:18.9222443Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:51:18.9270991Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:51:18.9341272Z Entering 'third_party/pocketfft' 2024-08-20T21:51:18.9388154Z Entering 'third_party/protobuf' 2024-08-20T21:51:18.9438904Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:51:18.9486086Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T21:51:18.9537485Z Entering 'third_party/psimd' 2024-08-20T21:51:18.9584430Z Entering 'third_party/pthreadpool' 2024-08-20T21:51:18.9633184Z Entering 'third_party/pybind11' 2024-08-20T21:51:18.9680738Z Entering 'third_party/python-peachpy' 2024-08-20T21:51:18.9728992Z Entering 'third_party/sleef' 2024-08-20T21:51:18.9776386Z Entering 'third_party/tensorpipe' 2024-08-20T21:51:18.9824867Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:51:18.9871659Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:51:18.9918507Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:51:18.9965665Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:51:19.0012546Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:51:19.0075767Z ##[endgroup] 2024-08-20T21:51:19.0078388Z ##[group]Persisting credentials for submodules 2024-08-20T21:51:19.0082482Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-20T21:51:19.0423717Z Entering 'android/libs/fbjni' 2024-08-20T21:51:19.0486189Z Entering 'third_party/FP16' 2024-08-20T21:51:19.0549051Z Entering 'third_party/FXdiv' 2024-08-20T21:51:19.0612448Z Entering 'third_party/NNPACK' 2024-08-20T21:51:19.0675165Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T21:51:19.0739416Z Entering 'third_party/XNNPACK' 2024-08-20T21:51:19.0820958Z Entering 'third_party/benchmark' 2024-08-20T21:51:19.0883589Z Entering 'third_party/cpp-httplib' 2024-08-20T21:51:19.0947097Z Entering 'third_party/cpuinfo' 2024-08-20T21:51:19.1010199Z Entering 'third_party/cudnn_frontend' 2024-08-20T21:51:19.1072943Z Entering 'third_party/cutlass' 2024-08-20T21:51:19.1145654Z Entering 'third_party/eigen' 2024-08-20T21:51:19.1211293Z Entering 'third_party/fbgemm' 2024-08-20T21:51:19.1273718Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:51:19.1336374Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:51:19.1399238Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:51:19.1468290Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:51:19.1531935Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:51:19.1594919Z Entering 'third_party/flatbuffers' 2024-08-20T21:51:19.1659795Z Entering 'third_party/fmt' 2024-08-20T21:51:19.1722665Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:51:19.1784114Z Entering 'third_party/gloo' 2024-08-20T21:51:19.1847150Z Entering 'third_party/googletest' 2024-08-20T21:51:19.1909148Z Entering 'third_party/ideep' 2024-08-20T21:51:19.1969729Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T21:51:19.2041343Z Entering 'third_party/ittapi' 2024-08-20T21:51:19.2104114Z Entering 'third_party/kineto' 2024-08-20T21:51:19.2165511Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:51:19.2228285Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:51:19.2292049Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:51:19.2356317Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:51:19.2419103Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:51:19.2479797Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:51:19.2545873Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:51:19.2609990Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:51:19.2674590Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:51:19.2739288Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:51:19.2804894Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:51:19.2865760Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:51:19.2929499Z Entering 'third_party/mimalloc' 2024-08-20T21:51:19.2991964Z Entering 'third_party/nccl/nccl' 2024-08-20T21:51:19.3054543Z Entering 'third_party/nlohmann' 2024-08-20T21:51:19.3119230Z Entering 'third_party/onnx' 2024-08-20T21:51:19.3199319Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T21:51:19.3260939Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T21:51:19.3326980Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T21:51:19.3389644Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:51:19.3451796Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:51:19.3513485Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:51:19.3574386Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:51:19.3637841Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:51:19.3699159Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:51:19.3760116Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:51:19.3820930Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:51:19.3884755Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:51:19.3949187Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:51:19.4034352Z Entering 'third_party/pocketfft' 2024-08-20T21:51:19.4096677Z Entering 'third_party/protobuf' 2024-08-20T21:51:19.4162318Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:51:19.4224097Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T21:51:19.4288053Z Entering 'third_party/psimd' 2024-08-20T21:51:19.4350551Z Entering 'third_party/pthreadpool' 2024-08-20T21:51:19.4412754Z Entering 'third_party/pybind11' 2024-08-20T21:51:19.4474696Z Entering 'third_party/python-peachpy' 2024-08-20T21:51:19.4537003Z Entering 'third_party/sleef' 2024-08-20T21:51:19.4600453Z Entering 'third_party/tensorpipe' 2024-08-20T21:51:19.4661541Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:51:19.4723906Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:51:19.4784788Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:51:19.4847153Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:51:19.4907442Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:51:19.4987365Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-20T21:51:19.5325290Z Entering 'android/libs/fbjni' 2024-08-20T21:51:19.5383980Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-20T21:51:19.5403008Z Entering 'third_party/FP16' 2024-08-20T21:51:19.5461533Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-20T21:51:19.5479640Z Entering 'third_party/FXdiv' 2024-08-20T21:51:19.5540982Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-20T21:51:19.5558921Z Entering 'third_party/NNPACK' 2024-08-20T21:51:19.5617846Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-20T21:51:19.5636104Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T21:51:19.5696061Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-20T21:51:19.5714205Z Entering 'third_party/XNNPACK' 2024-08-20T21:51:19.5772166Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-20T21:51:19.5809324Z Entering 'third_party/benchmark' 2024-08-20T21:51:19.5868351Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-20T21:51:19.5886033Z Entering 'third_party/cpp-httplib' 2024-08-20T21:51:19.5944566Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-20T21:51:19.5962699Z Entering 'third_party/cpuinfo' 2024-08-20T21:51:19.6021899Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T21:51:19.6040654Z Entering 'third_party/cudnn_frontend' 2024-08-20T21:51:19.6100328Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-20T21:51:19.6118325Z Entering 'third_party/cutlass' 2024-08-20T21:51:19.6175958Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-20T21:51:19.6202309Z Entering 'third_party/eigen' 2024-08-20T21:51:19.6259690Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-20T21:51:19.6281290Z Entering 'third_party/fbgemm' 2024-08-20T21:51:19.6341178Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-20T21:51:19.6359262Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:51:19.6418419Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-20T21:51:19.6436196Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:51:19.6494496Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T21:51:19.6513438Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:51:19.6571365Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-20T21:51:19.6597766Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:51:19.6655347Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:51:19.6673424Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:51:19.6733883Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-20T21:51:19.6752934Z Entering 'third_party/flatbuffers' 2024-08-20T21:51:19.6812383Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-20T21:51:19.6833901Z Entering 'third_party/fmt' 2024-08-20T21:51:19.6891992Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-20T21:51:19.6911113Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:51:19.6970825Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-20T21:51:19.6989471Z Entering 'third_party/gloo' 2024-08-20T21:51:19.7047550Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-20T21:51:19.7066277Z Entering 'third_party/googletest' 2024-08-20T21:51:19.7125671Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:51:19.7143995Z Entering 'third_party/ideep' 2024-08-20T21:51:19.7203026Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-20T21:51:19.7219631Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T21:51:19.7278158Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-20T21:51:19.7305562Z Entering 'third_party/ittapi' 2024-08-20T21:51:19.7362940Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-20T21:51:19.7381340Z Entering 'third_party/kineto' 2024-08-20T21:51:19.7441283Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-20T21:51:19.7458809Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:51:19.7520011Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-20T21:51:19.7537228Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:51:19.7597169Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-20T21:51:19.7616394Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:51:19.7675952Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-20T21:51:19.7694427Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:51:19.7754063Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-20T21:51:19.7772748Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:51:19.7833946Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-20T21:51:19.7850781Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:51:19.7911983Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-20T21:51:19.7931407Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:51:19.7989980Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-20T21:51:19.8008862Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:51:19.8067017Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:51:19.8085291Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:51:19.8145982Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-20T21:51:19.8165221Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:51:19.8225157Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-20T21:51:19.8244622Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:51:19.8304519Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-20T21:51:19.8322523Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:51:19.8380167Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-20T21:51:19.8401876Z Entering 'third_party/mimalloc' 2024-08-20T21:51:19.8459488Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-20T21:51:19.8478023Z Entering 'third_party/nccl/nccl' 2024-08-20T21:51:19.8536640Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-20T21:51:19.8555304Z Entering 'third_party/nlohmann' 2024-08-20T21:51:19.8614304Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-20T21:51:19.8634032Z Entering 'third_party/onnx' 2024-08-20T21:51:19.8691903Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-20T21:51:19.8726755Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T21:51:19.8784065Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-20T21:51:19.8802702Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T21:51:19.8861121Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-20T21:51:19.8881747Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T21:51:19.8941627Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-20T21:51:19.8960980Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:51:19.9020485Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-20T21:51:19.9038287Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:51:19.9096381Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:51:19.9114623Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:51:19.9172450Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-20T21:51:19.9190150Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:51:19.9249200Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-20T21:51:19.9268325Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:51:19.9326830Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-20T21:51:19.9344585Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:51:19.9403378Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-20T21:51:19.9420786Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:51:19.9479083Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-20T21:51:19.9497204Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:51:19.9555280Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-20T21:51:19.9575119Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:51:19.9637442Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-20T21:51:19.9656996Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:51:19.9715895Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-20T21:51:19.9754461Z Entering 'third_party/pocketfft' 2024-08-20T21:51:19.9812914Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-20T21:51:19.9830877Z Entering 'third_party/protobuf' 2024-08-20T21:51:19.9888581Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-20T21:51:19.9911223Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:51:19.9968812Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-20T21:51:19.9986638Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T21:51:20.0046013Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:51:20.0066335Z Entering 'third_party/psimd' 2024-08-20T21:51:20.0126029Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-20T21:51:20.0143757Z Entering 'third_party/pthreadpool' 2024-08-20T21:51:20.0202044Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-20T21:51:20.0219891Z Entering 'third_party/pybind11' 2024-08-20T21:51:20.0277862Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-20T21:51:20.0297542Z Entering 'third_party/python-peachpy' 2024-08-20T21:51:20.0354857Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-20T21:51:20.0372619Z Entering 'third_party/sleef' 2024-08-20T21:51:20.0431088Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-20T21:51:20.0449301Z Entering 'third_party/tensorpipe' 2024-08-20T21:51:20.0508705Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-20T21:51:20.0526608Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:51:20.0585604Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-20T21:51:20.0603970Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:51:20.0661514Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-20T21:51:20.0679339Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:51:20.0737998Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-20T21:51:20.0755874Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:51:20.0815618Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-20T21:51:20.0832097Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:51:20.0890226Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-20T21:51:20.1987011Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-20T21:51:20.2333623Z Entering 'android/libs/fbjni' 2024-08-20T21:51:20.2381019Z Entering 'third_party/FP16' 2024-08-20T21:51:20.2429990Z Entering 'third_party/FXdiv' 2024-08-20T21:51:20.2477189Z Entering 'third_party/NNPACK' 2024-08-20T21:51:20.2527733Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T21:51:20.2575525Z Entering 'third_party/XNNPACK' 2024-08-20T21:51:20.2642355Z Entering 'third_party/benchmark' 2024-08-20T21:51:20.2690132Z Entering 'third_party/cpp-httplib' 2024-08-20T21:51:20.2738654Z Entering 'third_party/cpuinfo' 2024-08-20T21:51:20.2786550Z Entering 'third_party/cudnn_frontend' 2024-08-20T21:51:20.2835245Z Entering 'third_party/cutlass' 2024-08-20T21:51:20.2891129Z Entering 'third_party/eigen' 2024-08-20T21:51:20.2941910Z Entering 'third_party/fbgemm' 2024-08-20T21:51:20.2990386Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:51:20.3039526Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:51:20.3088338Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:51:20.3143465Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:51:20.3190872Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:51:20.3240010Z Entering 'third_party/flatbuffers' 2024-08-20T21:51:20.3290891Z Entering 'third_party/fmt' 2024-08-20T21:51:20.3338667Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:51:20.3387532Z Entering 'third_party/gloo' 2024-08-20T21:51:20.3435951Z Entering 'third_party/googletest' 2024-08-20T21:51:20.3484636Z Entering 'third_party/ideep' 2024-08-20T21:51:20.3533066Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T21:51:20.3587674Z Entering 'third_party/ittapi' 2024-08-20T21:51:20.3635626Z Entering 'third_party/kineto' 2024-08-20T21:51:20.3684106Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:51:20.3734776Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:51:20.3783106Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:51:20.3831519Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:51:20.3879106Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:51:20.3925616Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:51:20.3975377Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:51:20.4025557Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:51:20.4072800Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:51:20.4124561Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:51:20.4174851Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:51:20.4222894Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:51:20.4271109Z Entering 'third_party/mimalloc' 2024-08-20T21:51:20.4320635Z Entering 'third_party/nccl/nccl' 2024-08-20T21:51:20.4368425Z Entering 'third_party/nlohmann' 2024-08-20T21:51:20.4418017Z Entering 'third_party/onnx' 2024-08-20T21:51:20.4485361Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T21:51:20.4533728Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T21:51:20.4584965Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T21:51:20.4636812Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:51:20.4683782Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:51:20.4731781Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:51:20.4781194Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:51:20.4829821Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:51:20.4875822Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:51:20.4922761Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:51:20.4969275Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:51:20.5019067Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:51:20.5068543Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:51:20.5138091Z Entering 'third_party/pocketfft' 2024-08-20T21:51:20.5185347Z Entering 'third_party/protobuf' 2024-08-20T21:51:20.5237105Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:51:20.5284296Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T21:51:20.5335676Z Entering 'third_party/psimd' 2024-08-20T21:51:20.5383032Z Entering 'third_party/pthreadpool' 2024-08-20T21:51:20.5431220Z Entering 'third_party/pybind11' 2024-08-20T21:51:20.5478761Z Entering 'third_party/python-peachpy' 2024-08-20T21:51:20.5528508Z Entering 'third_party/sleef' 2024-08-20T21:51:20.5575923Z Entering 'third_party/tensorpipe' 2024-08-20T21:51:20.5626041Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:51:20.5673207Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:51:20.5721010Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:51:20.5767909Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:51:20.5815221Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:51:20.5881535Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-20T21:51:20.6224102Z Entering 'android/libs/fbjni' 2024-08-20T21:51:20.6271168Z Entering 'third_party/FP16' 2024-08-20T21:51:20.6320701Z Entering 'third_party/FXdiv' 2024-08-20T21:51:20.6368094Z Entering 'third_party/NNPACK' 2024-08-20T21:51:20.6417099Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T21:51:20.6465111Z Entering 'third_party/XNNPACK' 2024-08-20T21:51:20.6529831Z Entering 'third_party/benchmark' 2024-08-20T21:51:20.6577077Z Entering 'third_party/cpp-httplib' 2024-08-20T21:51:20.6625955Z Entering 'third_party/cpuinfo' 2024-08-20T21:51:20.6674336Z Entering 'third_party/cudnn_frontend' 2024-08-20T21:51:20.6722899Z Entering 'third_party/cutlass' 2024-08-20T21:51:20.6778783Z Entering 'third_party/eigen' 2024-08-20T21:51:20.6828896Z Entering 'third_party/fbgemm' 2024-08-20T21:51:20.6876783Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T21:51:20.6924849Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T21:51:20.6973049Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T21:51:20.7029557Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T21:51:20.7076887Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T21:51:20.7126884Z Entering 'third_party/flatbuffers' 2024-08-20T21:51:20.7178081Z Entering 'third_party/fmt' 2024-08-20T21:51:20.7226537Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T21:51:20.7273848Z Entering 'third_party/gloo' 2024-08-20T21:51:20.7323005Z Entering 'third_party/googletest' 2024-08-20T21:51:20.7371657Z Entering 'third_party/ideep' 2024-08-20T21:51:20.7418713Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T21:51:20.7473720Z Entering 'third_party/ittapi' 2024-08-20T21:51:20.7522551Z Entering 'third_party/kineto' 2024-08-20T21:51:20.7570852Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T21:51:20.7619019Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T21:51:20.7668456Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T21:51:20.7716773Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T21:51:20.7764556Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T21:51:20.7812735Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T21:51:20.7862349Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T21:51:20.7910737Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T21:51:20.7958294Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T21:51:20.8007812Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T21:51:20.8056698Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T21:51:20.8104275Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T21:51:20.8153331Z Entering 'third_party/mimalloc' 2024-08-20T21:51:20.8201549Z Entering 'third_party/nccl/nccl' 2024-08-20T21:51:20.8249266Z Entering 'third_party/nlohmann' 2024-08-20T21:51:20.8299311Z Entering 'third_party/onnx' 2024-08-20T21:51:20.8365423Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T21:51:20.8416318Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T21:51:20.8466966Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T21:51:20.8517462Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T21:51:20.8564823Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T21:51:20.8613198Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T21:51:20.8660169Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T21:51:20.8709019Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T21:51:20.8755670Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T21:51:20.8803582Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T21:51:20.8850726Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T21:51:20.8901870Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T21:51:20.8949890Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T21:51:20.9020972Z Entering 'third_party/pocketfft' 2024-08-20T21:51:20.9068069Z Entering 'third_party/protobuf' 2024-08-20T21:51:20.9119750Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T21:51:20.9167222Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T21:51:20.9218756Z Entering 'third_party/psimd' 2024-08-20T21:51:20.9266290Z Entering 'third_party/pthreadpool' 2024-08-20T21:51:20.9314360Z Entering 'third_party/pybind11' 2024-08-20T21:51:20.9362414Z Entering 'third_party/python-peachpy' 2024-08-20T21:51:20.9411362Z Entering 'third_party/sleef' 2024-08-20T21:51:20.9458929Z Entering 'third_party/tensorpipe' 2024-08-20T21:51:20.9508142Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T21:51:20.9555167Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T21:51:20.9603303Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T21:51:20.9650163Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T21:51:20.9697118Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T21:51:20.9759880Z ##[endgroup] 2024-08-20T21:51:20.9804922Z [command]/usr/bin/git log -1 --format='%H' 2024-08-20T21:51:20.9835132Z '40ec5f6ddd9787aca0449b24128343ff4c4a88b3' 2024-08-20T21:51:21.0068974Z Prepare all required actions 2024-08-20T21:51:21.0069586Z Getting action download info 2024-08-20T21:51:21.1489343Z ##[group]Run ./.github/actions/setup-linux 2024-08-20T21:51:21.1489878Z env: 2024-08-20T21:51:21.1490198Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:21.1490603Z ##[endgroup] 2024-08-20T21:51:21.1598728Z ##[group]Run set -euo pipefail 2024-08-20T21:51:21.1599259Z set -euo pipefail 2024-08-20T21:51:21.1599737Z function get_ec2_metadata() { 2024-08-20T21:51:21.1600379Z  # Pulled from instance metadata endpoint for EC2 2024-08-20T21:51:21.1601492Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-20T21:51:21.1602657Z  category=$1 2024-08-20T21:51:21.1603287Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-20T21:51:21.1604068Z  runner_name_str=i-08a502d1ce8446e65 2024-08-20T21:51:21.1604685Z  if [[ -f /.inarc ]]; then 2024-08-20T21:51:21.1605271Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-20T21:51:21.1605967Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-20T21:51:21.1606828Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-20T21:51:21.1607566Z  else 2024-08-20T21:51:21.1608222Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-20T21:51:21.1609069Z  fi 2024-08-20T21:51:21.1609417Z } 2024-08-20T21:51:21.1609839Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-20T21:51:21.1610564Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-20T21:51:21.1611423Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-20T21:51:21.1612157Z echo "system info $(uname -a)" 2024-08-20T21:51:21.1621373Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:21.1622060Z env: 2024-08-20T21:51:21.1622389Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:21.1622816Z ##[endgroup] 2024-08-20T21:51:21.1730501Z ami-id: ami-06c68f701d8090592 2024-08-20T21:51:21.1793089Z instance-id: i-08a502d1ce8446e65 2024-08-20T21:51:21.1851499Z instance-type: g3.8xlarge 2024-08-20T21:51:21.1864851Z system info Linux ip-10-0-36-3.ec2.internal 6.1.94-99.176.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-20T21:51:21.1894984Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T21:51:21.1896142Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T21:51:21.1903217Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:21.1903832Z env: 2024-08-20T21:51:21.1904153Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:21.1904573Z ##[endgroup] 2024-08-20T21:51:21.1996657Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-20T21:51:21.1997415Z if systemctl is-active --quiet docker; then 2024-08-20T21:51:21.1998084Z  echo "Docker daemon is running..."; 2024-08-20T21:51:21.1998635Z else 2024-08-20T21:51:21.1999220Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-20T21:51:21.1999935Z fi 2024-08-20T21:51:21.2006823Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:21.2007422Z env: 2024-08-20T21:51:21.2007754Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:21.2008179Z ##[endgroup] 2024-08-20T21:51:21.2090637Z Docker daemon is running... 2024-08-20T21:51:21.2154786Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T21:51:21.2155511Z with: 2024-08-20T21:51:21.2155839Z shell: bash 2024-08-20T21:51:21.2156257Z timeout_minutes: 5 2024-08-20T21:51:21.2156651Z max_attempts: 3 2024-08-20T21:51:21.2157034Z retry_wait_seconds: 30 2024-08-20T21:51:21.2161046Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-08-20T21:51:21.2164992Z polling_interval_seconds: 1 2024-08-20T21:51:21.2165618Z warning_on_retry: true 2024-08-20T21:51:21.2166036Z continue_on_error: false 2024-08-20T21:51:21.2166446Z env: 2024-08-20T21:51:21.2166770Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:21.2167184Z AWS_RETRY_MODE: standard 2024-08-20T21:51:21.2167619Z AWS_MAX_ATTEMPTS: 5 2024-08-20T21:51:21.2168037Z AWS_DEFAULT_REGION: us-east-1 2024-08-20T21:51:21.2168486Z ##[endgroup] 2024-08-20T21:51:22.6630601Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T21:51:22.6631666Z Configure a credential helper to remove this warning. See 2024-08-20T21:51:22.6632731Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T21:51:22.6633409Z 2024-08-20T21:51:22.6633554Z Login Succeeded 2024-08-20T21:51:23.2909241Z Command completed after 1 attempt(s). 2024-08-20T21:51:23.2976196Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T21:51:23.2977130Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T21:51:23.2977994Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T21:51:23.2986817Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:23.2987439Z env: 2024-08-20T21:51:23.2987793Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:23.2988247Z ##[endgroup] 2024-08-20T21:51:23.3097352Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T21:51:23.3098336Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T21:51:23.3099082Z # shellcheck disable=SC2046 2024-08-20T21:51:23.3099660Z docker stop $(docker ps -q) || true 2024-08-20T21:51:23.3100241Z # Prune all of the docker images 2024-08-20T21:51:23.3100805Z docker system prune -af 2024-08-20T21:51:23.3108166Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:23.3108775Z env: 2024-08-20T21:51:23.3109121Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:23.3109559Z ##[endgroup] 2024-08-20T21:51:23.3481858Z "docker stop" requires at least 1 argument. 2024-08-20T21:51:23.3482700Z See 'docker stop --help'. 2024-08-20T21:51:23.3483023Z 2024-08-20T21:51:23.3483299Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-20T21:51:23.3483788Z 2024-08-20T21:51:23.3483972Z Stop one or more running containers 2024-08-20T21:51:23.3687556Z Total reclaimed space: 0B 2024-08-20T21:51:23.3733521Z ##[group]Run set +e 2024-08-20T21:51:23.3733995Z set +e 2024-08-20T21:51:23.3734344Z set -x 2024-08-20T21:51:23.3734699Z  2024-08-20T21:51:23.3735086Z PT_DOMAIN=download.pytorch.org 2024-08-20T21:51:23.3736062Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-20T21:51:23.3737609Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-20T21:51:23.3738546Z # one is returned at random 2024-08-20T21:51:23.3739216Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-20T21:51:23.3739848Z  2024-08-20T21:51:23.3740225Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T21:51:23.3740974Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-20T21:51:23.3742140Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-20T21:51:23.3742869Z  2024-08-20T21:51:23.3743257Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T21:51:23.3743923Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-20T21:51:23.3744548Z  exit 1 2024-08-20T21:51:23.3744924Z  fi 2024-08-20T21:51:23.3745248Z fi 2024-08-20T21:51:23.3745576Z  2024-08-20T21:51:23.3746006Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-20T21:51:23.3746622Z  # Clean up any old records first 2024-08-20T21:51:23.3747247Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-20T21:51:23.3747985Z fi 2024-08-20T21:51:23.3748301Z  2024-08-20T21:51:23.3748831Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-20T21:51:23.3749509Z cat /etc/hosts 2024-08-20T21:51:23.3756970Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:23.3757599Z env: 2024-08-20T21:51:23.3757938Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:23.3758353Z ##[endgroup] 2024-08-20T21:51:23.3788391Z + PT_DOMAIN=download.pytorch.org 2024-08-20T21:51:23.3795146Z ++ dig -4 +short download.pytorch.org 2024-08-20T21:51:23.3796012Z ++ tail -n1 2024-08-20T21:51:23.4057089Z + RESOLVED_IP=18.160.10.22 2024-08-20T21:51:23.4057946Z + '[' -z 18.160.10.22 ']' 2024-08-20T21:51:23.4058491Z + grep -r download.pytorch.org /etc/hosts 2024-08-20T21:51:23.4072351Z 108.138.64.116 download.pytorch.org 2024-08-20T21:51:23.4073876Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-08-20T21:51:23.5497741Z + echo '18.160.10.22 download.pytorch.org' 2024-08-20T21:51:23.5498926Z + sudo tee -a /etc/hosts 2024-08-20T21:51:23.6033648Z 18.160.10.22 download.pytorch.org 2024-08-20T21:51:23.6052651Z + cat /etc/hosts 2024-08-20T21:51:23.6062255Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-20T21:51:23.6077557Z ::1 localhost6 localhost6.localdomain6 2024-08-20T21:51:23.6078274Z 18.160.10.22 download.pytorch.org 2024-08-20T21:51:23.6281173Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-20T21:51:23.6281977Z with: 2024-08-20T21:51:23.6283170Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:23.6284552Z docker-build-dir: .ci/docker 2024-08-20T21:51:23.6285014Z working-directory: . 2024-08-20T21:51:23.6285573Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:23.6286228Z force-push: false 2024-08-20T21:51:23.6286608Z env: 2024-08-20T21:51:23.6286914Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:23.6287335Z ##[endgroup] 2024-08-20T21:51:23.6314427Z ##[group]Run set -ex 2024-08-20T21:51:23.6314866Z set -ex 2024-08-20T21:51:23.6315220Z  2024-08-20T21:51:23.6315896Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-20T21:51:23.6317188Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-20T21:51:23.6318188Z # job could then download the pre-built image as usual 2024-08-20T21:51:23.6319112Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-20T21:51:23.6319954Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:23.6320741Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:23.6321415Z  2024-08-20T21:51:23.6322042Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-20T21:51:23.6322908Z  exit 0 2024-08-20T21:51:23.6323256Z else 2024-08-20T21:51:23.6323699Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:23.6324254Z fi 2024-08-20T21:51:23.6324566Z  2024-08-20T21:51:23.6325142Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-20T21:51:23.6326219Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-20T21:51:23.6327176Z  # use it as it is, but first let's extract the tag 2024-08-20T21:51:23.6328057Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-20T21:51:23.6328961Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:23.6329830Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:23.6330504Z else 2024-08-20T21:51:23.6331033Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-20T21:51:23.6332083Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:23.6333180Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:23.6334122Z fi 2024-08-20T21:51:23.6342106Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:23.6342712Z env: 2024-08-20T21:51:23.6343044Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:23.6343478Z REPO_NAME: pytorch 2024-08-20T21:51:23.6344720Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:23.6346080Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T21:51:23.6346690Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:23.6347329Z ##[endgroup] 2024-08-20T21:51:23.6376549Z + [[ ! -d .ci/docker ]] 2024-08-20T21:51:23.6377092Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-20T21:51:23.6377540Z + echo skip=false 2024-08-20T21:51:23.6379679Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-20T21:51:23.6384875Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:23.6386302Z ++ awk -F '[:,]' '{print $2}' 2024-08-20T21:51:23.6408407Z + DOCKER_TAG=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:23.6409289Z + echo docker-tag=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:23.6411241Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:23.6448391Z ##[group]Run set +e 2024-08-20T21:51:23.6448851Z set +e 2024-08-20T21:51:23.6449217Z set -x 2024-08-20T21:51:23.6449550Z  2024-08-20T21:51:23.6449877Z login() { 2024-08-20T21:51:23.6450692Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T21:51:23.6451590Z } 2024-08-20T21:51:23.6451901Z  2024-08-20T21:51:23.6452238Z retry () { 2024-08-20T21:51:23.6452701Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T21:51:23.6453342Z } 2024-08-20T21:51:23.6453975Z  2024-08-20T21:51:23.6454372Z retry login "${DOCKER_REGISTRY}" 2024-08-20T21:51:23.6454886Z  2024-08-20T21:51:23.6455461Z # Check if image already exists, if it does then skip building it 2024-08-20T21:51:23.6456337Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-20T21:51:23.6456938Z  exit 0 2024-08-20T21:51:23.6457295Z fi 2024-08-20T21:51:23.6457631Z  2024-08-20T21:51:23.6458229Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-20T21:51:23.6459260Z # be empty. The default action would be to continue rebuild the image 2024-08-20T21:51:23.6460181Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-20T21:51:23.6461000Z  # if we're on the base branch then use the parent commit 2024-08-20T21:51:23.6461740Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-20T21:51:23.6462265Z else 2024-08-20T21:51:23.6462821Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-20T21:51:23.6463681Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-20T21:51:23.6464311Z fi 2024-08-20T21:51:23.6464914Z  2024-08-20T21:51:23.6465559Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-20T21:51:23.6466196Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:23.6466982Z  2024-08-20T21:51:23.6467779Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-20T21:51:23.6468740Z  exit 0 2024-08-20T21:51:23.6469101Z fi 2024-08-20T21:51:23.6469413Z  2024-08-20T21:51:23.6469944Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-20T21:51:23.6471172Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-20T21:51:23.6472171Z  exit 1 2024-08-20T21:51:23.6472538Z fi 2024-08-20T21:51:23.6472867Z  2024-08-20T21:51:23.6473450Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-20T21:51:23.6474645Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-20T21:51:23.6475736Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-20T21:51:23.6477465Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-20T21:51:23.6478840Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-20T21:51:23.6479652Z fi 2024-08-20T21:51:23.6479985Z  2024-08-20T21:51:23.6480576Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T21:51:23.6487285Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:23.6487908Z env: 2024-08-20T21:51:23.6488218Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:23.6488662Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T21:51:23.6489231Z BASE_REVISION: 91f3d614142df02f619e44a68e3d9e0dfeba49ec 2024-08-20T21:51:23.6490652Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:23.6492079Z DOCKER_TAG: f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:23.6493320Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:23.6493967Z ##[endgroup] 2024-08-20T21:51:23.6522006Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:23.6522869Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:23.6524985Z + aws ecr get-login-password --region us-east-1 2024-08-20T21:51:23.6526246Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:24.3174348Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T21:51:24.3175371Z Configure a credential helper to remove this warning. See 2024-08-20T21:51:24.3176486Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T21:51:24.3177147Z 2024-08-20T21:51:24.3177318Z Login Succeeded 2024-08-20T21:51:24.3191571Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:24.5597989Z { 2024-08-20T21:51:24.5598647Z "schemaVersion": 2, 2024-08-20T21:51:24.5599329Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-20T21:51:24.5600069Z "config": { 2024-08-20T21:51:24.5600670Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-20T21:51:24.5601342Z "size": 47467, 2024-08-20T21:51:24.5604787Z "digest": "sha256:35f4945dfeab4592c8657de7107f96a9fd9a51999f582a0064937246789c0825" 2024-08-20T21:51:24.5605664Z }, 2024-08-20T21:51:24.5605983Z "layers": [ 2024-08-20T21:51:24.5606336Z { 2024-08-20T21:51:24.5606893Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5607586Z "size": 28580681, 2024-08-20T21:51:24.5608292Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-08-20T21:51:24.5609067Z }, 2024-08-20T21:51:24.5609355Z { 2024-08-20T21:51:24.5610226Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5610937Z "size": 7943353, 2024-08-20T21:51:24.5611616Z "digest": "sha256:09d415c238d76b32a7ea4a6e6add9542db9a5641f7f183af70aae185d0709e58" 2024-08-20T21:51:24.5612434Z + exit 0 2024-08-20T21:51:24.5612820Z }, 2024-08-20T21:51:24.5613133Z { 2024-08-20T21:51:24.5613666Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5614371Z "size": 56226389, 2024-08-20T21:51:24.5615067Z "digest": "sha256:9fe6e2e61518cba6844870c03b285737daec35e62baf25ae7744629ed3a7b470" 2024-08-20T21:51:24.5615823Z }, 2024-08-20T21:51:24.5616129Z { 2024-08-20T21:51:24.5616669Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5617351Z "size": 187, 2024-08-20T21:51:24.5618029Z "digest": "sha256:41f16248e682693ff20b3032c1d5e5541cc87c5af898ae2ff9b24d2940e59100" 2024-08-20T21:51:24.5618796Z }, 2024-08-20T21:51:24.5619085Z { 2024-08-20T21:51:24.5619642Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5620339Z "size": 6885, 2024-08-20T21:51:24.5621003Z "digest": "sha256:95d7b781703928cf3c4eece39d800cccb76728c375fedf51ecd83833fb25e458" 2024-08-20T21:51:24.5621783Z }, 2024-08-20T21:51:24.5622158Z { 2024-08-20T21:51:24.5622879Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5623620Z "size": 1383662475, 2024-08-20T21:51:24.5624313Z "digest": "sha256:8f6c9048534734f4c873935293b7296225846ceb31c1a158400a67ea170dde7f" 2024-08-20T21:51:24.5625043Z }, 2024-08-20T21:51:24.5625353Z { 2024-08-20T21:51:24.5625904Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5626580Z "size": 62282, 2024-08-20T21:51:24.5627259Z "digest": "sha256:ab17245097e491b9368790714f9d90ed447bf0973bd677cfe6f2456d62b72a13" 2024-08-20T21:51:24.5628023Z }, 2024-08-20T21:51:24.5628311Z { 2024-08-20T21:51:24.5628849Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5629556Z "size": 1684, 2024-08-20T21:51:24.5630236Z "digest": "sha256:dfecd7e9912b76ed460b8edd5a85f1943666e38a973ab5458177cf2c7c3110e3" 2024-08-20T21:51:24.5631030Z }, 2024-08-20T21:51:24.5631335Z { 2024-08-20T21:51:24.5631859Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5632690Z "size": 1522, 2024-08-20T21:51:24.5633383Z "digest": "sha256:464a8f74544589bf7b57f9a4cadcb6681e5ed00758f6c35025e691df4e88e890" 2024-08-20T21:51:24.5634149Z }, 2024-08-20T21:51:24.5634435Z { 2024-08-20T21:51:24.5634977Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5635681Z "size": 2445884909, 2024-08-20T21:51:24.5636356Z "digest": "sha256:d67c111fa58886d8904335b5122039816c8e51af5844cb6fe4e14154d9a7c016" 2024-08-20T21:51:24.5637113Z }, 2024-08-20T21:51:24.5637419Z { 2024-08-20T21:51:24.5637952Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5638660Z "size": 85882, 2024-08-20T21:51:24.5639349Z "digest": "sha256:0723abde19c09431db80b5d0f0e003541f6232c2da5c4f35c635c861875937ae" 2024-08-20T21:51:24.5640102Z }, 2024-08-20T21:51:24.5640410Z { 2024-08-20T21:51:24.5640952Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5641625Z "size": 1822, 2024-08-20T21:51:24.5642322Z "digest": "sha256:72d2b53f7a0105206afbdce791a85d403e2413ac60a2ea3fec0bb057b69925ad" 2024-08-20T21:51:24.5643140Z }, 2024-08-20T21:51:24.5643433Z { 2024-08-20T21:51:24.5643975Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5644669Z "size": 654861296, 2024-08-20T21:51:24.5645368Z "digest": "sha256:dbc6fccd9c70404c8728c8d319bbe9cd5ef59e56b27d0a3ff9385dc2007420e2" 2024-08-20T21:51:24.5646155Z }, 2024-08-20T21:51:24.5646462Z { 2024-08-20T21:51:24.5646977Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5647807Z "size": 544, 2024-08-20T21:51:24.5648489Z "digest": "sha256:ff6a6ddc73105bc6954a81b0d5ed486362bfd4399595a378a7c6594a332069a3" 2024-08-20T21:51:24.5649268Z }, 2024-08-20T21:51:24.5649552Z { 2024-08-20T21:51:24.5650094Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5650791Z "size": 1258, 2024-08-20T21:51:24.5651470Z "digest": "sha256:fac2b1b246ed3a00ba10fdfe9dc50559f8978915b204b903bc5fb47908fb5bd5" 2024-08-20T21:51:24.5652256Z }, 2024-08-20T21:51:24.5652561Z { 2024-08-20T21:51:24.5653077Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5653775Z "size": 484, 2024-08-20T21:51:24.5654458Z "digest": "sha256:bc574f79516b484ef9cff3a621d4e8f6dd0ced47a0c7de6959a2213b313cd871" 2024-08-20T21:51:24.5655217Z }, 2024-08-20T21:51:24.5655521Z { 2024-08-20T21:51:24.5656055Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5656734Z "size": 91712209, 2024-08-20T21:51:24.5657453Z "digest": "sha256:c23d42adcadecf7e1d93290ebf26fe3936841777be574d8e2a4c71719d43f05c" 2024-08-20T21:51:24.5658232Z }, 2024-08-20T21:51:24.5658515Z { 2024-08-20T21:51:24.5659054Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5659745Z "size": 3390, 2024-08-20T21:51:24.5660527Z "digest": "sha256:59b0a331db9f6d35dab3eef54ae05d703663a7cfe6ea9ec139fdc8c2221834da" 2024-08-20T21:51:24.5661336Z }, 2024-08-20T21:51:24.5661640Z { 2024-08-20T21:51:24.5662161Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5662853Z "size": 1909, 2024-08-20T21:51:24.5663536Z "digest": "sha256:478a9a2bdefc98de190384a54d89524fb0d691b7155ffba4df69a35a50e4afae" 2024-08-20T21:51:24.5664297Z }, 2024-08-20T21:51:24.5664600Z { 2024-08-20T21:51:24.5665138Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5665811Z "size": 699, 2024-08-20T21:51:24.5666488Z "digest": "sha256:28e54511f1b116a9d632caa0c4be2cd1b6ca7542f815560ec598b81d6d956d31" 2024-08-20T21:51:24.5667273Z }, 2024-08-20T21:51:24.5667556Z { 2024-08-20T21:51:24.5668100Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5668797Z "size": 2891829542, 2024-08-20T21:51:24.5669477Z "digest": "sha256:5247f18de8815b263c1380f613ce6f8b836db6766c3b137ef30b18fac324a4a3" 2024-08-20T21:51:24.5670252Z }, 2024-08-20T21:51:24.5670558Z { 2024-08-20T21:51:24.5671077Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5671768Z "size": 380, 2024-08-20T21:51:24.5672563Z "digest": "sha256:38760708e2d79494fe04b599b375145b8968797e5a063609cb007de5037ff0cf" 2024-08-20T21:51:24.5673296Z }, 2024-08-20T21:51:24.5673607Z { 2024-08-20T21:51:24.5674148Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5674819Z "size": 12876, 2024-08-20T21:51:24.5675521Z "digest": "sha256:863dbde6ed0c5edba3f86f2907198ab33eed3dd4eed51375189fd722a66b051a" 2024-08-20T21:51:24.5676323Z }, 2024-08-20T21:51:24.5676606Z { 2024-08-20T21:51:24.5677144Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5677841Z "size": 805, 2024-08-20T21:51:24.5678506Z "digest": "sha256:f283ab568a00b37f29b9d76f77c2f99a05b336ee3b1ffff8577b6bd7df2e95f3" 2024-08-20T21:51:24.5679318Z }, 2024-08-20T21:51:24.5679608Z { 2024-08-20T21:51:24.5680157Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5680834Z "size": 106, 2024-08-20T21:51:24.5681502Z "digest": "sha256:3e39f30b1d5c27e19c987aa8f8fbb36778a68cd534872e718053f71997457dab" 2024-08-20T21:51:24.5682275Z }, 2024-08-20T21:51:24.5682558Z { 2024-08-20T21:51:24.5683096Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5683790Z "size": 503, 2024-08-20T21:51:24.5684448Z "digest": "sha256:521c3d72534c80920fa11f85e7276b2130851936dba93947a4dc647eaf8855ba" 2024-08-20T21:51:24.5685306Z }, 2024-08-20T21:51:24.5685613Z { 2024-08-20T21:51:24.5686135Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5686835Z "size": 121477349, 2024-08-20T21:51:24.5687541Z "digest": "sha256:a975a4480a6d2abb7c44d3b8fe0e198c4d2666336f05f12936968ac5ac02fbe3" 2024-08-20T21:51:24.5688312Z }, 2024-08-20T21:51:24.5688594Z { 2024-08-20T21:51:24.5689144Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5689835Z "size": 109, 2024-08-20T21:51:24.5690489Z "digest": "sha256:1f2c278212ad728862a858a586ea5b666cea7384cb3b51900fc12afdd1fb0508" 2024-08-20T21:51:24.5691256Z }, 2024-08-20T21:51:24.5691561Z { 2024-08-20T21:51:24.5692077Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5693506Z "size": 489, 2024-08-20T21:51:24.5694194Z "digest": "sha256:b721dd867e80edbea20e5691ce6254378a75d20d38436ffc551dbb91453aefc5" 2024-08-20T21:51:24.5694949Z }, 2024-08-20T21:51:24.5695256Z { 2024-08-20T21:51:24.5695810Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5696488Z "size": 297, 2024-08-20T21:51:24.5697175Z "digest": "sha256:7a74b7ad649d3f0cc23bef1d20005cc436d2f7d478b2e81519d40ad7aa8ea251" 2024-08-20T21:51:24.5697955Z }, 2024-08-20T21:51:24.5698238Z { 2024-08-20T21:51:24.5698966Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5699699Z "size": 103, 2024-08-20T21:51:24.5700369Z "digest": "sha256:b53e1ad1deec07d1a5f093f57f5048627e112a68a5f512f9d9d229d0a91b9b2b" 2024-08-20T21:51:24.5701376Z }, 2024-08-20T21:51:24.5701679Z { 2024-08-20T21:51:24.5702199Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5702895Z "size": 1472, 2024-08-20T21:51:24.5703582Z "digest": "sha256:016dc3faa7efaa82e2cb84df6d34535024761fcfe67540e7e1f82686190841ee" 2024-08-20T21:51:24.5704343Z }, 2024-08-20T21:51:24.5704646Z { 2024-08-20T21:51:24.5705178Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5705872Z "size": 594729536, 2024-08-20T21:51:24.5706568Z "digest": "sha256:384dce3a4ba2bb506140986d6a45133d58345f055cbf72d018cff59716419012" 2024-08-20T21:51:24.5707331Z }, 2024-08-20T21:51:24.5707614Z { 2024-08-20T21:51:24.5708152Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5708849Z "size": 164, 2024-08-20T21:51:24.5709493Z "digest": "sha256:75461ad9f7bb4e53c4a6bd26539108b012c321304219c48a8852353fc2260613" 2024-08-20T21:51:24.5710247Z }, 2024-08-20T21:51:24.5710548Z { 2024-08-20T21:51:24.5711065Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5711760Z "size": 562, 2024-08-20T21:51:24.5712436Z "digest": "sha256:eb2062102d1137b2c485af726affb99a0f62f1a159a6c6ff0fbc936dac18596d" 2024-08-20T21:51:24.5713190Z }, 2024-08-20T21:51:24.5713493Z { 2024-08-20T21:51:24.5714028Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5714714Z "size": 35806203, 2024-08-20T21:51:24.5715407Z "digest": "sha256:3cc17f105239b758d0e1ff69aa738939b0a89561da055484a0a9e2602e5a543f" 2024-08-20T21:51:24.5716177Z }, 2024-08-20T21:51:24.5716458Z { 2024-08-20T21:51:24.5716994Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5717689Z "size": 104, 2024-08-20T21:51:24.5718350Z "digest": "sha256:585501be07997851ac265aba0a5b71e2c2e9afb4cc9c30da46cd9f2f5ffe063d" 2024-08-20T21:51:24.5719126Z }, 2024-08-20T21:51:24.5719430Z { 2024-08-20T21:51:24.5719947Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5720640Z "size": 424, 2024-08-20T21:51:24.5721341Z "digest": "sha256:6c49da68c9b4f370ed5ecfbb742ff5cad0ccc3d855e71fc10fdf603cabe9d87b" 2024-08-20T21:51:24.5722112Z }, 2024-08-20T21:51:24.5722468Z { 2024-08-20T21:51:24.5723005Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5723852Z "size": 20262169, 2024-08-20T21:51:24.5724531Z "digest": "sha256:497539689260668c0d109b8e22341ac13b1c482fd9c0b2e8371b8c88bd9fc766" 2024-08-20T21:51:24.5725279Z }, 2024-08-20T21:51:24.5725562Z { 2024-08-20T21:51:24.5726101Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5726795Z "size": 438, 2024-08-20T21:51:24.5727461Z "digest": "sha256:5843bb8b81bb68867db16a80d2ebb31396b14b3fa08b419ce9ab07b0791eb662" 2024-08-20T21:51:24.5728242Z }, 2024-08-20T21:51:24.5728545Z { 2024-08-20T21:51:24.5729063Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5729760Z "size": 699, 2024-08-20T21:51:24.5730437Z "digest": "sha256:28e54511f1b116a9d632caa0c4be2cd1b6ca7542f815560ec598b81d6d956d31" 2024-08-20T21:51:24.5731193Z }, 2024-08-20T21:51:24.5731498Z { 2024-08-20T21:51:24.5732034Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5732705Z "size": 141, 2024-08-20T21:51:24.5733371Z "digest": "sha256:222174e4e2b25c47241345129f9ec8c9c219ef78391d65f6089d89339be6ddb7" 2024-08-20T21:51:24.5734120Z }, 2024-08-20T21:51:24.5734402Z { 2024-08-20T21:51:24.5734937Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5735633Z "size": 135, 2024-08-20T21:51:24.5736386Z "digest": "sha256:8d55064a9f207c06db9da40e295a3430ac0b288cfdf6968903dddb487789b380" 2024-08-20T21:51:24.5737184Z }, 2024-08-20T21:51:24.5737489Z { 2024-08-20T21:51:24.5738008Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5738705Z "size": 32, 2024-08-20T21:51:24.5739385Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:51:24.5740142Z }, 2024-08-20T21:51:24.5740452Z { 2024-08-20T21:51:24.5740990Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5741661Z "size": 194, 2024-08-20T21:51:24.5742330Z "digest": "sha256:25114d5bae5b7775201e9a1f79c06cc0ab7ca238a8357e83018a63d782aef5d5" 2024-08-20T21:51:24.5743101Z }, 2024-08-20T21:51:24.5743382Z { 2024-08-20T21:51:24.5743921Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5744618Z "size": 565, 2024-08-20T21:51:24.5745267Z "digest": "sha256:7dd58e631031354e323ac12b1e8223aa4b198de49bd1ac0d10d11ee97933d4a8" 2024-08-20T21:51:24.5746034Z }, 2024-08-20T21:51:24.5746347Z { 2024-08-20T21:51:24.5746864Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5747557Z "size": 43162887, 2024-08-20T21:51:24.5748245Z "digest": "sha256:b125137ca92e3bd051d165ca69937cd4939aa82b5095a4379e3b940187fdb4fe" 2024-08-20T21:51:24.5748988Z }, 2024-08-20T21:51:24.5749290Z { 2024-08-20T21:51:24.5749824Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5750494Z "size": 104, 2024-08-20T21:51:24.5751174Z "digest": "sha256:12dfa73284bb2bfe40228d03e187cbf9f4d59c5300419db1d5d562ad9ca4a5cd" 2024-08-20T21:51:24.5751958Z }, 2024-08-20T21:51:24.5752242Z { 2024-08-20T21:51:24.5752779Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5753468Z "size": 1379, 2024-08-20T21:51:24.5754119Z "digest": "sha256:045427066d9a1cb897779196ada05b7b790c1dce05bead327f658f20933e9a76" 2024-08-20T21:51:24.5754881Z }, 2024-08-20T21:51:24.5755183Z { 2024-08-20T21:51:24.5755705Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5756405Z "size": 699, 2024-08-20T21:51:24.5757077Z "digest": "sha256:28e54511f1b116a9d632caa0c4be2cd1b6ca7542f815560ec598b81d6d956d31" 2024-08-20T21:51:24.5757825Z }, 2024-08-20T21:51:24.5758134Z { 2024-08-20T21:51:24.5758673Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5759347Z "size": 139, 2024-08-20T21:51:24.5760033Z "digest": "sha256:b6428379fcaafd1025a47060dccafde4f6ac9d22475fff6ce5e51fc531bd9d98" 2024-08-20T21:51:24.5760818Z }, 2024-08-20T21:51:24.5761218Z { 2024-08-20T21:51:24.5761761Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5762456Z "size": 119, 2024-08-20T21:51:24.5763119Z "digest": "sha256:7f8afc2d089702d4d576f8a6a8841b21e66c0acbf3b735f76a4fe1c4cb0f1fa2" 2024-08-20T21:51:24.5763895Z }, 2024-08-20T21:51:24.5764198Z { 2024-08-20T21:51:24.5764724Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5765424Z "size": 4327084570, 2024-08-20T21:51:24.5766133Z "digest": "sha256:ae2172ebb76524e0aabfc93971b239448922f1f340cb20dbe61460dbd250f202" 2024-08-20T21:51:24.5766883Z }, 2024-08-20T21:51:24.5767190Z { 2024-08-20T21:51:24.5767726Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5768398Z "size": 173, 2024-08-20T21:51:24.5769086Z "digest": "sha256:c6cba7526eea3b7a7febd01cb06e32f421ab15e17a3692967040d95eb0675cca" 2024-08-20T21:51:24.5769875Z }, 2024-08-20T21:51:24.5770158Z { 2024-08-20T21:51:24.5770706Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5771400Z "size": 907, 2024-08-20T21:51:24.5772059Z "digest": "sha256:5be7fac2e2a0876700857b9032bc92b1f3b8470f6bcff815e18101d7d71c4054" 2024-08-20T21:51:24.5772834Z }, 2024-08-20T21:51:24.5773138Z { 2024-08-20T21:51:24.5773655Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5774442Z "size": 699, 2024-08-20T21:51:24.5775136Z "digest": "sha256:28e54511f1b116a9d632caa0c4be2cd1b6ca7542f815560ec598b81d6d956d31" 2024-08-20T21:51:24.5775889Z }, 2024-08-20T21:51:24.5776193Z { 2024-08-20T21:51:24.5776731Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5777403Z "size": 134, 2024-08-20T21:51:24.5778083Z "digest": "sha256:eea60c1ffc991e92d764514ee25756132c369fa2fc0de806c6f1f8eb64718a5b" 2024-08-20T21:51:24.5778858Z }, 2024-08-20T21:51:24.5779140Z { 2024-08-20T21:51:24.5779680Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5780381Z "size": 32, 2024-08-20T21:51:24.5781037Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:51:24.5781817Z }, 2024-08-20T21:51:24.5782120Z { 2024-08-20T21:51:24.5782636Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5783331Z "size": 158, 2024-08-20T21:51:24.5784020Z "digest": "sha256:e4b1fa4f961f28a8004b20020e9f3e14ddda223a8cf6273a7ec5dfc7b58a9b04" 2024-08-20T21:51:24.5784777Z }, 2024-08-20T21:51:24.5785081Z { 2024-08-20T21:51:24.5785619Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5786293Z "size": 1841, 2024-08-20T21:51:24.5786992Z "digest": "sha256:ff20aefbe592a1fd39676d339cf9ebce003fde8a0b5702b8301d756bdee912c7" 2024-08-20T21:51:24.5787785Z }, 2024-08-20T21:51:24.5788072Z { 2024-08-20T21:51:24.5788610Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5789315Z "size": 7529768, 2024-08-20T21:51:24.5789981Z "digest": "sha256:39f017df093b119d169de8c649be9cf342a3075430b27bf2388dc9ccdefafac8" 2024-08-20T21:51:24.5790753Z }, 2024-08-20T21:51:24.5791059Z { 2024-08-20T21:51:24.5791576Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5792269Z "size": 163, 2024-08-20T21:51:24.5793603Z "digest": "sha256:dcb4b1c5a6f604842cbe094756013849c141d2c73b9764babb9b3cf35684a797" 2024-08-20T21:51:24.5794366Z }, 2024-08-20T21:51:24.5794671Z { 2024-08-20T21:51:24.5795214Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5795889Z "size": 7943, 2024-08-20T21:51:24.5796558Z "digest": "sha256:862b4f5c5cd13e6c7641578efd673953421540d1b64af3e8a362b709a3ef443a" 2024-08-20T21:51:24.5797320Z }, 2024-08-20T21:51:24.5797602Z { 2024-08-20T21:51:24.5798140Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5798833Z "size": 8070, 2024-08-20T21:51:24.5799839Z "digest": "sha256:46546d0a53bb910aa9a28f5073146d18a700ca0d1ca7407044085170059104cf" 2024-08-20T21:51:24.5800595Z }, 2024-08-20T21:51:24.5800897Z { 2024-08-20T21:51:24.5801412Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5802108Z "size": 301, 2024-08-20T21:51:24.5802802Z "digest": "sha256:82a31e7860af47e63f76dcfa09bf02b9ed5ba7f0c2c17804cbdd4e4f9dc7ab1d" 2024-08-20T21:51:24.5803560Z }, 2024-08-20T21:51:24.5803863Z { 2024-08-20T21:51:24.5804399Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5805070Z "size": 7629503, 2024-08-20T21:51:24.5805777Z "digest": "sha256:ac2f6809f82b0e504eb4e1bfada74e84542eaff1a26c06047ffecdcabc8f2f04" 2024-08-20T21:51:24.5806572Z }, 2024-08-20T21:51:24.5806852Z { 2024-08-20T21:51:24.5807387Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5808079Z "size": 108, 2024-08-20T21:51:24.5808737Z "digest": "sha256:2b45e44613094d77c76cfbe3d12bd0a314f3cfea24cf97303d9c6ee51715a021" 2024-08-20T21:51:24.5809519Z }, 2024-08-20T21:51:24.5809821Z { 2024-08-20T21:51:24.5810373Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5811065Z "size": 54145661, 2024-08-20T21:51:24.5811754Z "digest": "sha256:403462e9ee27186eada6b3b99891b31b4394b51e12d017f8fc1c302e33444a8a" 2024-08-20T21:51:24.5812622Z }, 2024-08-20T21:51:24.5812948Z { 2024-08-20T21:51:24.5813487Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5814159Z "size": 472, 2024-08-20T21:51:24.5814812Z "digest": "sha256:889f478638723a13f19e34f1f415d0c7dd7070c02484cb105d03601eb16e45f4" 2024-08-20T21:51:24.5815560Z }, 2024-08-20T21:51:24.5815844Z { 2024-08-20T21:51:24.5816385Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5817080Z "size": 1373229330, 2024-08-20T21:51:24.5817757Z "digest": "sha256:23898c5065b1e76bbf101c2151c1f06c50633abb686f1a87ee578de1acce2eaf" 2024-08-20T21:51:24.5818528Z }, 2024-08-20T21:51:24.5818830Z { 2024-08-20T21:51:24.5819344Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5820039Z "size": 106, 2024-08-20T21:51:24.5820707Z "digest": "sha256:f1caf060fc92271565fb07a66ff67d61a3bb761687913767cebd4240754275db" 2024-08-20T21:51:24.5821471Z }, 2024-08-20T21:51:24.5821759Z { 2024-08-20T21:51:24.5822340Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5823038Z "size": 559, 2024-08-20T21:51:24.5823681Z "digest": "sha256:621b6cedbe16b797e97c910d222978f9f3077752e0386617189079966c30ee02" 2024-08-20T21:51:24.5824436Z }, 2024-08-20T21:51:24.5824736Z { 2024-08-20T21:51:24.5825251Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5825943Z "size": 36532053, 2024-08-20T21:51:24.5826630Z "digest": "sha256:b9c038a9eb4ac830af69e080b96c2d591156429f179e4e922d025624341693d4" 2024-08-20T21:51:24.5827384Z }, 2024-08-20T21:51:24.5827684Z { 2024-08-20T21:51:24.5828220Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5828888Z "size": 111, 2024-08-20T21:51:24.5829568Z "digest": "sha256:74046cd1f067b5edf32140ab31499a79acf8919e57afd4bc4debeafeb7f993ed" 2024-08-20T21:51:24.5830342Z }, 2024-08-20T21:51:24.5830623Z { 2024-08-20T21:51:24.5831165Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5831859Z "size": 32, 2024-08-20T21:51:24.5832518Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:51:24.5833294Z }, 2024-08-20T21:51:24.5833626Z { 2024-08-20T21:51:24.5834138Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5834834Z "size": 32, 2024-08-20T21:51:24.5835511Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:51:24.5836265Z }, 2024-08-20T21:51:24.5836695Z { 2024-08-20T21:51:24.5837236Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5837903Z "size": 32, 2024-08-20T21:51:24.5838577Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:51:24.5839353Z }, 2024-08-20T21:51:24.5839631Z { 2024-08-20T21:51:24.5840166Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T21:51:24.5840859Z "size": 32, 2024-08-20T21:51:24.5841514Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T21:51:24.5842283Z } 2024-08-20T21:51:24.5842580Z ] 2024-08-20T21:51:24.5842859Z } 2024-08-20T21:51:24.5981017Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T21:51:24.5981651Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T21:51:24.5982334Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-20T21:51:24.5989844Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:24.5990481Z env: 2024-08-20T21:51:24.5990817Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:24.5992108Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:24.5994190Z ##[endgroup] 2024-08-20T21:51:24.6026164Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9-f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:24.6091000Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-20T21:51:24.6091730Z with: 2024-08-20T21:51:24.6093595Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:24.6095114Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:24.6095740Z env: 2024-08-20T21:51:24.6096075Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:24.6096509Z ##[endgroup] 2024-08-20T21:51:24.6120821Z ##[group]Run set -x 2024-08-20T21:51:24.6121234Z set -x 2024-08-20T21:51:24.6121601Z set +e 2024-08-20T21:51:24.6121953Z  2024-08-20T21:51:24.6122335Z login() { 2024-08-20T21:51:24.6123157Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T21:51:24.6124047Z } 2024-08-20T21:51:24.6124362Z  2024-08-20T21:51:24.6124745Z retry () { 2024-08-20T21:51:24.6125234Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T21:51:24.6125784Z } 2024-08-20T21:51:24.6126097Z  2024-08-20T21:51:24.6126478Z retry login "${DOCKER_REGISTRY}" 2024-08-20T21:51:24.6126985Z  2024-08-20T21:51:24.6127292Z set -e 2024-08-20T21:51:24.6127873Z # ignore output since only exit code is used for conditional 2024-08-20T21:51:24.6128744Z # only pull docker image if it's not available locally 2024-08-20T21:51:24.6129707Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-20T21:51:24.6130548Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-20T21:51:24.6131087Z fi 2024-08-20T21:51:24.6137460Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:51:24.6138062Z env: 2024-08-20T21:51:24.6138396Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:51:24.6139668Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:24.6141123Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:24.6141766Z ##[endgroup] 2024-08-20T21:51:24.6169118Z + set +e 2024-08-20T21:51:24.6169787Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:24.6170600Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:24.6173239Z + aws ecr get-login-password --region us-east-1 2024-08-20T21:51:24.6174665Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T21:51:25.2897514Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T21:51:25.2898534Z Configure a credential helper to remove this warning. See 2024-08-20T21:51:25.2899598Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T21:51:25.2900254Z 2024-08-20T21:51:25.2900430Z Login Succeeded 2024-08-20T21:51:25.2910937Z + set -e 2024-08-20T21:51:25.2912423Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:25.3086128Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:25.3088517Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:51:25.5278357Z f6d216893d65c7b8ae43df4daaf247db808378e9: Pulling from pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9 2024-08-20T21:51:25.5279524Z 7a2c55901189: Pulling fs layer 2024-08-20T21:51:25.5280024Z 09d415c238d7: Pulling fs layer 2024-08-20T21:51:25.5280504Z 9fe6e2e61518: Pulling fs layer 2024-08-20T21:51:25.5280955Z 41f16248e682: Pulling fs layer 2024-08-20T21:51:25.5281723Z 95d7b7817039: Pulling fs layer 2024-08-20T21:51:25.5282242Z 8f6c90485347: Pulling fs layer 2024-08-20T21:51:25.5282697Z ab17245097e4: Pulling fs layer 2024-08-20T21:51:25.5283171Z dfecd7e9912b: Pulling fs layer 2024-08-20T21:51:25.5283647Z 464a8f745445: Pulling fs layer 2024-08-20T21:51:25.5284079Z ab17245097e4: Waiting 2024-08-20T21:51:25.5284497Z 41f16248e682: Waiting 2024-08-20T21:51:25.5284889Z 464a8f745445: Waiting 2024-08-20T21:51:25.5285263Z 8f6c90485347: Waiting 2024-08-20T21:51:25.5285666Z dfecd7e9912b: Waiting 2024-08-20T21:51:25.5286111Z d67c111fa588: Pulling fs layer 2024-08-20T21:51:25.5286558Z 0723abde19c0: Pulling fs layer 2024-08-20T21:51:25.5287045Z 72d2b53f7a01: Pulling fs layer 2024-08-20T21:51:25.5287525Z dbc6fccd9c70: Pulling fs layer 2024-08-20T21:51:25.5288353Z ff6a6ddc7310: Pulling fs layer 2024-08-20T21:51:25.5289261Z d67c111fa588: Waiting 2024-08-20T21:51:25.5289954Z fac2b1b246ed: Pulling fs layer 2024-08-20T21:51:25.5290423Z bc574f79516b: Pulling fs layer 2024-08-20T21:51:25.5291177Z 72d2b53f7a01: Waiting 2024-08-20T21:51:25.5291594Z 0723abde19c0: Waiting 2024-08-20T21:51:25.5292142Z c23d42adcade: Pulling fs layer 2024-08-20T21:51:25.5293566Z 59b0a331db9f: Pulling fs layer 2024-08-20T21:51:25.5294479Z ff6a6ddc7310: Waiting 2024-08-20T21:51:25.5295156Z bc574f79516b: Waiting 2024-08-20T21:51:25.5295825Z dbc6fccd9c70: Waiting 2024-08-20T21:51:25.5296252Z fac2b1b246ed: Waiting 2024-08-20T21:51:25.5296639Z c23d42adcade: Waiting 2024-08-20T21:51:25.5297065Z 478a9a2bdefc: Pulling fs layer 2024-08-20T21:51:25.5297526Z 28e54511f1b1: Pulling fs layer 2024-08-20T21:51:25.5298008Z 5247f18de881: Pulling fs layer 2024-08-20T21:51:25.5298477Z 38760708e2d7: Pulling fs layer 2024-08-20T21:51:25.5298932Z 863dbde6ed0c: Pulling fs layer 2024-08-20T21:51:25.5299408Z f283ab568a00: Pulling fs layer 2024-08-20T21:51:25.5299879Z 3e39f30b1d5c: Pulling fs layer 2024-08-20T21:51:25.5300326Z 521c3d72534c: Pulling fs layer 2024-08-20T21:51:25.5300778Z 59b0a331db9f: Waiting 2024-08-20T21:51:25.5301181Z 28e54511f1b1: Waiting 2024-08-20T21:51:25.5301558Z 5247f18de881: Waiting 2024-08-20T21:51:25.5302032Z a975a4480a6d: Pulling fs layer 2024-08-20T21:51:25.5302490Z 863dbde6ed0c: Waiting 2024-08-20T21:51:25.5302888Z f283ab568a00: Waiting 2024-08-20T21:51:25.5303282Z 1f2c278212ad: Pulling fs layer 2024-08-20T21:51:25.5303756Z b721dd867e80: Pulling fs layer 2024-08-20T21:51:25.5304231Z 7a74b7ad649d: Pulling fs layer 2024-08-20T21:51:25.5304657Z 38760708e2d7: Waiting 2024-08-20T21:51:25.5305056Z a975a4480a6d: Waiting 2024-08-20T21:51:25.5305478Z b53e1ad1deec: Pulling fs layer 2024-08-20T21:51:25.5307087Z 016dc3faa7ef: Pulling fs layer 2024-08-20T21:51:25.5307969Z 1f2c278212ad: Waiting 2024-08-20T21:51:25.5308395Z b721dd867e80: Waiting 2024-08-20T21:51:25.5308795Z 384dce3a4ba2: Pulling fs layer 2024-08-20T21:51:25.5309380Z 75461ad9f7bb: Pulling fs layer 2024-08-20T21:51:25.5310249Z b53e1ad1deec: Waiting 2024-08-20T21:51:25.5310930Z eb2062102d11: Pulling fs layer 2024-08-20T21:51:25.5311752Z 3cc17f105239: Pulling fs layer 2024-08-20T21:51:25.5312598Z 016dc3faa7ef: Waiting 2024-08-20T21:51:25.5313325Z 585501be0799: Pulling fs layer 2024-08-20T21:51:25.5314115Z 3e39f30b1d5c: Waiting 2024-08-20T21:51:25.5314919Z 6c49da68c9b4: Pulling fs layer 2024-08-20T21:51:25.5315811Z 3cc17f105239: Waiting 2024-08-20T21:51:25.5316609Z 497539689260: Pulling fs layer 2024-08-20T21:51:25.5317557Z 5843bb8b81bb: Pulling fs layer 2024-08-20T21:51:25.5318410Z 585501be0799: Waiting 2024-08-20T21:51:25.5319033Z 6c49da68c9b4: Waiting 2024-08-20T21:51:25.5319662Z 222174e4e2b2: Pulling fs layer 2024-08-20T21:51:25.5320330Z 497539689260: Waiting 2024-08-20T21:51:25.5320990Z 8d55064a9f20: Pulling fs layer 2024-08-20T21:51:25.5321739Z 4f4fb700ef54: Pulling fs layer 2024-08-20T21:51:25.5322621Z 25114d5bae5b: Pulling fs layer 2024-08-20T21:51:25.5323447Z 7dd58e631031: Pulling fs layer 2024-08-20T21:51:25.5324259Z b125137ca92e: Pulling fs layer 2024-08-20T21:51:25.5324722Z 12dfa73284bb: Pulling fs layer 2024-08-20T21:51:25.5325438Z 045427066d9a: Pulling fs layer 2024-08-20T21:51:25.5325935Z 25114d5bae5b: Waiting 2024-08-20T21:51:25.5326543Z 7dd58e631031: Waiting 2024-08-20T21:51:25.5327289Z b125137ca92e: Waiting 2024-08-20T21:51:25.5328025Z 8d55064a9f20: Waiting 2024-08-20T21:51:25.5328788Z b6428379fcaa: Pulling fs layer 2024-08-20T21:51:25.5329356Z 045427066d9a: Waiting 2024-08-20T21:51:25.5329771Z 7f8afc2d0897: Pulling fs layer 2024-08-20T21:51:25.5330208Z 12dfa73284bb: Waiting 2024-08-20T21:51:25.5330630Z ae2172ebb765: Pulling fs layer 2024-08-20T21:51:25.5331107Z c6cba7526eea: Pulling fs layer 2024-08-20T21:51:25.5331577Z 5be7fac2e2a0: Pulling fs layer 2024-08-20T21:51:25.5332053Z eea60c1ffc99: Pulling fs layer 2024-08-20T21:51:25.5332511Z 5be7fac2e2a0: Waiting 2024-08-20T21:51:25.5332886Z 222174e4e2b2: Waiting 2024-08-20T21:51:25.5333303Z e4b1fa4f961f: Pulling fs layer 2024-08-20T21:51:25.5333796Z ff20aefbe592: Pulling fs layer 2024-08-20T21:51:25.5334226Z e4b1fa4f961f: Waiting 2024-08-20T21:51:25.5334640Z 39f017df093b: Pulling fs layer 2024-08-20T21:51:25.5335122Z dcb4b1c5a6f6: Pulling fs layer 2024-08-20T21:51:25.5335578Z 862b4f5c5cd1: Pulling fs layer 2024-08-20T21:51:25.5336034Z ff20aefbe592: Waiting 2024-08-20T21:51:25.5336430Z 46546d0a53bb: Pulling fs layer 2024-08-20T21:51:25.5336877Z 39f017df093b: Waiting 2024-08-20T21:51:25.5337289Z 82a31e7860af: Pulling fs layer 2024-08-20T21:51:25.5337738Z ac2f6809f82b: Pulling fs layer 2024-08-20T21:51:25.5338208Z 2b45e4461309: Pulling fs layer 2024-08-20T21:51:25.5338655Z 82a31e7860af: Waiting 2024-08-20T21:51:25.5339046Z 403462e9ee27: Pulling fs layer 2024-08-20T21:51:25.5339522Z 889f47863872: Pulling fs layer 2024-08-20T21:51:25.5339983Z 23898c5065b1: Pulling fs layer 2024-08-20T21:51:25.5340434Z f1caf060fc92: Pulling fs layer 2024-08-20T21:51:25.5340888Z ac2f6809f82b: Waiting 2024-08-20T21:51:25.5341307Z 621b6cedbe16: Pulling fs layer 2024-08-20T21:51:25.5341738Z ae2172ebb765: Waiting 2024-08-20T21:51:25.5342159Z b9c038a9eb4a: Pulling fs layer 2024-08-20T21:51:25.5342606Z 889f47863872: Waiting 2024-08-20T21:51:25.5342985Z f1caf060fc92: Waiting 2024-08-20T21:51:25.5343380Z 23898c5065b1: Waiting 2024-08-20T21:51:25.5343790Z 74046cd1f067: Pulling fs layer 2024-08-20T21:51:25.5344221Z 621b6cedbe16: Waiting 2024-08-20T21:51:25.5344613Z 403462e9ee27: Waiting 2024-08-20T21:51:25.5345009Z dcb4b1c5a6f6: Waiting 2024-08-20T21:51:25.5345390Z 7f8afc2d0897: Waiting 2024-08-20T21:51:25.5345784Z 2b45e4461309: Waiting 2024-08-20T21:51:25.6611466Z 09d415c238d7: Verifying Checksum 2024-08-20T21:51:25.6612022Z 09d415c238d7: Download complete 2024-08-20T21:51:25.7413688Z 41f16248e682: Download complete 2024-08-20T21:51:25.8115802Z 95d7b7817039: Verifying Checksum 2024-08-20T21:51:25.8116326Z 95d7b7817039: Download complete 2024-08-20T21:51:25.8724243Z 7a2c55901189: Verifying Checksum 2024-08-20T21:51:25.8724744Z 7a2c55901189: Download complete 2024-08-20T21:51:25.9628308Z ab17245097e4: Verifying Checksum 2024-08-20T21:51:25.9629373Z ab17245097e4: Download complete 2024-08-20T21:51:26.0476835Z dfecd7e9912b: Download complete 2024-08-20T21:51:26.1257683Z 464a8f745445: Verifying Checksum 2024-08-20T21:51:26.1258709Z 464a8f745445: Download complete 2024-08-20T21:51:26.1551357Z 9fe6e2e61518: Verifying Checksum 2024-08-20T21:51:26.1551916Z 9fe6e2e61518: Download complete 2024-08-20T21:51:26.2357743Z 0723abde19c0: Download complete 2024-08-20T21:51:26.3103556Z 72d2b53f7a01: Verifying Checksum 2024-08-20T21:51:26.3104165Z 72d2b53f7a01: Download complete 2024-08-20T21:51:27.1648055Z 7a2c55901189: Pull complete 2024-08-20T21:51:27.5125769Z 09d415c238d7: Pull complete 2024-08-20T21:51:28.5444194Z 9fe6e2e61518: Pull complete 2024-08-20T21:51:28.5672851Z 41f16248e682: Pull complete 2024-08-20T21:51:28.5886534Z 95d7b7817039: Pull complete 2024-08-20T21:51:32.9326827Z dbc6fccd9c70: Verifying Checksum 2024-08-20T21:51:32.9327435Z dbc6fccd9c70: Download complete 2024-08-20T21:51:33.0034883Z ff6a6ddc7310: Download complete 2024-08-20T21:51:33.0832984Z fac2b1b246ed: Verifying Checksum 2024-08-20T21:51:33.0833507Z fac2b1b246ed: Download complete 2024-08-20T21:51:33.1635979Z bc574f79516b: Download complete 2024-08-20T21:51:34.1367435Z c23d42adcade: Verifying Checksum 2024-08-20T21:51:34.1368134Z c23d42adcade: Download complete 2024-08-20T21:51:34.2067430Z 59b0a331db9f: Verifying Checksum 2024-08-20T21:51:34.2067971Z 59b0a331db9f: Download complete 2024-08-20T21:51:34.2788424Z 478a9a2bdefc: Verifying Checksum 2024-08-20T21:51:34.2788971Z 478a9a2bdefc: Download complete 2024-08-20T21:51:34.3674728Z 28e54511f1b1: Verifying Checksum 2024-08-20T21:51:34.3675283Z 28e54511f1b1: Download complete 2024-08-20T21:51:39.6965694Z 8f6c90485347: Verifying Checksum 2024-08-20T21:51:39.6966525Z 8f6c90485347: Download complete 2024-08-20T21:51:39.7612094Z 38760708e2d7: Verifying Checksum 2024-08-20T21:51:39.7612648Z 38760708e2d7: Download complete 2024-08-20T21:51:39.8311049Z 863dbde6ed0c: Download complete 2024-08-20T21:51:39.9005028Z f283ab568a00: Download complete 2024-08-20T21:51:39.9700722Z 3e39f30b1d5c: Verifying Checksum 2024-08-20T21:51:39.9701868Z 3e39f30b1d5c: Download complete 2024-08-20T21:51:41.0671683Z 521c3d72534c: Verifying Checksum 2024-08-20T21:51:41.0672292Z 521c3d72534c: Download complete 2024-08-20T21:51:42.3352051Z a975a4480a6d: Verifying Checksum 2024-08-20T21:51:42.3353162Z a975a4480a6d: Download complete 2024-08-20T21:51:42.4252436Z 1f2c278212ad: Verifying Checksum 2024-08-20T21:51:42.4253537Z 1f2c278212ad: Download complete 2024-08-20T21:51:42.5087190Z b721dd867e80: Download complete 2024-08-20T21:51:42.5990146Z 7a74b7ad649d: Verifying Checksum 2024-08-20T21:51:42.5990730Z 7a74b7ad649d: Download complete 2024-08-20T21:51:42.6835703Z b53e1ad1deec: Verifying Checksum 2024-08-20T21:51:42.6836332Z b53e1ad1deec: Download complete 2024-08-20T21:51:42.7516657Z 016dc3faa7ef: Download complete 2024-08-20T21:51:48.7516857Z 384dce3a4ba2: Verifying Checksum 2024-08-20T21:51:48.7517462Z 384dce3a4ba2: Download complete 2024-08-20T21:51:48.8420649Z 75461ad9f7bb: Verifying Checksum 2024-08-20T21:51:48.8421571Z 75461ad9f7bb: Download complete 2024-08-20T21:51:48.9218369Z eb2062102d11: Verifying Checksum 2024-08-20T21:51:48.9218952Z eb2062102d11: Download complete 2024-08-20T21:51:49.3673757Z 3cc17f105239: Verifying Checksum 2024-08-20T21:51:49.3674385Z 3cc17f105239: Download complete 2024-08-20T21:51:49.4419382Z 585501be0799: Verifying Checksum 2024-08-20T21:51:49.4420461Z 585501be0799: Download complete 2024-08-20T21:51:49.5163359Z 6c49da68c9b4: Verifying Checksum 2024-08-20T21:51:49.5164419Z 6c49da68c9b4: Download complete 2024-08-20T21:51:49.7792424Z 497539689260: Verifying Checksum 2024-08-20T21:51:49.7793703Z 497539689260: Download complete 2024-08-20T21:51:49.8619615Z 5843bb8b81bb: Download complete 2024-08-20T21:51:49.9483484Z 222174e4e2b2: Download complete 2024-08-20T21:51:50.0331618Z 8d55064a9f20: Verifying Checksum 2024-08-20T21:51:50.0332562Z 8d55064a9f20: Download complete 2024-08-20T21:51:50.0433187Z 4f4fb700ef54: Verifying Checksum 2024-08-20T21:51:50.0433733Z 4f4fb700ef54: Download complete 2024-08-20T21:51:50.1189998Z 25114d5bae5b: Download complete 2024-08-20T21:51:50.1854333Z 7dd58e631031: Verifying Checksum 2024-08-20T21:51:50.1854906Z 7dd58e631031: Download complete 2024-08-20T21:51:50.6337749Z d67c111fa588: Verifying Checksum 2024-08-20T21:51:50.6338336Z d67c111fa588: Download complete 2024-08-20T21:51:50.6788778Z b125137ca92e: Verifying Checksum 2024-08-20T21:51:50.6789400Z b125137ca92e: Download complete 2024-08-20T21:51:50.7085041Z 12dfa73284bb: Verifying Checksum 2024-08-20T21:51:50.7085569Z 12dfa73284bb: Download complete 2024-08-20T21:51:50.7420484Z 045427066d9a: Download complete 2024-08-20T21:51:50.8109434Z b6428379fcaa: Verifying Checksum 2024-08-20T21:51:50.8109991Z b6428379fcaa: Download complete 2024-08-20T21:51:50.8155377Z 7f8afc2d0897: Verifying Checksum 2024-08-20T21:51:50.8155879Z 7f8afc2d0897: Download complete 2024-08-20T21:51:50.8922564Z c6cba7526eea: Download complete 2024-08-20T21:51:50.9719790Z 5be7fac2e2a0: Verifying Checksum 2024-08-20T21:51:50.9720375Z 5be7fac2e2a0: Download complete 2024-08-20T21:51:51.0405617Z eea60c1ffc99: Verifying Checksum 2024-08-20T21:51:51.0406228Z eea60c1ffc99: Download complete 2024-08-20T21:51:51.1126345Z e4b1fa4f961f: Verifying Checksum 2024-08-20T21:51:51.1126976Z e4b1fa4f961f: Download complete 2024-08-20T21:51:51.2060364Z ff20aefbe592: Verifying Checksum 2024-08-20T21:51:51.2060942Z ff20aefbe592: Download complete 2024-08-20T21:51:51.3402590Z 39f017df093b: Verifying Checksum 2024-08-20T21:51:51.3403185Z 39f017df093b: Download complete 2024-08-20T21:51:51.4182328Z dcb4b1c5a6f6: Download complete 2024-08-20T21:51:51.4822106Z 862b4f5c5cd1: Download complete 2024-08-20T21:51:51.6641251Z 82a31e7860af: Verifying Checksum 2024-08-20T21:51:51.6641961Z 82a31e7860af: Download complete 2024-08-20T21:51:51.9269307Z ac2f6809f82b: Verifying Checksum 2024-08-20T21:51:51.9270042Z ac2f6809f82b: Download complete 2024-08-20T21:51:52.0076776Z 2b45e4461309: Verifying Checksum 2024-08-20T21:51:52.0077392Z 2b45e4461309: Download complete 2024-08-20T21:51:52.6724313Z 403462e9ee27: Verifying Checksum 2024-08-20T21:51:52.6725121Z 403462e9ee27: Download complete 2024-08-20T21:51:52.7405689Z 889f47863872: Verifying Checksum 2024-08-20T21:51:52.7406248Z 889f47863872: Download complete 2024-08-20T21:51:57.6862242Z 8f6c90485347: Pull complete 2024-08-20T21:51:57.9475760Z ab17245097e4: Pull complete 2024-08-20T21:51:58.2204470Z dfecd7e9912b: Pull complete 2024-08-20T21:51:58.5875248Z 464a8f745445: Pull complete 2024-08-20T21:52:03.3399819Z 5247f18de881: Verifying Checksum 2024-08-20T21:52:03.3400410Z 5247f18de881: Download complete 2024-08-20T21:52:03.4373412Z f1caf060fc92: Verifying Checksum 2024-08-20T21:52:03.4374067Z f1caf060fc92: Download complete 2024-08-20T21:52:03.5229398Z 621b6cedbe16: Verifying Checksum 2024-08-20T21:52:03.5229976Z 621b6cedbe16: Download complete 2024-08-20T21:52:03.9693521Z b9c038a9eb4a: Verifying Checksum 2024-08-20T21:52:03.9694092Z b9c038a9eb4a: Download complete 2024-08-20T21:52:04.0490067Z 74046cd1f067: Download complete 2024-08-20T21:52:06.5347150Z 23898c5065b1: Verifying Checksum 2024-08-20T21:52:06.5347776Z 23898c5065b1: Download complete 2024-08-20T21:52:32.6569783Z d67c111fa588: Pull complete 2024-08-20T21:52:33.1135409Z 0723abde19c0: Pull complete 2024-08-20T21:52:33.5601868Z 72d2b53f7a01: Pull complete 2024-08-20T21:52:34.1425327Z ae2172ebb765: Verifying Checksum 2024-08-20T21:52:34.1426324Z ae2172ebb765: Download complete 2024-08-20T21:52:47.3897786Z dbc6fccd9c70: Pull complete 2024-08-20T21:52:47.8567757Z ff6a6ddc7310: Pull complete 2024-08-20T21:52:48.1202540Z fac2b1b246ed: Pull complete 2024-08-20T21:52:48.4833086Z bc574f79516b: Pull complete 2024-08-20T21:52:51.7328305Z c23d42adcade: Pull complete 2024-08-20T21:52:52.1866112Z 59b0a331db9f: Pull complete 2024-08-20T21:52:52.6425149Z 478a9a2bdefc: Pull complete 2024-08-20T21:52:53.0899151Z 28e54511f1b1: Pull complete 2024-08-20T21:53:58.3454350Z 5247f18de881: Pull complete 2024-08-20T21:53:58.7600721Z 38760708e2d7: Pull complete 2024-08-20T21:53:59.2403263Z 863dbde6ed0c: Pull complete 2024-08-20T21:53:59.6970388Z f283ab568a00: Pull complete 2024-08-20T21:54:00.1560256Z 3e39f30b1d5c: Pull complete 2024-08-20T21:54:00.5996874Z 521c3d72534c: Pull complete 2024-08-20T21:54:04.6519303Z a975a4480a6d: Pull complete 2024-08-20T21:54:04.9032539Z 1f2c278212ad: Pull complete 2024-08-20T21:54:05.1423890Z b721dd867e80: Pull complete 2024-08-20T21:54:05.5674645Z 7a74b7ad649d: Pull complete 2024-08-20T21:54:05.9430728Z b53e1ad1deec: Pull complete 2024-08-20T21:54:06.1688208Z 016dc3faa7ef: Pull complete 2024-08-20T21:54:18.5942744Z 384dce3a4ba2: Pull complete 2024-08-20T21:54:19.0479404Z 75461ad9f7bb: Pull complete 2024-08-20T21:54:19.4010584Z eb2062102d11: Pull complete 2024-08-20T21:54:20.7385260Z 3cc17f105239: Pull complete 2024-08-20T21:54:21.1891280Z 585501be0799: Pull complete 2024-08-20T21:54:21.6507402Z 6c49da68c9b4: Pull complete 2024-08-20T21:54:22.4568896Z 497539689260: Pull complete 2024-08-20T21:54:22.7037540Z 5843bb8b81bb: Pull complete 2024-08-20T21:54:23.6023143Z 222174e4e2b2: Pull complete 2024-08-20T21:54:24.0118692Z 8d55064a9f20: Pull complete 2024-08-20T21:54:24.4368706Z 4f4fb700ef54: Pull complete 2024-08-20T21:54:24.8737021Z 25114d5bae5b: Pull complete 2024-08-20T21:54:25.3082174Z 7dd58e631031: Pull complete 2024-08-20T21:54:28.5082167Z b125137ca92e: Pull complete 2024-08-20T21:54:28.9566698Z 12dfa73284bb: Pull complete 2024-08-20T21:54:29.4196215Z 045427066d9a: Pull complete 2024-08-20T21:54:30.2358286Z b6428379fcaa: Pull complete 2024-08-20T21:54:30.6257068Z 7f8afc2d0897: Pull complete 2024-08-20T21:56:06.8103255Z ae2172ebb765: Pull complete 2024-08-20T21:56:07.2723842Z c6cba7526eea: Pull complete 2024-08-20T21:56:07.7240133Z 5be7fac2e2a0: Pull complete 2024-08-20T21:56:08.6489300Z eea60c1ffc99: Pull complete 2024-08-20T21:56:09.5609406Z e4b1fa4f961f: Pull complete 2024-08-20T21:56:10.0152211Z ff20aefbe592: Pull complete 2024-08-20T21:56:10.6729622Z 39f017df093b: Pull complete 2024-08-20T21:56:11.1261798Z dcb4b1c5a6f6: Pull complete 2024-08-20T21:56:11.5365975Z 862b4f5c5cd1: Pull complete 2024-08-20T21:56:11.9878984Z 46546d0a53bb: Pull complete 2024-08-20T21:56:12.4294931Z 82a31e7860af: Pull complete 2024-08-20T21:56:14.2401037Z ac2f6809f82b: Pull complete 2024-08-20T21:56:14.7063445Z 2b45e4461309: Pull complete 2024-08-20T21:56:17.4764124Z 403462e9ee27: Pull complete 2024-08-20T21:56:17.9360316Z 889f47863872: Pull complete 2024-08-20T21:56:33.1334603Z 23898c5065b1: Pull complete 2024-08-20T21:56:33.4131337Z f1caf060fc92: Pull complete 2024-08-20T21:56:33.8572017Z 621b6cedbe16: Pull complete 2024-08-20T21:56:34.8809658Z b9c038a9eb4a: Pull complete 2024-08-20T21:56:35.3383219Z 74046cd1f067: Pull complete 2024-08-20T21:56:37.4035934Z Digest: sha256:0b3aa0836d4f18656c98dcf355b46cbae2123243424ebbed0929206c3d798126 2024-08-20T21:56:37.4973162Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:56:37.5469177Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:56:37.5556984Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T21:56:37.5558193Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T21:56:37.5567424Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:56:37.5568060Z env: 2024-08-20T21:56:37.5568386Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:56:37.5568816Z ##[endgroup] 2024-08-20T21:56:37.5831968Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-08-20T21:56:37.5832905Z with: 2024-08-20T21:56:37.5833250Z driver-version: 550.54.15 2024-08-20T21:56:37.5833668Z env: 2024-08-20T21:56:37.5833977Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:56:37.5834400Z ##[endgroup] 2024-08-20T21:56:37.6079933Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T21:56:37.6080631Z with: 2024-08-20T21:56:37.6080962Z timeout_minutes: 10 2024-08-20T21:56:37.6081341Z max_attempts: 3 2024-08-20T21:56:37.6123892Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-08-20T21:56:37.6166645Z retry_wait_seconds: 10 2024-08-20T21:56:37.6167097Z polling_interval_seconds: 1 2024-08-20T21:56:37.6167549Z warning_on_retry: true 2024-08-20T21:56:37.6167980Z continue_on_error: false 2024-08-20T21:56:37.6168390Z env: 2024-08-20T21:56:37.6168701Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:56:37.6169132Z DRIVER_VERSION: 550.54.15 2024-08-20T21:56:37.6169557Z ##[endgroup] 2024-08-20T21:56:37.7147712Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-08-20T21:56:37.7149189Z + pre_install_nvidia_driver_amzn2 2024-08-20T21:56:37.7149854Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-08-20T21:56:38.2011892Z No match for argument: nvidia-driver-latest-dkms 2024-08-20T21:56:38.2014597Z No packages marked for removal. 2024-08-20T21:56:38.2089455Z Dependencies resolved. 2024-08-20T21:56:38.2103355Z Nothing to do. 2024-08-20T21:56:38.2105563Z Complete! 2024-08-20T21:56:38.3322642Z + install_nvidia_driver_common 2024-08-20T21:56:38.3327483Z + echo 'Before installing NVIDIA driver' 2024-08-20T21:56:38.3328501Z Before installing NVIDIA driver 2024-08-20T21:56:38.3329206Z + lspci 2024-08-20T21:56:38.3523107Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-08-20T21:56:38.3524103Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T21:56:38.3525088Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-08-20T21:56:38.3526003Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-08-20T21:56:38.3526804Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-08-20T21:56:38.3529305Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T21:56:38.3530366Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-20T21:56:38.3531415Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-20T21:56:38.3532704Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-08-20T21:56:38.3533465Z + lsmod 2024-08-20T21:56:38.3566797Z Module Size Used by 2024-08-20T21:56:38.3567290Z ib_core 454656 0 2024-08-20T21:56:38.3567780Z veth 36864 0 2024-08-20T21:56:38.3568228Z nvidia_modeset 1351680 0 2024-08-20T21:56:38.3568702Z video 65536 1 nvidia_modeset 2024-08-20T21:56:38.3569248Z wmi 36864 1 video 2024-08-20T21:56:38.3569744Z nvidia_uvm 4706304 0 2024-08-20T21:56:38.3570276Z nvidia 54071296 11 nvidia_uvm,nvidia_modeset 2024-08-20T21:56:38.3570911Z drm 602112 1 nvidia 2024-08-20T21:56:38.3571450Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T21:56:38.3572089Z backlight 24576 3 video,drm,nvidia_modeset 2024-08-20T21:56:38.3572697Z i2c_core 106496 2 nvidia,drm 2024-08-20T21:56:38.3573218Z xt_conntrack 16384 1 2024-08-20T21:56:38.3573654Z nft_chain_nat 16384 3 2024-08-20T21:56:38.3574142Z xt_MASQUERADE 20480 1 2024-08-20T21:56:38.3574719Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T21:56:38.3575303Z nf_conntrack_netlink 57344 0 2024-08-20T21:56:38.3575996Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T21:56:38.3576788Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T21:56:38.3577347Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T21:56:38.3577926Z xfrm_user 57344 1 2024-08-20T21:56:38.3578419Z xfrm_algo 16384 1 xfrm_user 2024-08-20T21:56:38.3578915Z xt_addrtype 16384 2 2024-08-20T21:56:38.3579349Z nft_compat 20480 4 2024-08-20T21:56:38.3579907Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T21:56:38.3580661Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T21:56:38.3581316Z br_netfilter 36864 0 2024-08-20T21:56:38.3581827Z bridge 307200 1 br_netfilter 2024-08-20T21:56:38.3582369Z stp 16384 1 bridge 2024-08-20T21:56:38.3582872Z llc 16384 2 bridge,stp 2024-08-20T21:56:38.3583384Z overlay 167936 0 2024-08-20T21:56:38.3583830Z tls 114688 0 2024-08-20T21:56:38.3584256Z sunrpc 692224 1 2024-08-20T21:56:38.3584706Z nls_ascii 16384 1 2024-08-20T21:56:38.3585161Z nls_cp437 20480 1 2024-08-20T21:56:38.3585575Z vfat 24576 1 2024-08-20T21:56:38.3586049Z fat 86016 1 vfat 2024-08-20T21:56:38.3586526Z ata_piix 45056 0 2024-08-20T21:56:38.3586960Z libata 401408 1 ata_piix 2024-08-20T21:56:38.3587476Z ghash_clmulni_intel 16384 0 2024-08-20T21:56:38.3587940Z ena 167936 0 2024-08-20T21:56:38.3588382Z scsi_mod 290816 1 libata 2024-08-20T21:56:38.3588862Z ptp 36864 1 ena 2024-08-20T21:56:38.3589340Z pps_core 24576 1 ptp 2024-08-20T21:56:38.3589863Z scsi_common 16384 2 scsi_mod,libata 2024-08-20T21:56:38.3590416Z aesni_intel 393216 0 2024-08-20T21:56:38.3590875Z i8042 45056 0 2024-08-20T21:56:38.3591300Z serio 28672 3 i8042 2024-08-20T21:56:38.3591808Z crypto_simd 16384 1 aesni_intel 2024-08-20T21:56:38.3592460Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T21:56:38.3593422Z button 24576 0 2024-08-20T21:56:38.3593873Z sch_fq_codel 20480 9 2024-08-20T21:56:38.3594327Z dm_mod 188416 0 2024-08-20T21:56:38.3594750Z fuse 163840 1 2024-08-20T21:56:38.3595197Z configfs 57344 1 2024-08-20T21:56:38.3595656Z dax 45056 1 dm_mod 2024-08-20T21:56:38.3596119Z loop 36864 0 2024-08-20T21:56:38.3596564Z dmi_sysfs 20480 0 2024-08-20T21:56:38.3597016Z crc32_pclmul 16384 0 2024-08-20T21:56:38.3597618Z crc32c_intel 24576 0 2024-08-20T21:56:38.3598178Z + modinfo nvidia 2024-08-20T21:56:38.3599130Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T21:56:38.3600007Z alias: char-major-195-* 2024-08-20T21:56:38.3600459Z version: 550.54.15 2024-08-20T21:56:38.3600878Z supported: external 2024-08-20T21:56:38.3601287Z license: NVIDIA 2024-08-20T21:56:38.3601731Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T21:56:38.3602324Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T21:56:38.3602884Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T21:56:38.3603439Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T21:56:38.3604060Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T21:56:38.3604672Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T21:56:38.3605273Z depends: i2c-core,drm 2024-08-20T21:56:38.3605713Z retpoline: Y 2024-08-20T21:56:38.3606091Z name: nvidia 2024-08-20T21:56:38.3606847Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T21:56:38.3607723Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T21:56:38.3608526Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T21:56:38.3609263Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T21:56:38.3609788Z parm: NVreg_RmLogonRC:int 2024-08-20T21:56:38.3610316Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T21:56:38.3610873Z parm: NVreg_DeviceFileUID:int 2024-08-20T21:56:38.3611386Z parm: NVreg_DeviceFileGID:int 2024-08-20T21:56:38.3611922Z parm: NVreg_DeviceFileMode:int 2024-08-20T21:56:38.3612576Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T21:56:38.3613254Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T21:56:38.3613844Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T21:56:38.3614383Z parm: NVreg_EnableMSI:int 2024-08-20T21:56:38.3614879Z parm: NVreg_TCEBypassMode:int 2024-08-20T21:56:38.3615452Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T21:56:38.3616108Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T21:56:38.3616812Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T21:56:38.3617511Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T21:56:38.3618269Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T21:56:38.3618998Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T21:56:38.3619763Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T21:56:38.3620497Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T21:56:38.3621090Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T21:56:38.3621755Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T21:56:38.3622430Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T21:56:38.3623034Z parm: NVreg_MemoryPoolSize:int 2024-08-20T21:56:38.3623613Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T21:56:38.3624206Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T21:56:38.3624772Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T21:56:38.3625326Z parm: NVreg_NvLinkDisable:int 2024-08-20T21:56:38.3625948Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T21:56:38.3626577Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T21:56:38.3627164Z parm: NVreg_EnableResizableBar:int 2024-08-20T21:56:38.3627758Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T21:56:38.3628372Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T21:56:38.3628956Z parm: NVreg_RegistryDwords:charp 2024-08-20T21:56:38.3629574Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T21:56:38.3630164Z parm: NVreg_RmMsg:charp 2024-08-20T21:56:38.3630651Z parm: NVreg_GpuBlacklist:charp 2024-08-20T21:56:38.3631319Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T21:56:38.3631994Z parm: NVreg_ExcludedGpus:charp 2024-08-20T21:56:38.3632539Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T21:56:38.3633131Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T21:56:38.3633719Z parm: NVreg_ImexChannelCount:int 2024-08-20T21:56:38.3634255Z parm: rm_firmware_active:charp 2024-08-20T21:56:38.3634767Z + HAS_NVIDIA_DRIVER=0 2024-08-20T21:56:38.3635241Z ++ command -v nvidia-smi 2024-08-20T21:56:38.3635723Z + '[' -x /usr/bin/nvidia-smi ']' 2024-08-20T21:56:38.3636169Z + set +e 2024-08-20T21:56:38.3636805Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-08-20T21:56:38.3863365Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-08-20T21:56:38.3863893Z + NVIDIA_SMI_STATUS=0 2024-08-20T21:56:38.3864445Z + '[' 0 -ne 0 ']' 2024-08-20T21:56:38.3864901Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-08-20T21:56:38.3865336Z + HAS_NVIDIA_DRIVER=1 2024-08-20T21:56:38.3866259Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-08-20T21:56:38.3867179Z + set -e 2024-08-20T21:56:38.3867567Z + '[' 1 -eq 0 ']' 2024-08-20T21:56:38.3868307Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-08-20T21:56:38.3869190Z + post_install_nvidia_driver_common 2024-08-20T21:56:38.3872433Z + sudo modprobe nvidia 2024-08-20T21:56:38.5603416Z + echo 'After installing NVIDIA driver' 2024-08-20T21:56:38.5603952Z + lspci 2024-08-20T21:56:38.5604310Z After installing NVIDIA driver 2024-08-20T21:56:38.5813652Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-08-20T21:56:38.5814677Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T21:56:38.5815648Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-08-20T21:56:38.5816586Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-08-20T21:56:38.5817662Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-08-20T21:56:38.5818808Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T21:56:38.5819840Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-20T21:56:38.5820898Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-20T21:56:38.5821911Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-08-20T21:56:38.5822650Z + lsmod 2024-08-20T21:56:38.5845141Z Module Size Used by 2024-08-20T21:56:38.5845667Z ib_core 454656 0 2024-08-20T21:56:38.5846103Z veth 36864 0 2024-08-20T21:56:38.5846561Z nvidia_modeset 1351680 0 2024-08-20T21:56:38.5847077Z video 65536 1 nvidia_modeset 2024-08-20T21:56:38.5847598Z wmi 36864 1 video 2024-08-20T21:56:38.5848080Z nvidia_uvm 4706304 0 2024-08-20T21:56:38.5848647Z nvidia 54071296 11 nvidia_uvm,nvidia_modeset 2024-08-20T21:56:38.5849265Z drm 602112 1 nvidia 2024-08-20T21:56:38.5849806Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T21:56:38.5850454Z backlight 24576 3 video,drm,nvidia_modeset 2024-08-20T21:56:38.5851064Z i2c_core 106496 2 nvidia,drm 2024-08-20T21:56:38.5851585Z xt_conntrack 16384 1 2024-08-20T21:56:38.5852048Z nft_chain_nat 16384 3 2024-08-20T21:56:38.5852494Z xt_MASQUERADE 20480 1 2024-08-20T21:56:38.5853063Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T21:56:38.5853664Z nf_conntrack_netlink 57344 0 2024-08-20T21:56:38.5854364Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T21:56:38.5855163Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T21:56:38.5855738Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T21:56:38.5856254Z xfrm_user 57344 1 2024-08-20T21:56:38.5856972Z xfrm_algo 16384 1 xfrm_user 2024-08-20T21:56:38.5857651Z xt_addrtype 16384 2 2024-08-20T21:56:38.5858094Z nft_compat 20480 4 2024-08-20T21:56:38.5858651Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T21:56:38.5859411Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T21:56:38.5860085Z br_netfilter 36864 0 2024-08-20T21:56:38.5860572Z bridge 307200 1 br_netfilter 2024-08-20T21:56:38.5861107Z stp 16384 1 bridge 2024-08-20T21:56:38.5861622Z llc 16384 2 bridge,stp 2024-08-20T21:56:38.5862119Z overlay 167936 0 2024-08-20T21:56:38.5863035Z tls 114688 0 2024-08-20T21:56:38.5863581Z sunrpc 692224 1 2024-08-20T21:56:38.5864022Z nls_ascii 16384 1 2024-08-20T21:56:38.5864476Z nls_cp437 20480 1 2024-08-20T21:56:38.5864917Z vfat 24576 1 2024-08-20T21:56:38.5865359Z fat 86016 1 vfat 2024-08-20T21:56:38.5865848Z ata_piix 45056 0 2024-08-20T21:56:38.5866317Z libata 401408 1 ata_piix 2024-08-20T21:56:38.5866810Z ghash_clmulni_intel 16384 0 2024-08-20T21:56:38.5867272Z ena 167936 0 2024-08-20T21:56:38.5867731Z scsi_mod 290816 1 libata 2024-08-20T21:56:38.5868208Z ptp 36864 1 ena 2024-08-20T21:56:38.5868688Z pps_core 24576 1 ptp 2024-08-20T21:56:38.5869222Z scsi_common 16384 2 scsi_mod,libata 2024-08-20T21:56:38.5869747Z aesni_intel 393216 0 2024-08-20T21:56:38.5870194Z i8042 45056 0 2024-08-20T21:56:38.5870643Z serio 28672 3 i8042 2024-08-20T21:56:38.5871144Z crypto_simd 16384 1 aesni_intel 2024-08-20T21:56:38.5871793Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T21:56:38.5872404Z button 24576 0 2024-08-20T21:56:38.5872848Z sch_fq_codel 20480 9 2024-08-20T21:56:38.5873303Z dm_mod 188416 0 2024-08-20T21:56:38.5873751Z fuse 163840 1 2024-08-20T21:56:38.5874181Z configfs 57344 1 2024-08-20T21:56:38.5874642Z dax 45056 1 dm_mod 2024-08-20T21:56:38.5875123Z loop 36864 0 2024-08-20T21:56:38.5875553Z dmi_sysfs 20480 0 2024-08-20T21:56:38.5876006Z crc32_pclmul 16384 0 2024-08-20T21:56:38.5876456Z crc32c_intel 24576 0 2024-08-20T21:56:38.5876877Z + modinfo nvidia 2024-08-20T21:56:38.5877763Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T21:56:38.5878635Z alias: char-major-195-* 2024-08-20T21:56:38.5879085Z version: 550.54.15 2024-08-20T21:56:38.5879505Z supported: external 2024-08-20T21:56:38.5879920Z license: NVIDIA 2024-08-20T21:56:38.5880362Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T21:56:38.5881361Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T21:56:38.5882077Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T21:56:38.5882639Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T21:56:38.5883258Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T21:56:38.5883864Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T21:56:38.5884475Z depends: i2c-core,drm 2024-08-20T21:56:38.5884918Z retpoline: Y 2024-08-20T21:56:38.5885287Z name: nvidia 2024-08-20T21:56:38.5886053Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T21:56:38.5886929Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T21:56:38.5887734Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T21:56:38.5888475Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T21:56:38.5889007Z parm: NVreg_RmLogonRC:int 2024-08-20T21:56:38.5889671Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T21:56:38.5890259Z parm: NVreg_DeviceFileUID:int 2024-08-20T21:56:38.5890877Z parm: NVreg_DeviceFileGID:int 2024-08-20T21:56:38.5891421Z parm: NVreg_DeviceFileMode:int 2024-08-20T21:56:38.5892076Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T21:56:38.5893276Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T21:56:38.5893875Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T21:56:38.5894404Z parm: NVreg_EnableMSI:int 2024-08-20T21:56:38.5894899Z parm: NVreg_TCEBypassMode:int 2024-08-20T21:56:38.5895464Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T21:56:38.5896120Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T21:56:38.5896831Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T21:56:38.5897531Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T21:56:38.5898294Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T21:56:38.5899028Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T21:56:38.5899799Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T21:56:38.5901053Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T21:56:38.5901697Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T21:56:38.5902360Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T21:56:38.5903037Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T21:56:38.5903633Z parm: NVreg_MemoryPoolSize:int 2024-08-20T21:56:38.5904203Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T21:56:38.5904796Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T21:56:38.5905363Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T21:56:38.5905913Z parm: NVreg_NvLinkDisable:int 2024-08-20T21:56:38.5906535Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T21:56:38.5907169Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T21:56:38.5907772Z parm: NVreg_EnableResizableBar:int 2024-08-20T21:56:38.5908375Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T21:56:38.5908984Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T21:56:38.5909587Z parm: NVreg_RegistryDwords:charp 2024-08-20T21:56:38.5910207Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T21:56:38.5910806Z parm: NVreg_RmMsg:charp 2024-08-20T21:56:38.5911291Z parm: NVreg_GpuBlacklist:charp 2024-08-20T21:56:38.5911866Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T21:56:38.5912442Z parm: NVreg_ExcludedGpus:charp 2024-08-20T21:56:38.5912986Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T21:56:38.5913577Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T21:56:38.5914164Z parm: NVreg_ImexChannelCount:int 2024-08-20T21:56:38.5914703Z parm: rm_firmware_active:charp 2024-08-20T21:56:38.5915194Z + set +e 2024-08-20T21:56:38.5915596Z + nvidia-smi 2024-08-20T21:56:38.6132320Z Tue Aug 20 21:56:38 2024 2024-08-20T21:56:38.6133149Z +-----------------------------------------------------------------------------------------+ 2024-08-20T21:56:38.6134204Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T21:56:38.6135827Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T21:56:38.6136858Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T21:56:38.6137999Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T21:56:38.6138869Z | | | MIG M. | 2024-08-20T21:56:38.6139536Z |=========================================+========================+======================| 2024-08-20T21:56:38.6217496Z | 0 Tesla M60 On | 00000000:00:1D.0 Off | 0 | 2024-08-20T21:56:38.6218643Z | N/A 29C P8 14W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-20T21:56:38.6219580Z | | | N/A | 2024-08-20T21:56:38.6220470Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T21:56:38.6281966Z | 1 Tesla M60 On | 00000000:00:1E.0 Off | 0 | 2024-08-20T21:56:38.6282879Z | N/A 30C P8 15W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-20T21:56:38.6283656Z | | | N/A | 2024-08-20T21:56:38.6284537Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T21:56:38.6285263Z 2024-08-20T21:56:38.6286096Z +-----------------------------------------------------------------------------------------+ 2024-08-20T21:56:38.6286890Z | Processes: | 2024-08-20T21:56:38.6287786Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T21:56:38.6289001Z | ID ID Usage | 2024-08-20T21:56:38.6289924Z |=========================================================================================| 2024-08-20T21:56:38.6290751Z | No running processes found | 2024-08-20T21:56:38.6291741Z +-----------------------------------------------------------------------------------------+ 2024-08-20T21:56:38.7375942Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-08-20T21:56:38.7556351Z Tesla M60 2024-08-20T21:56:38.7622016Z + NVIDIA_SMI_STATUS=0 2024-08-20T21:56:38.7622557Z + '[' 0 -eq 0 ']' 2024-08-20T21:56:38.7623072Z + echo 'INFO: Ignoring allowed status 0' 2024-08-20T21:56:38.7623577Z + set -e 2024-08-20T21:56:38.7623953Z INFO: Ignoring allowed status 0 2024-08-20T21:56:38.7629828Z == Installing nvidia container toolkit for amzn2023 == 2024-08-20T21:56:38.7633785Z + sudo yum install -y yum-utils 2024-08-20T21:56:39.3267749Z Last metadata expiration check: 2:48:19 ago on Tue Aug 20 19:08:20 2024. 2024-08-20T21:56:39.3560997Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-08-20T21:56:39.3981508Z Dependencies resolved. 2024-08-20T21:56:39.4165393Z Nothing to do. 2024-08-20T21:56:39.4166232Z Complete! 2024-08-20T21:56:39.4833484Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-08-20T21:56:39.4835043Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T21:56:39.4836740Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T21:56:39.8319711Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T21:56:39.9132880Z + sudo yum install -y nvidia-docker2 2024-08-20T21:56:40.5388611Z nvidia-container-toolkit 8.5 kB/s | 833 B 00:00 2024-08-20T21:56:40.5685435Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-08-20T21:56:40.6096968Z Dependencies resolved. 2024-08-20T21:56:40.6282232Z Nothing to do. 2024-08-20T21:56:40.6282966Z Complete! 2024-08-20T21:56:40.7825431Z + sudo systemctl restart docker 2024-08-20T21:57:19.8570626Z nvidia-persistenced failed to initialize. Check syslog for more details. 2024-08-20T21:57:19.8851579Z Tue Aug 20 21:57:19 2024 2024-08-20T21:57:19.8853328Z +-----------------------------------------------------------------------------------------+ 2024-08-20T21:57:19.8855503Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T21:57:19.8859312Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T21:57:19.8861391Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T21:57:19.8863938Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T21:57:19.8865622Z | | | MIG M. | 2024-08-20T21:57:19.8866928Z |=========================================+========================+======================| 2024-08-20T21:57:19.8945665Z | 0 Tesla M60 On | 00000000:00:1D.0 Off | 0 | 2024-08-20T21:57:19.8947276Z | N/A 29C P8 14W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-20T21:57:19.8948752Z | | | N/A | 2024-08-20T21:57:19.8951261Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T21:57:19.9012547Z | 1 Tesla M60 On | 00000000:00:1E.0 Off | 0 | 2024-08-20T21:57:19.9014350Z | N/A 30C P8 13W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-20T21:57:19.9015907Z | | | N/A | 2024-08-20T21:57:19.9017571Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T21:57:19.9018879Z 2024-08-20T21:57:19.9020406Z +-----------------------------------------------------------------------------------------+ 2024-08-20T21:57:19.9021851Z | Processes: | 2024-08-20T21:57:19.9023334Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T21:57:19.9024788Z | ID ID Usage | 2024-08-20T21:57:19.9026028Z |=========================================================================================| 2024-08-20T21:57:19.9027468Z | No running processes found | 2024-08-20T21:57:19.9029145Z +-----------------------------------------------------------------------------------------+ 2024-08-20T21:57:20.7399209Z Command completed after 1 attempt(s). 2024-08-20T21:57:20.7521275Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T21:57:20.7522259Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T21:57:20.7523099Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-20T21:57:20.7523909Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-20T21:57:20.7535182Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:57:20.7535786Z env: 2024-08-20T21:57:20.7536096Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:57:20.7536625Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:57:20.7537216Z ##[endgroup] 2024-08-20T21:57:21.0831935Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T21:57:21.1049130Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.9/site-packages (5.9.1) 2024-08-20T21:57:21.1059014Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.9/site-packages (11.525.84) 2024-08-20T21:57:21.2929354Z Prepare all required actions 2024-08-20T21:57:21.2930065Z Getting action download info 2024-08-20T21:57:21.4086832Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-20T21:57:21.6088061Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-20T21:57:21.7466475Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-20T21:57:21.7467090Z with: 2024-08-20T21:57:21.7467462Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-08-20T21:57:21.7468234Z s3-bucket: gha-artifacts 2024-08-20T21:57:21.7468648Z env: 2024-08-20T21:57:21.7468958Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:57:21.7469516Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:57:21.7470106Z ##[endgroup] 2024-08-20T21:57:21.7516424Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T21:57:21.7516964Z with: 2024-08-20T21:57:21.7517407Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-08-20T21:57:21.7517926Z s3-bucket: gha-artifacts 2024-08-20T21:57:21.7518354Z region: us-east-1 2024-08-20T21:57:21.7518717Z env: 2024-08-20T21:57:21.7519023Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:57:21.7519579Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:57:21.7520169Z ##[endgroup] 2024-08-20T21:57:22.3667072Z (node:379190) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T21:57:22.3667988Z 2024-08-20T21:57:22.3668315Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T21:57:22.3669257Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T21:57:22.3670375Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T21:57:22.4607325Z Found 1 objects with prefix pytorch/pytorch/10479309237/linux-focal-cuda11.8-py3.10-gcc9/ 2024-08-20T21:57:22.4608703Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T21:57:46.7999764Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T21:57:46.8010938Z Artifact download has finished successfully 2024-08-20T21:57:46.8249743Z ##[group]Run unzip -o artifacts.zip 2024-08-20T21:57:46.8250342Z unzip -o artifacts.zip 2024-08-20T21:57:46.8258425Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:57:46.8259054Z env: 2024-08-20T21:57:46.8259481Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:57:46.8260027Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:57:46.8260646Z ##[endgroup] 2024-08-20T21:57:46.8311517Z Archive: artifacts.zip 2024-08-20T21:57:46.8312829Z creating: dist/ 2024-08-20T21:57:49.2743249Z inflating: dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T21:57:49.2744101Z creating: build/custom_test_artifacts/ 2024-08-20T21:57:49.2744857Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-20T21:57:49.2747435Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-20T21:57:49.2748589Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-20T21:57:49.2753274Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T21:57:49.2754919Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-20T21:57:49.2756178Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T21:57:49.2757492Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T21:57:49.2758728Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T21:57:49.2760162Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T21:57:49.2761946Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T21:57:49.2763311Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T21:57:49.2764592Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T21:57:49.2766090Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T21:57:49.2767599Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T21:57:49.2769483Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T21:57:49.2771214Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T21:57:49.2772728Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T21:57:49.2774251Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T21:57:49.2775604Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T21:57:49.2776885Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T21:57:49.2831426Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T21:57:49.2833277Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T21:57:49.2835161Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T21:57:49.2837068Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T21:57:49.2838963Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T21:57:49.2840770Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T21:57:49.2842557Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T21:57:49.2844376Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T21:57:49.2846180Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T21:57:49.2886602Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T21:57:49.2934692Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T21:57:49.2936499Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T21:57:49.2938179Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T21:57:49.2939785Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T21:57:49.2941374Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T21:57:49.2942950Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T21:57:49.2944512Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T21:57:49.2946095Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T21:57:49.3017064Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T21:57:49.3090388Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T21:57:49.3091941Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T21:57:49.3093481Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-20T21:57:49.3094580Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-20T21:57:49.3095753Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-20T21:57:49.3097173Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-20T21:57:49.3098498Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-20T21:57:49.3099969Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-20T21:57:49.3101425Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-20T21:57:49.3102776Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-20T21:57:49.3104162Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-20T21:57:49.3105540Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-20T21:57:49.3106942Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-20T21:57:49.3108343Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-20T21:57:49.3109709Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-20T21:57:49.3126860Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-20T21:57:49.3299224Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-20T21:57:49.3300526Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-20T21:57:49.3301923Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-20T21:57:49.3303495Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-20T21:57:49.3305011Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-20T21:57:49.3306424Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-20T21:57:49.3307876Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-20T21:57:49.3309357Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-20T21:57:49.3310830Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-20T21:57:49.3312277Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-20T21:57:49.3313720Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-20T21:57:49.3332706Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-20T21:57:49.3433121Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-20T21:57:49.3434658Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T21:57:49.3436059Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-20T21:57:49.3437514Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-20T21:57:49.3438754Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-20T21:57:49.3439894Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-20T21:57:49.3441064Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-08-20T21:57:49.3442487Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-20T21:57:49.3443775Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-20T21:57:49.3444989Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-20T21:57:49.3589202Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-20T21:57:49.3665363Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-20T21:57:49.3666316Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-20T21:57:49.3667196Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-20T21:57:49.3668245Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-20T21:57:49.3675461Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T21:57:49.3676634Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-20T21:57:49.3677827Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T21:57:49.3679090Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T21:57:49.3680338Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T21:57:49.3681723Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T21:57:49.3683145Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T21:57:49.3684420Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T21:57:49.3685673Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T21:57:49.3687151Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T21:57:49.3688632Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T21:57:49.3690411Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T21:57:49.3691868Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T21:57:49.3694015Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T21:57:49.3695674Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T21:57:49.3697007Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T21:57:49.3698271Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T21:57:49.3752720Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T21:57:49.3754554Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T21:57:49.3756403Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T21:57:49.3758297Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T21:57:49.3760168Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T21:57:49.3762160Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T21:57:49.3764007Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T21:57:49.3765789Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T21:57:49.3767577Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T21:57:49.3808659Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T21:57:49.3856017Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T21:57:49.3857799Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T21:57:49.3859465Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T21:57:49.3861052Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T21:57:49.3862624Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T21:57:49.3864182Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T21:57:49.3865723Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T21:57:49.3867290Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T21:57:49.3938715Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T21:57:49.4012681Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T21:57:49.4014216Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T21:57:49.4015502Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-20T21:57:49.4016607Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-20T21:57:49.4017746Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-20T21:57:49.4018972Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-20T21:57:49.4020343Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-20T21:57:49.4021850Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-20T21:57:49.4023345Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-20T21:57:49.4024725Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-20T21:57:49.4026156Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-20T21:57:49.4027567Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-20T21:57:49.4029010Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-20T21:57:49.4030442Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-20T21:57:49.4031863Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-20T21:57:49.4049343Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-20T21:57:49.4126974Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-20T21:57:49.4128562Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T21:57:49.4129934Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-20T21:57:49.4131162Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-20T21:57:49.4132322Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-20T21:57:49.4133623Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-20T21:57:49.4134752Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-08-20T21:57:49.4135840Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-20T21:57:49.4137492Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-20T21:57:49.4138960Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-20T21:57:49.4198943Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-20T21:57:49.4199914Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-20T21:57:49.4200891Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-20T21:57:49.4202036Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-20T21:57:49.4209137Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T21:57:49.4210421Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-20T21:57:49.4211720Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T21:57:49.4213075Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T21:57:49.4214415Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T21:57:49.4215908Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T21:57:49.4217421Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T21:57:49.4218783Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T21:57:49.4220118Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T21:57:49.4221701Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T21:57:49.4223277Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T21:57:49.4224824Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T21:57:49.4226371Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T21:57:49.4227954Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T21:57:49.4229537Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T21:57:49.4230951Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T21:57:49.4232310Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T21:57:49.4285656Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T21:57:49.4287586Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T21:57:49.4289984Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T21:57:49.4292029Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T21:57:49.4294318Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T21:57:49.4296193Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T21:57:49.4298250Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T21:57:49.4300135Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T21:57:49.4302031Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T21:57:49.4341245Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T21:57:49.4388904Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T21:57:49.4390770Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T21:57:49.4392767Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T21:57:49.4394493Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T21:57:49.4396156Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T21:57:49.4397815Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T21:57:49.4399440Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T21:57:49.4401091Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T21:57:49.4471544Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T21:57:49.4546018Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T21:57:49.4547651Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T21:57:49.4549010Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-20T21:57:49.4550240Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-20T21:57:49.4551491Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-20T21:57:49.4552791Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-20T21:57:49.4554253Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-20T21:57:49.4555884Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-20T21:57:49.4557442Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-20T21:57:49.4558926Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-20T21:57:49.4560453Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-20T21:57:49.4562152Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-20T21:57:49.4563741Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-20T21:57:49.4565268Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-20T21:57:49.4566793Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-20T21:57:49.4568390Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-20T21:57:49.4714538Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-20T21:57:49.4716059Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-20T21:57:49.4717600Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-20T21:57:49.4719291Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-20T21:57:49.4720938Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-20T21:57:49.4722492Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-20T21:57:49.4724094Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-20T21:57:49.4725691Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-20T21:57:49.4727301Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-20T21:57:49.4728910Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-20T21:57:49.4730498Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-20T21:57:49.4747852Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-20T21:57:49.4814840Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-20T21:57:49.4816521Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T21:57:49.4818009Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-20T21:57:49.4819327Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-20T21:57:49.4820575Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-20T21:57:49.4821811Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-20T21:57:49.4823037Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-08-20T21:57:49.4824209Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-20T21:57:49.4825284Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-20T21:57:49.4826393Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-20T21:57:49.4950769Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-20T21:57:49.5002985Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-20T21:57:49.5003790Z creating: build/lib/ 2024-08-20T21:57:49.5014871Z inflating: build/lib/libpthreadpool.a 2024-08-20T21:57:49.5015843Z inflating: build/lib/libclog.a 2024-08-20T21:57:49.5019013Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-20T21:57:49.5028927Z inflating: build/lib/libcpuinfo.a 2024-08-20T21:57:49.5038814Z inflating: build/lib/libcpuinfo_internals.a 2024-08-20T21:57:49.5145696Z inflating: build/lib/libprotobuf-lite.a 2024-08-20T21:57:49.5694429Z inflating: build/lib/libprotobuf.a 2024-08-20T21:57:49.5773297Z inflating: build/lib/libgtest.a 2024-08-20T21:57:49.5864128Z inflating: build/lib/libbenchmark.a 2024-08-20T21:57:49.5873295Z inflating: build/lib/libittnotify.a 2024-08-20T21:57:49.5950935Z inflating: build/lib/libasmjit.a 2024-08-20T21:57:49.5984575Z inflating: build/lib/libtensorpipe_uv.a 2024-08-20T21:57:49.6137995Z inflating: build/lib/libgloo.a 2024-08-20T21:57:49.6163088Z inflating: build/lib/libfmt.a 2024-08-20T21:57:49.6165304Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-20T21:57:49.6280231Z inflating: build/lib/libc10.so 2024-08-20T21:57:49.6282180Z inflating: build/lib/libtorch_global_deps.so 2024-08-20T21:57:49.6888007Z inflating: build/lib/libprotoc.a 2024-08-20T21:57:49.6911713Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-20T21:57:49.6933601Z inflating: build/lib/libgmock.a 2024-08-20T21:57:49.6934437Z inflating: build/lib/libgtest_main.a 2024-08-20T21:57:49.6935567Z inflating: build/lib/libbenchmark_main.a 2024-08-20T21:57:50.9057954Z inflating: build/lib/libdnnl.a 2024-08-20T21:57:50.9079563Z inflating: build/lib/libnnpack.a 2024-08-20T21:57:50.9766483Z inflating: build/lib/libtensorpipe.a 2024-08-20T21:57:50.9836280Z inflating: build/lib/libc10_cuda.so 2024-08-20T21:57:51.1333611Z inflating: build/lib/libfbgemm.a 2024-08-20T21:57:51.1334443Z inflating: build/lib/libgmock_main.a 2024-08-20T21:57:51.1945412Z inflating: build/lib/libkineto.a 2024-08-20T21:57:51.2251956Z inflating: build/lib/libtensorpipe_cuda.a 2024-08-20T21:57:51.2475080Z inflating: build/lib/libXNNPACK.a 2024-08-20T21:57:51.2935887Z inflating: build/lib/libgloo_cuda.a 2024-08-20T21:57:51.2986583Z inflating: build/lib/libonnx_proto.a 2024-08-20T21:57:51.3818019Z inflating: build/lib/libonnx.a 2024-08-20T21:57:54.3266139Z inflating: build/lib/libtorch_cpu.so 2024-08-20T21:57:54.3270765Z inflating: build/lib/libshm.so 2024-08-20T21:57:54.3276783Z inflating: build/lib/libunbox_lib.a 2024-08-20T21:57:56.7567706Z inflating: build/lib/libtorch_cuda.so 2024-08-20T21:57:56.7568729Z inflating: build/lib/libtorch.so 2024-08-20T21:57:56.7572167Z inflating: build/lib/libc10d_cuda_test.so 2024-08-20T21:57:57.8804820Z inflating: build/lib/libtorch_cuda_linalg.so 2024-08-20T21:57:57.8829937Z inflating: build/lib/libjitbackend_test.so 2024-08-20T21:57:57.8916939Z inflating: build/lib/libtorchbind_test.so 2024-08-20T21:57:57.8948272Z inflating: build/lib/libaoti_custom_ops.so 2024-08-20T21:57:57.8979293Z inflating: build/lib/libbackend_with_compiler.so 2024-08-20T21:57:58.1392610Z inflating: build/lib/libtorch_python.so 2024-08-20T21:57:58.1435050Z inflating: build/lib/libnnapi_backend.so 2024-08-20T21:57:58.1435656Z creating: build/bin/ 2024-08-20T21:57:58.1496951Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-20T21:57:58.1559554Z inflating: build/bin/c10_DeviceGuard_test 2024-08-20T21:57:58.1621924Z inflating: build/bin/c10_Device_test 2024-08-20T21:57:58.1694041Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-20T21:57:58.1753689Z inflating: build/bin/c10_StreamGuard_test 2024-08-20T21:57:58.1818886Z inflating: build/bin/c10_Scalar_test 2024-08-20T21:57:58.1881363Z inflating: build/bin/c10_SymInt_test 2024-08-20T21:57:58.1948014Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-20T21:57:58.2015979Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-20T21:57:58.2084338Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-20T21:57:58.2170615Z inflating: build/bin/c10_cow_test 2024-08-20T21:57:58.2235364Z inflating: build/bin/c10_Bitset_test 2024-08-20T21:57:58.2295060Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-20T21:57:58.2355968Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-20T21:57:58.2418047Z inflating: build/bin/c10_Half_test 2024-08-20T21:57:58.2486295Z inflating: build/bin/c10_LeftRight_test 2024-08-20T21:57:58.2553645Z inflating: build/bin/c10_Metaprogramming_test 2024-08-20T21:57:58.2614805Z inflating: build/bin/c10_Synchronized_test 2024-08-20T21:57:58.2682173Z inflating: build/bin/c10_ThreadLocal_test 2024-08-20T21:57:58.2745094Z inflating: build/bin/c10_TypeIndex_test 2024-08-20T21:57:58.2807592Z inflating: build/bin/c10_TypeList_test 2024-08-20T21:57:58.2867238Z inflating: build/bin/c10_TypeTraits_test 2024-08-20T21:57:58.2930501Z inflating: build/bin/c10_accumulate_test 2024-08-20T21:57:58.2998365Z inflating: build/bin/c10_bfloat16_test 2024-08-20T21:57:58.3059592Z inflating: build/bin/c10_bit_cast_test 2024-08-20T21:57:58.3128494Z inflating: build/bin/c10_complex_math_test 2024-08-20T21:57:58.3192547Z inflating: build/bin/c10_exception_test 2024-08-20T21:57:58.3260541Z inflating: build/bin/c10_complex_test 2024-08-20T21:57:58.3322205Z inflating: build/bin/c10_flags_test 2024-08-20T21:57:58.3383183Z inflating: build/bin/c10_generic_math_test 2024-08-20T21:57:58.3581635Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-20T21:57:58.3643836Z inflating: build/bin/c10_irange_test 2024-08-20T21:57:58.3709174Z inflating: build/bin/c10_lazy_test 2024-08-20T21:57:58.3778675Z inflating: build/bin/c10_logging_test 2024-08-20T21:57:58.3869643Z inflating: build/bin/c10_optional_test 2024-08-20T21:57:58.3945807Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-20T21:57:58.4011634Z inflating: build/bin/c10_registry_test 2024-08-20T21:57:58.4193887Z inflating: build/bin/c10_small_vector_test 2024-08-20T21:57:58.4256689Z inflating: build/bin/c10_ssize_test 2024-08-20T21:57:58.4320116Z inflating: build/bin/c10_string_util_test 2024-08-20T21:57:58.4391488Z inflating: build/bin/c10_string_view_test 2024-08-20T21:57:58.4453033Z inflating: build/bin/c10_tempfile_test 2024-08-20T21:57:58.4521583Z inflating: build/bin/c10_typeid_test 2024-08-20T21:57:58.4580786Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-20T21:57:58.5124015Z inflating: build/bin/protoc-3.13.0.0 2024-08-20T21:57:58.5666693Z inflating: build/bin/protoc 2024-08-20T21:57:58.5731801Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-08-20T21:57:58.5796461Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-08-20T21:57:58.5860518Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-08-20T21:57:58.5924250Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-08-20T21:57:58.5984052Z inflating: build/bin/c10_cuda_CUDATest 2024-08-20T21:57:58.6049042Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-20T21:57:58.6113645Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-20T21:57:58.6177571Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-08-20T21:57:58.6611333Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-20T21:57:58.7025662Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-20T21:57:58.7423833Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-20T21:57:58.7488415Z inflating: build/bin/FileStoreTest 2024-08-20T21:57:58.7556445Z inflating: build/bin/TCPStoreTest 2024-08-20T21:57:58.7621607Z inflating: build/bin/HashStoreTest 2024-08-20T21:57:58.7686453Z inflating: build/bin/BackoffTest 2024-08-20T21:57:58.7703525Z inflating: build/bin/ProcessGroupMPITest 2024-08-20T21:57:58.7708773Z inflating: build/bin/torch_shm_manager 2024-08-20T21:57:58.7774189Z inflating: build/bin/test_edge_op_registration 2024-08-20T21:57:58.7777802Z inflating: build/bin/example_allreduce 2024-08-20T21:57:58.7841635Z inflating: build/bin/Dimname_test 2024-08-20T21:57:58.7931276Z inflating: build/bin/Dict_test 2024-08-20T21:57:58.8009674Z inflating: build/bin/MaybeOwned_test 2024-08-20T21:57:58.8079469Z inflating: build/bin/NamedTensor_test 2024-08-20T21:57:58.8151503Z inflating: build/bin/apply_utils_test 2024-08-20T21:57:58.8223636Z inflating: build/bin/atest 2024-08-20T21:57:58.8300500Z inflating: build/bin/basic 2024-08-20T21:57:58.8367444Z inflating: build/bin/broadcast_test 2024-08-20T21:57:58.8438453Z inflating: build/bin/cpu_generator_test 2024-08-20T21:57:58.8500610Z inflating: build/bin/cpu_allocator_test 2024-08-20T21:57:58.8565349Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-20T21:57:58.8627156Z inflating: build/bin/dlconvertor_test 2024-08-20T21:57:58.8687951Z inflating: build/bin/dispatch_key_set_test 2024-08-20T21:57:58.8801555Z inflating: build/bin/cpu_rng_test 2024-08-20T21:57:58.8872151Z inflating: build/bin/extension_backend_test 2024-08-20T21:57:58.8939640Z inflating: build/bin/half_test 2024-08-20T21:57:58.9056414Z inflating: build/bin/ivalue_test 2024-08-20T21:57:58.9117431Z inflating: build/bin/lazy_tensor_test 2024-08-20T21:57:58.9183594Z inflating: build/bin/math_kernel_test 2024-08-20T21:57:58.9249644Z inflating: build/bin/memory_format_test 2024-08-20T21:57:58.9314792Z inflating: build/bin/memory_overlapping_test 2024-08-20T21:57:58.9379432Z inflating: build/bin/mobile_memory_cleanup 2024-08-20T21:57:58.9447577Z inflating: build/bin/native_test 2024-08-20T21:57:58.9509615Z inflating: build/bin/operator_name_test 2024-08-20T21:57:58.9572676Z inflating: build/bin/packedtensoraccessor_test 2024-08-20T21:57:58.9636015Z inflating: build/bin/operators_test 2024-08-20T21:57:58.9718094Z inflating: build/bin/pow_test 2024-08-20T21:57:58.9780416Z inflating: build/bin/reportMemoryUsage_test 2024-08-20T21:57:58.9850126Z inflating: build/bin/quantized_test 2024-08-20T21:57:58.9911572Z inflating: build/bin/reduce_ops_test 2024-08-20T21:57:58.9980289Z inflating: build/bin/scalar_tensor_test 2024-08-20T21:57:59.0051887Z inflating: build/bin/scalar_test 2024-08-20T21:57:59.0115387Z inflating: build/bin/StorageUtils_test 2024-08-20T21:57:59.0179409Z inflating: build/bin/stride_properties_test 2024-08-20T21:57:59.0275341Z inflating: build/bin/tensor_iterator_test 2024-08-20T21:57:59.0343487Z inflating: build/bin/type_ptr_test 2024-08-20T21:57:59.0410413Z inflating: build/bin/test_parallel 2024-08-20T21:57:59.0414257Z inflating: build/bin/thread_init_test 2024-08-20T21:57:59.0477809Z inflating: build/bin/undefined_tensor_test 2024-08-20T21:57:59.0551980Z inflating: build/bin/type_test 2024-08-20T21:57:59.0553916Z inflating: build/bin/verify_api_visibility 2024-08-20T21:57:59.0617631Z inflating: build/bin/weakref_test 2024-08-20T21:57:59.0702417Z inflating: build/bin/legacy_vmap_test 2024-08-20T21:57:59.0765185Z inflating: build/bin/wrapdim_test 2024-08-20T21:57:59.0838600Z inflating: build/bin/IListRef_test 2024-08-20T21:57:59.0902180Z inflating: build/bin/xla_tensor_test 2024-08-20T21:57:59.1030832Z inflating: build/bin/List_test 2024-08-20T21:57:59.1178753Z inflating: build/bin/kernel_function_legacy_test 2024-08-20T21:57:59.1296671Z inflating: build/bin/kernel_function_test 2024-08-20T21:57:59.1377614Z inflating: build/bin/KernelFunction_test 2024-08-20T21:57:59.1533414Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-20T21:57:59.1659185Z inflating: build/bin/kernel_lambda_test 2024-08-20T21:57:59.1734039Z inflating: build/bin/kernel_stackbased_test 2024-08-20T21:57:59.1796789Z inflating: build/bin/CppSignature_test 2024-08-20T21:57:59.1914251Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-20T21:57:59.1973823Z inflating: build/bin/op_allowlist_test 2024-08-20T21:57:59.2042230Z inflating: build/bin/backend_fallback_test 2024-08-20T21:57:59.2120626Z inflating: build/bin/inline_container_test 2024-08-20T21:57:59.2484391Z inflating: build/bin/op_registration_test 2024-08-20T21:57:59.2550249Z inflating: build/bin/cuda_apply_test 2024-08-20T21:57:59.2614028Z inflating: build/bin/cuda_allocator_test 2024-08-20T21:57:59.2685614Z inflating: build/bin/cuda_atomic_ops_test 2024-08-20T21:57:59.2753028Z inflating: build/bin/cuda_caching_host_allocator_test 2024-08-20T21:57:59.2838881Z inflating: build/bin/cuda_complex_math_test 2024-08-20T21:57:59.2911118Z inflating: build/bin/cuda_complex_test 2024-08-20T21:57:59.2971690Z inflating: build/bin/cuda_device_test 2024-08-20T21:57:59.3042562Z inflating: build/bin/cuda_cub_test 2024-08-20T21:57:59.3105048Z inflating: build/bin/cuda_dlconvertor_test 2024-08-20T21:57:59.3167291Z inflating: build/bin/cuda_integer_divider_test 2024-08-20T21:57:59.3247549Z inflating: build/bin/cuda_distributions_test 2024-08-20T21:57:59.3317045Z inflating: build/bin/cuda_generator_test 2024-08-20T21:57:59.3377613Z inflating: build/bin/cuda_half_test 2024-08-20T21:57:59.3438204Z inflating: build/bin/cuda_optional_test 2024-08-20T21:57:59.3499905Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-08-20T21:57:59.3573616Z inflating: build/bin/cuda_stream_test 2024-08-20T21:57:59.3638111Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-08-20T21:57:59.3701869Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-08-20T21:57:59.3762140Z inflating: build/bin/cuda_cudnn_test 2024-08-20T21:57:59.3826179Z inflating: build/bin/cuda_vectorized_test 2024-08-20T21:57:59.3906530Z inflating: build/bin/ProcessGroupGlooTest 2024-08-20T21:57:59.3924208Z inflating: build/bin/tutorial_tensorexpr 2024-08-20T21:57:59.3994660Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-08-20T21:57:59.4072328Z inflating: build/bin/ProcessGroupNCCLTest 2024-08-20T21:57:59.4136747Z inflating: build/bin/ProcessGroupUCCTest 2024-08-20T21:57:59.4212912Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-08-20T21:57:59.4279532Z inflating: build/bin/test_dist_autograd 2024-08-20T21:57:59.4363937Z inflating: build/bin/test_cpp_rpc 2024-08-20T21:57:59.4367303Z inflating: build/bin/parallel_benchmark 2024-08-20T21:57:59.4378409Z inflating: build/bin/aot_model_compiler_test 2024-08-20T21:57:59.4460212Z inflating: build/bin/test_mobile_nnc 2024-08-20T21:57:59.4878707Z inflating: build/bin/test_lazy 2024-08-20T21:57:59.5900182Z inflating: build/bin/test_tensorexpr 2024-08-20T21:57:59.7312121Z inflating: build/bin/test_api 2024-08-20T21:57:59.8015200Z inflating: build/bin/test_jit 2024-08-20T21:57:59.8015748Z creating: .additional_ci_files/ 2024-08-20T21:57:59.8087589Z inflating: .additional_ci_files/test-times.json 2024-08-20T21:57:59.8375009Z inflating: .additional_ci_files/test-class-times.json 2024-08-20T21:57:59.8414777Z ##[group]Run rm artifacts.zip 2024-08-20T21:57:59.8415248Z rm artifacts.zip 2024-08-20T21:57:59.8422720Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:57:59.8423347Z env: 2024-08-20T21:57:59.8423664Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:57:59.8424225Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:57:59.8424818Z ##[endgroup] 2024-08-20T21:57:59.9072978Z ##[group]Run df -H 2024-08-20T21:57:59.9073373Z df -H 2024-08-20T21:57:59.9080205Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:57:59.9080809Z env: 2024-08-20T21:57:59.9081144Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:57:59.9081736Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:57:59.9082316Z ##[endgroup] 2024-08-20T21:57:59.9128699Z Filesystem Size Used Avail Use% Mounted on 2024-08-20T21:57:59.9129362Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-08-20T21:57:59.9131302Z tmpfs 129G 6.4M 129G 1% /dev/shm 2024-08-20T21:57:59.9131898Z tmpfs 52G 635k 52G 1% /run 2024-08-20T21:57:59.9132651Z /dev/xvda1 161G 47G 115G 30% / 2024-08-20T21:57:59.9133237Z tmpfs 129G 13k 129G 1% /tmp 2024-08-20T21:57:59.9133805Z /dev/xvda128 11M 1.4M 9.2M 13% /boot/efi 2024-08-20T21:57:59.9134468Z tmpfs 26G 0 26G 0% /run/user/0 2024-08-20T21:57:59.9186909Z Prepare all required actions 2024-08-20T21:57:59.9187485Z Getting action download info 2024-08-20T21:58:00.0294873Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-20T21:58:00.0295459Z with: 2024-08-20T21:58:00.0295747Z env: 2024-08-20T21:58:00.0296079Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:00.0296824Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:00.0297399Z ##[endgroup] 2024-08-20T21:58:00.0342060Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T21:58:00.0342615Z with: 2024-08-20T21:58:00.0342916Z name: td_results 2024-08-20T21:58:00.0343304Z s3-bucket: gha-artifacts 2024-08-20T21:58:00.0343733Z region: us-east-1 2024-08-20T21:58:00.0344072Z env: 2024-08-20T21:58:00.0344398Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:00.0344959Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:00.0345532Z ##[endgroup] 2024-08-20T21:58:00.6419475Z (node:379220) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T21:58:00.6420354Z 2024-08-20T21:58:00.6420683Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T21:58:00.6421594Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T21:58:00.6422736Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T21:58:00.7218764Z Found 1 objects with prefix pytorch/pytorch/10479309237/td_results/ 2024-08-20T21:58:00.7220000Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-08-20T21:58:00.7743623Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-08-20T21:58:00.7754429Z Artifact download has finished successfully 2024-08-20T21:58:00.7961138Z ##[group]Run mkdir -p .additional_ci_files 2024-08-20T21:58:00.7961739Z mkdir -p .additional_ci_files 2024-08-20T21:58:00.7962437Z mv td_results.json .additional_ci_files/td_results.json 2024-08-20T21:58:00.7970386Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:58:00.7971013Z env: 2024-08-20T21:58:00.7971328Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:00.7971885Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:00.7972478Z ##[endgroup] 2024-08-20T21:58:00.8069509Z ##[group]Run .github/scripts/parse_ref.py 2024-08-20T21:58:00.8070138Z .github/scripts/parse_ref.py 2024-08-20T21:58:00.8076656Z shell: /usr/bin/bash -e {0} 2024-08-20T21:58:00.8077093Z env: 2024-08-20T21:58:00.8077423Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:00.8077968Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:00.8078566Z ##[endgroup] 2024-08-20T21:58:00.8433556Z Prepare all required actions 2024-08-20T21:58:00.8486725Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-20T21:58:00.8487300Z with: 2024-08-20T21:58:00.8488039Z github-token: *** 2024-08-20T21:58:00.8488404Z env: 2024-08-20T21:58:00.8488731Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:00.8489292Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:00.8489876Z ##[endgroup] 2024-08-20T21:58:00.8513966Z ##[group]Run set -eux 2024-08-20T21:58:00.8514384Z set -eux 2024-08-20T21:58:00.8515117Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-20T21:58:00.8522182Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:58:00.8522802Z env: 2024-08-20T21:58:00.8523138Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:00.8524022Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:00.8524847Z GITHUB_TOKEN: *** 2024-08-20T21:58:00.8525228Z ##[endgroup] 2024-08-20T21:58:00.8553091Z + python3 .github/scripts/get_workflow_job_id.py 10479309237 i-08a502d1ce8446e65 2024-08-20T21:58:03.0427460Z setting job-id=29025850427 2024-08-20T21:58:03.0428644Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, amz2023.linux.8xlarge.nvidia.gpu) 2024-08-20T21:58:03.0702175Z Prepare all required actions 2024-08-20T21:58:03.0702753Z Getting action download info 2024-08-20T21:58:03.1824998Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-20T21:58:03.1825567Z with: 2024-08-20T21:58:03.1826139Z github-token: *** 2024-08-20T21:58:03.1828334Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}]} 2024-08-20T21:58:03.1830815Z job-name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, amz2023.linux.8xlarge.nvidia.gpu) 2024-08-20T21:58:03.1831748Z env: 2024-08-20T21:58:03.1832075Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:03.1832613Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:03.1833203Z ##[endgroup] 2024-08-20T21:58:03.1914639Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T21:58:03.1915327Z with: 2024-08-20T21:58:03.1915628Z shell: bash 2024-08-20T21:58:03.1915989Z timeout_minutes: 10 2024-08-20T21:58:03.1916380Z max_attempts: 5 2024-08-20T21:58:03.1916742Z retry_wait_seconds: 30 2024-08-20T21:58:03.1918152Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T21:58:03.1919630Z polling_interval_seconds: 1 2024-08-20T21:58:03.1920062Z warning_on_retry: true 2024-08-20T21:58:03.1920481Z continue_on_error: false 2024-08-20T21:58:03.1920887Z env: 2024-08-20T21:58:03.1921194Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:03.1921750Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:03.1922569Z GITHUB_TOKEN: *** 2024-08-20T21:58:03.1922936Z ##[endgroup] 2024-08-20T21:58:03.2855590Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T21:58:03.6119166Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T21:58:03.6320518Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.9/site-packages (2.27.1) 2024-08-20T21:58:03.6326372Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.9/site-packages (6.0.1) 2024-08-20T21:58:03.6480039Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2024.7.4) 2024-08-20T21:58:03.6494345Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-08-20T21:58:03.6507765Z Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2.0.12) 2024-08-20T21:58:03.6515736Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-08-20T21:58:04.2645874Z Command completed after 1 attempt(s). 2024-08-20T21:58:04.2711124Z ##[group]Run set -x 2024-08-20T21:58:04.2711548Z set -x 2024-08-20T21:58:04.2711922Z  2024-08-20T21:58:04.2712609Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T21:58:04.2713470Z # in runner workspace 2024-08-20T21:58:04.2714133Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-20T21:58:04.2722106Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:58:04.2722712Z env: 2024-08-20T21:58:04.2723056Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:04.2723626Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:04.2724218Z ##[endgroup] 2024-08-20T21:58:04.2757123Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-20T21:58:04.3107013Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T21:58:04.3107727Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T21:58:04.3108287Z echo "Job name: ${JOB_NAME}" 2024-08-20T21:58:04.3108781Z  2024-08-20T21:58:04.3109453Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T21:58:04.3110475Z # in runner workspace 2024-08-20T21:58:04.3111192Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-20T21:58:04.3111992Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-20T21:58:04.3112541Z  --job-name "${JOB_NAME}" \ 2024-08-20T21:58:04.3114669Z  --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}]}" \ 2024-08-20T21:58:04.3116842Z  --selected-test-configs "" \ 2024-08-20T21:58:04.3117414Z  --pr-number "${PR_NUMBER}" \ 2024-08-20T21:58:04.3117935Z  --tag "${TAG}" \ 2024-08-20T21:58:04.3118399Z  --event-name "${EVENT_NAME}" \ 2024-08-20T21:58:04.3118952Z  --schedule "${SCHEDULE}" \ 2024-08-20T21:58:04.3119484Z  --branch "${HEAD_BRANCH}" 2024-08-20T21:58:04.3126583Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:58:04.3127206Z env: 2024-08-20T21:58:04.3127546Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:04.3128091Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:04.3128943Z GITHUB_TOKEN: *** 2024-08-20T21:58:04.3129817Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, amz2023.linux.8xlarge.nvidia.gpu) 2024-08-20T21:58:04.3130754Z PR_NUMBER: 133712 2024-08-20T21:58:04.3131129Z TAG: 2024-08-20T21:58:04.3131473Z EVENT_NAME: pull_request 2024-08-20T21:58:04.3131889Z SCHEDULE: 2024-08-20T21:58:04.3132245Z HEAD_BRANCH: 2024-08-20T21:58:04.3132608Z ##[endgroup] 2024-08-20T21:58:04.3161446Z Workflow: pull 2024-08-20T21:58:04.3162576Z Job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, amz2023.linux.8xlarge.nvidia.gpu) 2024-08-20T21:58:04.6708088Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-20T21:58:04.8948577Z ##[group]Run echo "Filtered matrix:" 2024-08-20T21:58:04.8949124Z echo "Filtered matrix:" 2024-08-20T21:58:04.8951151Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "amz2023.linux.8xlarge.nvidia.gpu"}]}" 2024-08-20T21:58:04.8953238Z  2024-08-20T21:58:04.8953545Z echo 2024-08-20T21:58:04.8954021Z echo "Is the current job unstable? False" 2024-08-20T21:58:04.8954582Z  2024-08-20T21:58:04.8954886Z echo 2024-08-20T21:58:04.8955317Z echo "Is keep-going label set? False" 2024-08-20T21:58:04.8955854Z  2024-08-20T21:58:04.8956156Z echo 2024-08-20T21:58:04.8956548Z echo "Renabled issues? " 2024-08-20T21:58:04.8963560Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:58:04.8964158Z env: 2024-08-20T21:58:04.8964490Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:04.8965051Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:04.8965625Z ##[endgroup] 2024-08-20T21:58:04.8994716Z Filtered matrix: 2024-08-20T21:58:04.8996814Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: amz2023.linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 2, num_shards: 3, runner: amz2023.linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 3, num_shards: 3, runner: amz2023.linux.8xlarge.nvidia.gpu}]} 2024-08-20T21:58:04.8998839Z 2024-08-20T21:58:04.8999043Z Is the current job unstable? False 2024-08-20T21:58:04.8999410Z 2024-08-20T21:58:04.8999769Z Is keep-going label set? False 2024-08-20T21:58:04.9000084Z 2024-08-20T21:58:04.9000250Z Renabled issues? 2024-08-20T21:58:04.9054016Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T21:58:04.9055090Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T21:58:04.9061590Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T21:58:04.9062188Z env: 2024-08-20T21:58:04.9062518Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:04.9063074Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:04.9063658Z JOB_TIMEOUT: 360 2024-08-20T21:58:04.9064026Z ##[endgroup] 2024-08-20T21:58:04.9162011Z ##[group]Run set -x 2024-08-20T21:58:04.9162553Z set -x 2024-08-20T21:58:04.9162918Z  2024-08-20T21:58:04.9163352Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-20T21:58:04.9164018Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-20T21:58:04.9164721Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-20T21:58:04.9165357Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-20T21:58:04.9165850Z else 2024-08-20T21:58:04.9166281Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T21:58:04.9166802Z fi 2024-08-20T21:58:04.9167118Z  2024-08-20T21:58:04.9167697Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-20T21:58:04.9168643Z # TODO: Stop building test binaries as part of the build phase 2024-08-20T21:58:04.9169475Z # Used for GPU_FLAG since that doesn't play nice 2024-08-20T21:58:04.9170212Z # shellcheck disable=SC2086,SC2090 2024-08-20T21:58:04.9170770Z container_name=$(docker run \ 2024-08-20T21:58:04.9171288Z  ${GPU_FLAG:-} \ 2024-08-20T21:58:04.9171748Z  -e BUILD_ENVIRONMENT \ 2024-08-20T21:58:04.9172222Z  -e PR_NUMBER \ 2024-08-20T21:58:04.9172666Z  -e GITHUB_ACTIONS \ 2024-08-20T21:58:04.9173147Z  -e GITHUB_REPOSITORY \ 2024-08-20T21:58:04.9173625Z  -e GITHUB_WORKFLOW \ 2024-08-20T21:58:04.9174095Z  -e GITHUB_JOB \ 2024-08-20T21:58:04.9174547Z  -e GITHUB_RUN_ID \ 2024-08-20T21:58:04.9175015Z  -e GITHUB_RUN_NUMBER \ 2024-08-20T21:58:04.9175494Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-20T21:58:04.9175981Z  -e JOB_ID \ 2024-08-20T21:58:04.9176391Z  -e JOB_NAME \ 2024-08-20T21:58:04.9176799Z  -e BASE_SHA \ 2024-08-20T21:58:04.9177215Z  -e BRANCH \ 2024-08-20T21:58:04.9177618Z  -e SHA1 \ 2024-08-20T21:58:04.9178023Z  -e AWS_DEFAULT_REGION \ 2024-08-20T21:58:04.9178522Z  -e IN_WHEEL_TEST \ 2024-08-20T21:58:04.9178985Z  -e SHARD_NUMBER \ 2024-08-20T21:58:04.9179417Z  -e TEST_CONFIG \ 2024-08-20T21:58:04.9179872Z  -e NUM_TEST_SHARDS \ 2024-08-20T21:58:04.9180351Z  -e REENABLED_ISSUES \ 2024-08-20T21:58:04.9180840Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-20T21:58:04.9181359Z  -e VERBOSE_TEST_LOGS \ 2024-08-20T21:58:04.9181850Z  -e TEST_SHOWLOCALS \ 2024-08-20T21:58:04.9182317Z  -e NO_TEST_TIMEOUT \ 2024-08-20T21:58:04.9182778Z  -e NO_TD \ 2024-08-20T21:58:04.9183192Z  -e TD_DISTRIBUTED \ 2024-08-20T21:58:04.9183638Z  -e PR_LABELS \ 2024-08-20T21:58:04.9184137Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-20T21:58:04.9184698Z  -e SCCACHE_BUCKET \ 2024-08-20T21:58:04.9185164Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-20T21:58:04.9185662Z  -e XLA_CUDA \ 2024-08-20T21:58:04.9186161Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-20T21:58:04.9186774Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-20T21:58:04.9187418Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-20T21:58:04.9188049Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-20T21:58:04.9188608Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-20T21:58:04.9189170Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-08-20T21:58:04.9189713Z  -e DASHBOARD_TAG \ 2024-08-20T21:58:04.9190495Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-20T21:58:04.9191182Z  --security-opt seccomp=unconfined \ 2024-08-20T21:58:04.9191755Z  --cap-add=SYS_PTRACE \ 2024-08-20T21:58:04.9192234Z  --ipc=host \ 2024-08-20T21:58:04.9193045Z  --shm-size="${SHM_SIZE}" \ 2024-08-20T21:58:04.9193540Z  --tty \ 2024-08-20T21:58:04.9193920Z  --detach \ 2024-08-20T21:58:04.9194511Z  --name="${container_name}" \ 2024-08-20T21:58:04.9195058Z  --user jenkins \ 2024-08-20T21:58:04.9195659Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-20T21:58:04.9196335Z  -w /var/lib/jenkins/workspace \ 2024-08-20T21:58:04.9196871Z  "${DOCKER_IMAGE}" 2024-08-20T21:58:04.9197296Z ) 2024-08-20T21:58:04.9197778Z # Propagate download.pytorch.org IP to container 2024-08-20T21:58:04.9198933Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-20T21:58:04.9200157Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-20T21:58:04.9201273Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-20T21:58:04.9208269Z shell: /usr/bin/bash -e {0} 2024-08-20T21:58:04.9208711Z env: 2024-08-20T21:58:04.9209029Z GIT_DEFAULT_BRANCH: main 2024-08-20T21:58:04.9209602Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:04.9210334Z BUILD_ENVIRONMENT: linux-focal-cuda11.8-py3.10-gcc9 2024-08-20T21:58:04.9210932Z PR_NUMBER: 133712 2024-08-20T21:58:04.9211337Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-20T21:58:04.9211845Z GITHUB_WORKFLOW: pull 2024-08-20T21:58:04.9212254Z GITHUB_JOB: test 2024-08-20T21:58:04.9212622Z GITHUB_RUN_ID: 10479309237 2024-08-20T21:58:04.9213071Z GITHUB_RUN_NUMBER: 238988 2024-08-20T21:58:04.9213508Z GITHUB_RUN_ATTEMPT: 1 2024-08-20T21:58:04.9213908Z JOB_ID: 29025850427 2024-08-20T21:58:04.9214761Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, amz2023.linux.8xlarge.nvidia.gpu) 2024-08-20T21:58:04.9215710Z BRANCH: pull/133712 2024-08-20T21:58:04.9216164Z SHA1: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:58:04.9216810Z BASE_SHA: 91f3d614142df02f619e44a68e3d9e0dfeba49ec 2024-08-20T21:58:04.9217391Z TEST_CONFIG: distributed 2024-08-20T21:58:04.9217800Z SHARD_NUMBER: 1 2024-08-20T21:58:04.9218182Z NUM_TEST_SHARDS: 3 2024-08-20T21:58:04.9218578Z REENABLED_ISSUES: 2024-08-20T21:58:04.9218972Z CONTINUE_THROUGH_ERROR: False 2024-08-20T21:58:04.9219449Z VERBOSE_TEST_LOGS: False 2024-08-20T21:58:04.9219881Z TEST_SHOWLOCALS: False 2024-08-20T21:58:04.9220289Z NO_TEST_TIMEOUT: False 2024-08-20T21:58:04.9220702Z NO_TD: False 2024-08-20T21:58:04.9221063Z TD_DISTRIBUTED: False 2024-08-20T21:58:04.9221557Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-20T21:58:04.9222160Z SCCACHE_S3_KEY_PREFIX: pull 2024-08-20T21:58:04.9222607Z SHM_SIZE: 2g 2024-08-20T21:58:04.9223777Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:58:04.9225080Z XLA_CUDA: 2024-08-20T21:58:04.9225672Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-20T21:58:04.9226415Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-20T21:58:04.9226954Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-20T21:58:04.9227458Z DASHBOARD_TAG: 2024-08-20T21:58:04.9227842Z HUGGING_FACE_HUB_TOKEN: 2024-08-20T21:58:04.9228269Z SCRIBE_GRAPHQL_ACCESS_TOKEN: 2024-08-20T21:58:04.9228721Z ##[endgroup] 2024-08-20T21:58:04.9256938Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-08-20T21:58:04.9257740Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *onnx* ]] 2024-08-20T21:58:04.9258345Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T21:58:04.9266848Z +++ nproc --ignore=2 2024-08-20T21:58:04.9285797Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10479309237 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T21:58:16.0681070Z + container_name=9afe50b973f3f5524a846944b5650c0e68adccbc8ed115dca00a11ff567068c4 2024-08-20T21:58:16.0683515Z + grep download.pytorch.org /etc/hosts 2024-08-20T21:58:16.0686200Z + docker exec -i 9afe50b973f3f5524a846944b5650c0e68adccbc8ed115dca00a11ff567068c4 sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-20T21:58:16.2241443Z + echo DOCKER_CONTAINER_ID=9afe50b973f3f5524a846944b5650c0e68adccbc8ed115dca00a11ff567068c4 2024-08-20T21:58:16.2245786Z ++ echo dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T21:58:16.2248540Z + docker exec -t 9afe50b973f3f5524a846944b5650c0e68adccbc8ed115dca00a11ff567068c4 sh -c 'pip install dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-20T21:58:16.7347187Z Processing ./dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+git40ec5f6) 2024-08-20T21:58:17.8640203Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.13.1) 2024-08-20T21:58:17.8644476Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (4.12.2) 2024-08-20T21:58:17.8648888Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.8.8) 2024-08-20T21:58:17.8653615Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.1.4) 2024-08-20T21:58:17.8658352Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2024.6.1) 2024-08-20T21:58:17.8665879Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.13.1) 2024-08-20T21:58:17.8688930Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.3.0) 2024-08-20T21:58:17.8709087Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.3.0) 2024-08-20T21:58:17.8734156Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.22.4) 2024-08-20T21:58:17.9261503Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.1.5) 2024-08-20T21:58:18.3567641Z Installing collected packages: torch 2024-08-20T21:58:31.4332398Z Successfully installed torch-2.5.0a0+git40ec5f6 2024-08-20T21:58:31.5058355Z + export TERM=vt100 2024-08-20T21:58:31.5061329Z + TERM=vt100 2024-08-20T21:58:31.5062514Z ++ dirname .ci/pytorch/test.sh 2024-08-20T21:58:31.5072247Z + source .ci/pytorch/common.sh 2024-08-20T21:58:31.5076263Z +++ dirname .ci/pytorch/common.sh 2024-08-20T21:58:31.5084980Z ++ source .ci/pytorch/common_utils.sh 2024-08-20T21:58:31.5087598Z +++ declare -f -t trap_add 2024-08-20T21:58:31.5095439Z ++ set -ex 2024-08-20T21:58:31.5095994Z ++ [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-08-20T21:58:31.5096577Z ++ BUILD_TEST_LIBTORCH=0 2024-08-20T21:58:31.5097530Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-08-20T21:58:31.5101316Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-20T21:58:31.5118616Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-20T21:58:31.5119127Z + trap_add cleanup_workspace EXIT 2024-08-20T21:58:31.5119653Z + trap_add_cmd=cleanup_workspace 2024-08-20T21:58:31.5120104Z + shift 2024-08-20T21:58:31.5122779Z + for trap_add_name in "$@" 2024-08-20T21:58:31.5127560Z +++ trap -p EXIT 2024-08-20T21:58:31.5130692Z ++ eval 'extract_trap_cmd ' 2024-08-20T21:58:31.5131150Z +++ extract_trap_cmd 2024-08-20T21:58:31.5131599Z +++ printf '%s\n' '' 2024-08-20T21:58:31.5132058Z ++ printf '%s\n' cleanup_workspace 2024-08-20T21:58:31.5134050Z + trap -- ' 2024-08-20T21:58:31.5134491Z cleanup_workspace' EXIT 2024-08-20T21:58:31.5135050Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-20T21:58:32.1147858Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-20T21:58:32.1168273Z + echo 'Environment variables:' 2024-08-20T21:58:32.1168756Z Environment variables: 2024-08-20T21:58:32.1169155Z + env 2024-08-20T21:58:32.1178428Z INSTALLED_DB=yes 2024-08-20T21:58:32.1179446Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-08-20T21:58:32.1180431Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T21:58:32.1181119Z NV_NVML_DEV_VERSION=11.8.86-1 2024-08-20T21:58:32.1181863Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T21:58:32.1182940Z CONTINUE_THROUGH_ERROR=False 2024-08-20T21:58:32.1183715Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-08-20T21:58:32.1184391Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-08-20T21:58:32.1185090Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-08-20T21:58:32.1185678Z HOSTNAME=9afe50b973f3 2024-08-20T21:58:32.1186792Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_e5b7ade3-2ed2-4699-b26f-cc21fbc289cf 2024-08-20T21:58:32.1187832Z GITHUB_ACTION=__self 2024-08-20T21:58:32.1188241Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T21:58:32.1190923Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-08-20T21:58:32.1194204Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-08-20T21:58:32.1194847Z NV_NVTX_VERSION=11.8.86-1 2024-08-20T21:58:32.1195264Z GITHUB_RUN_NUMBER=238988 2024-08-20T21:58:32.1195691Z TEST_CONFIG=distributed 2024-08-20T21:58:32.1196176Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T21:58:32.1196789Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T21:58:32.1197348Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-08-20T21:58:32.1197903Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-08-20T21:58:32.1198398Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T21:58:32.1198871Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-08-20T21:58:32.1199365Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2024-08-20T21:58:32.1199981Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T21:58:32.1200544Z GITHUB_REF_TYPE=branch 2024-08-20T21:58:32.1200962Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T21:58:32.1201763Z NCCL_VERSION=2.16.2-1 2024-08-20T21:58:32.1202228Z BASE_SHA=91f3d614142df02f619e44a68e3d9e0dfeba49ec 2024-08-20T21:58:32.1202776Z XLA_CUDA= 2024-08-20T21:58:32.1203119Z HUGGING_FACE_HUB_TOKEN= 2024-08-20T21:58:32.1205722Z *** 2024-08-20T21:58:32.1206096Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T21:58:32.1206596Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T21:58:32.1207022Z GITHUB_ACTIONS=true 2024-08-20T21:58:32.1207651Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:32.1208358Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-08-20T21:58:32.1209015Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-08-20T21:58:32.1209599Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:58:32.1210245Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T21:58:32.1210836Z GITHUB_SHA=f2fb9405c2fa9f9502a76363091cce6fd8179736 2024-08-20T21:58:32.1211730Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/133712/merge 2024-08-20T21:58:32.1212536Z UCC_HOME=/usr 2024-08-20T21:58:32.1212946Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-08-20T21:58:32.1213445Z VERBOSE_TEST_LOGS=False 2024-08-20T21:58:32.1213866Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T21:58:32.1214426Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-08-20T21:58:32.1215011Z GITHUB_REF=refs/pull/133712/merge 2024-08-20T21:58:32.1215533Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-08-20T21:58:32.1215978Z SHARD_NUMBER=1 2024-08-20T21:58:32.1216364Z GITHUB_REF_PROTECTED=false 2024-08-20T21:58:32.1216800Z HOME=/var/lib/jenkins 2024-08-20T21:58:32.1217237Z GITHUB_API_URL=https://api.github.com 2024-08-20T21:58:32.1217783Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T21:58:32.1218368Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T21:58:32.1218944Z SCCACHE_S3_KEY_PREFIX=pull 2024-08-20T21:58:32.1219375Z CUDA_VERSION=11.8.0 2024-08-20T21:58:32.1219914Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-08-20T21:58:32.1220442Z NUM_TEST_SHARDS=3 2024-08-20T21:58:32.1220807Z UCX_HOME=/usr 2024-08-20T21:58:32.1221475Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-08-20T21:58:32.1222844Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_e5b7ade3-2ed2-4699-b26f-cc21fbc289cf 2024-08-20T21:58:32.1224464Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, amz2023.linux.8xlarge.nvidia.gpu) 2024-08-20T21:58:32.1226080Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_e5b7ade3-2ed2-4699-b26f-cc21fbc289cf 2024-08-20T21:58:32.1227524Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T21:58:32.1228339Z GITHUB_EVENT_NAME=pull_request 2024-08-20T21:58:32.1228793Z DASHBOARD_TAG= 2024-08-20T21:58:32.1229160Z GITHUB_RUN_ID=10479309237 2024-08-20T21:58:32.1229728Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-08-20T21:58:32.1230406Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-08-20T21:58:32.1231707Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_e5b7ade3-2ed2-4699-b26f-cc21fbc289cf 2024-08-20T21:58:32.1232804Z GITHUB_ACTOR=pytorchmergebot 2024-08-20T21:58:32.1233310Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-08-20T21:58:32.1233781Z PR_NUMBER=133712 2024-08-20T21:58:32.1234174Z GITHUB_RUN_ATTEMPT=1 2024-08-20T21:58:32.1234586Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T21:58:32.1235137Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T21:58:32.1235695Z TERM=vt100 2024-08-20T21:58:32.1236113Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-08-20T21:58:32.1236617Z INSTALLED_VISION=yes 2024-08-20T21:58:32.1236992Z BRANCH=pull/133712 2024-08-20T21:58:32.1237392Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T21:58:32.1237888Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T21:58:32.1238396Z CUDA_PATH=/usr/local/cuda 2024-08-20T21:58:32.1239370Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T21:58:32.1240318Z GITHUB_SERVER_URL=https://github.com 2024-08-20T21:58:32.1241025Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T21:58:32.1241608Z REENABLED_ISSUES= 2024-08-20T21:58:32.1241971Z SHLVL=1 2024-08-20T21:58:32.1242268Z MAX_JOBS=30 2024-08-20T21:58:32.1242671Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-08-20T21:58:32.1243108Z NVARCH=x86_64 2024-08-20T21:58:32.1243451Z GITHUB_ACTOR_ID=97764156 2024-08-20T21:58:32.1244016Z GITHUB_WORKFLOW_SHA=f2fb9405c2fa9f9502a76363091cce6fd8179736 2024-08-20T21:58:32.1244749Z GITHUB_REF_NAME=133712/merge 2024-08-20T21:58:32.1245303Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-08-20T21:58:32.1246148Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T21:58:32.1246865Z GITHUB_JOB=test 2024-08-20T21:58:32.1247349Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-08-20T21:58:32.1248038Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T21:58:32.1248675Z NO_TEST_TIMEOUT=False 2024-08-20T21:58:32.1249065Z TD_DISTRIBUTED=False 2024-08-20T21:58:32.1249548Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-08-20T21:58:32.1250096Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T21:58:32.1250594Z NV_NVPROF_VERSION=11.8.87-1 2024-08-20T21:58:32.1251039Z GITHUB_RETENTION_DAYS=90 2024-08-20T21:58:32.1251468Z OPENSSL_DIR=/opt/openssl 2024-08-20T21:58:32.1251880Z GITHUB_ACTION_REPOSITORY= 2024-08-20T21:58:32.1253196Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T21:58:32.1254643Z GITHUB_BASE_REF=gh/XuehaiPan/146/base 2024-08-20T21:58:32.1255170Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T21:58:32.1255607Z CI=true 2024-08-20T21:58:32.1256006Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-08-20T21:58:32.1256520Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T21:58:32.1256964Z JOB_ID=29025850427 2024-08-20T21:58:32.1257345Z INSTALLED_PROTOBUF=yes 2024-08-20T21:58:32.1257768Z GITHUB_HEAD_REF=gh/XuehaiPan/146/head 2024-08-20T21:58:32.1258270Z GITHUB_ACTION_REF= 2024-08-20T21:58:32.1258833Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T21:58:32.1259394Z TEST_SHOWLOCALS=False 2024-08-20T21:58:32.1259802Z GITHUB_WORKFLOW=pull 2024-08-20T21:58:32.1260221Z DEBIAN_FRONTEND=noninteractive 2024-08-20T21:58:32.1261380Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_e5b7ade3-2ed2-4699-b26f-cc21fbc289cf 2024-08-20T21:58:32.1262407Z NO_TD=False 2024-08-20T21:58:32.1262785Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T21:58:32.1263233Z _=/usr/bin/env 2024-08-20T21:58:32.1263823Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-20T21:58:32.1480010Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-08-20T21:58:32.1481138Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T21:58:32.1482250Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-08-20T21:58:32.1483373Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-08-20T21:58:32.1484147Z + BUILD_DIR=build 2024-08-20T21:58:32.1484530Z + BUILD_RENAMED_DIR=build_renamed 2024-08-20T21:58:32.1485007Z + BUILD_BIN_DIR=build/bin 2024-08-20T21:58:32.1485424Z + SHARD_NUMBER=1 2024-08-20T21:58:32.1485778Z + NUM_TEST_SHARDS=3 2024-08-20T21:58:32.1486168Z + export VALGRIND=ON 2024-08-20T21:58:32.1486559Z + VALGRIND=ON 2024-08-20T21:58:32.1487100Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *clang9* ]] 2024-08-20T21:58:32.1487686Z + [[ 0 == \1 ]] 2024-08-20T21:58:32.1488048Z + [[ False == \1 ]] 2024-08-20T21:58:32.1488585Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *bazel* ]] 2024-08-20T21:58:32.1489210Z ++ realpath build/custom_test_artifacts 2024-08-20T21:58:32.1499630Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-20T21:58:32.1500522Z + [[ -n '' ]] 2024-08-20T21:58:32.1500939Z + echo 'Environment variables' 2024-08-20T21:58:32.1501406Z Environment variables 2024-08-20T21:58:32.1502021Z + env 2024-08-20T21:58:32.1508524Z INSTALLED_DB=yes 2024-08-20T21:58:32.1509006Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-08-20T21:58:32.1509488Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T21:58:32.1509947Z NV_NVML_DEV_VERSION=11.8.86-1 2024-08-20T21:58:32.1510735Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T21:58:32.1511441Z CONTINUE_THROUGH_ERROR=False 2024-08-20T21:58:32.1512047Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-08-20T21:58:32.1512903Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-08-20T21:58:32.1513649Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-08-20T21:58:32.1514220Z HOSTNAME=9afe50b973f3 2024-08-20T21:58:32.1515307Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_e5b7ade3-2ed2-4699-b26f-cc21fbc289cf 2024-08-20T21:58:32.1516325Z GITHUB_ACTION=__self 2024-08-20T21:58:32.1516737Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T21:58:32.1519402Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-08-20T21:58:32.1522241Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-08-20T21:58:32.1522880Z NV_NVTX_VERSION=11.8.86-1 2024-08-20T21:58:32.1523295Z GITHUB_RUN_NUMBER=238988 2024-08-20T21:58:32.1523725Z TEST_CONFIG=distributed 2024-08-20T21:58:32.1524165Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T21:58:32.1524742Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T21:58:32.1525316Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-08-20T21:58:32.1525978Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-08-20T21:58:32.1526703Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T21:58:32.1527217Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-08-20T21:58:32.1527708Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2024-08-20T21:58:32.1528311Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T21:58:32.1528893Z GITHUB_REF_TYPE=branch 2024-08-20T21:58:32.1529310Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T21:58:32.1529766Z NCCL_VERSION=2.16.2-1 2024-08-20T21:58:32.1530248Z BASE_SHA=91f3d614142df02f619e44a68e3d9e0dfeba49ec 2024-08-20T21:58:32.1530791Z XLA_CUDA= 2024-08-20T21:58:32.1531118Z HUGGING_FACE_HUB_TOKEN= 2024-08-20T21:58:32.1531607Z *** 2024-08-20T21:58:32.1532018Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T21:58:32.1532509Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T21:58:32.1532943Z GITHUB_ACTIONS=true 2024-08-20T21:58:32.1533352Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T21:58:32.1533980Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-08-20T21:58:32.1534632Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-08-20T21:58:32.1535221Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T21:58:32.1535866Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T21:58:32.1536456Z GITHUB_SHA=f2fb9405c2fa9f9502a76363091cce6fd8179736 2024-08-20T21:58:32.1537356Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/133712/merge 2024-08-20T21:58:32.1538146Z UCC_HOME=/usr 2024-08-20T21:58:32.1538552Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-08-20T21:58:32.1539052Z VERBOSE_TEST_LOGS=False 2024-08-20T21:58:32.1539471Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T21:58:32.1540033Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-08-20T21:58:32.1540630Z GITHUB_REF=refs/pull/133712/merge 2024-08-20T21:58:32.1541157Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-08-20T21:58:32.1541602Z SHARD_NUMBER=1 2024-08-20T21:58:32.1541983Z GITHUB_REF_PROTECTED=false 2024-08-20T21:58:32.1542419Z HOME=/var/lib/jenkins 2024-08-20T21:58:32.1542852Z GITHUB_API_URL=https://api.github.com 2024-08-20T21:58:32.1543392Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T21:58:32.1543977Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T21:58:32.1544554Z SCCACHE_S3_KEY_PREFIX=pull 2024-08-20T21:58:32.1545135Z CUDA_VERSION=11.8.0 2024-08-20T21:58:32.1545676Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-08-20T21:58:32.1546202Z NUM_TEST_SHARDS=3 2024-08-20T21:58:32.1546572Z UCX_HOME=/usr 2024-08-20T21:58:32.1547227Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-08-20T21:58:32.1548599Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_e5b7ade3-2ed2-4699-b26f-cc21fbc289cf 2024-08-20T21:58:32.1550312Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, amz2023.linux.8xlarge.nvidia.gpu) 2024-08-20T21:58:32.1551946Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_e5b7ade3-2ed2-4699-b26f-cc21fbc289cf 2024-08-20T21:58:32.1553429Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T21:58:32.1554244Z GITHUB_EVENT_NAME=pull_request 2024-08-20T21:58:32.1554698Z DASHBOARD_TAG= 2024-08-20T21:58:32.1555071Z GITHUB_RUN_ID=10479309237 2024-08-20T21:58:32.1555651Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-08-20T21:58:32.1556329Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-08-20T21:58:32.1557621Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_e5b7ade3-2ed2-4699-b26f-cc21fbc289cf 2024-08-20T21:58:32.1558714Z GITHUB_ACTOR=pytorchmergebot 2024-08-20T21:58:32.1559218Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-08-20T21:58:32.1559686Z PR_NUMBER=133712 2024-08-20T21:58:32.1560051Z GITHUB_RUN_ATTEMPT=1 2024-08-20T21:58:32.1560443Z VALGRIND=ON 2024-08-20T21:58:32.1560806Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T21:58:32.1561340Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T21:58:32.1561904Z TERM=vt100 2024-08-20T21:58:32.1562319Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-08-20T21:58:32.1562806Z INSTALLED_VISION=yes 2024-08-20T21:58:32.1563202Z BRANCH=pull/133712 2024-08-20T21:58:32.1563598Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T21:58:32.1564074Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T21:58:32.1564607Z CUDA_PATH=/usr/local/cuda 2024-08-20T21:58:32.1565575Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T21:58:32.1566502Z GITHUB_SERVER_URL=https://github.com 2024-08-20T21:58:32.1567102Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T21:58:32.1567683Z REENABLED_ISSUES= 2024-08-20T21:58:32.1568023Z SHLVL=1 2024-08-20T21:58:32.1568341Z MAX_JOBS=30 2024-08-20T21:58:32.1568737Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-08-20T21:58:32.1569151Z NVARCH=x86_64 2024-08-20T21:58:32.1569514Z GITHUB_ACTOR_ID=97764156 2024-08-20T21:58:32.1570078Z GITHUB_WORKFLOW_SHA=f2fb9405c2fa9f9502a76363091cce6fd8179736 2024-08-20T21:58:32.1570704Z GITHUB_REF_NAME=133712/merge 2024-08-20T21:58:32.1571253Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-08-20T21:58:32.1572096Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T21:58:32.1572788Z GITHUB_JOB=test 2024-08-20T21:58:32.1573292Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-08-20T21:58:32.1573995Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T21:58:32.1574613Z NO_TEST_TIMEOUT=False 2024-08-20T21:58:32.1575017Z TD_DISTRIBUTED=False 2024-08-20T21:58:32.1575490Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-08-20T21:58:32.1593720Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T21:58:32.1594334Z NV_NVPROF_VERSION=11.8.87-1 2024-08-20T21:58:32.1594772Z GITHUB_RETENTION_DAYS=90 2024-08-20T21:58:32.1595225Z OPENSSL_DIR=/opt/openssl 2024-08-20T21:58:32.1595663Z GITHUB_ACTION_REPOSITORY= 2024-08-20T21:58:32.1596977Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T21:58:32.1598367Z GITHUB_BASE_REF=gh/XuehaiPan/146/base 2024-08-20T21:58:32.1598901Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T21:58:32.1599336Z CI=true 2024-08-20T21:58:32.1599740Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-08-20T21:58:32.1600498Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T21:58:32.1600941Z JOB_ID=29025850427 2024-08-20T21:58:32.1601328Z INSTALLED_PROTOBUF=yes 2024-08-20T21:58:32.1601772Z GITHUB_HEAD_REF=gh/XuehaiPan/146/head 2024-08-20T21:58:32.1602254Z GITHUB_ACTION_REF= 2024-08-20T21:58:32.1602823Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T21:58:32.1603400Z TEST_SHOWLOCALS=False 2024-08-20T21:58:32.1603789Z GITHUB_WORKFLOW=pull 2024-08-20T21:58:32.1604210Z DEBIAN_FRONTEND=noninteractive 2024-08-20T21:58:32.1605536Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_e5b7ade3-2ed2-4699-b26f-cc21fbc289cf 2024-08-20T21:58:32.1606578Z NO_TD=False 2024-08-20T21:58:32.1606946Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T21:58:32.1607396Z _=/usr/bin/env 2024-08-20T21:58:32.1607815Z + echo 'Testing pytorch' 2024-08-20T21:58:32.1608238Z Testing pytorch 2024-08-20T21:58:32.1608644Z + export LANG=C.UTF-8 2024-08-20T21:58:32.1609047Z + LANG=C.UTF-8 2024-08-20T21:58:32.1609412Z + PR_NUMBER=133712 2024-08-20T21:58:32.1609840Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-08-20T21:58:32.1610381Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-20T21:58:32.1611095Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-08-20T21:58:32.1611696Z + [[ distributed == \s\l\o\w ]] 2024-08-20T21:58:32.1612390Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *slow-gradcheck* ]] 2024-08-20T21:58:32.1613194Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-08-20T21:58:32.1613852Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T21:58:32.1614397Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T21:58:32.1614927Z + [[ distributed == *crossref* ]] 2024-08-20T21:58:32.1615579Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-08-20T21:58:32.1616295Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *xpu* ]] 2024-08-20T21:58:32.1617057Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-08-20T21:58:32.1617740Z + pip_install --user ninja==1.10.2 2024-08-20T21:58:32.1618399Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-20T21:58:32.7237802Z Collecting ninja==1.10.2 2024-08-20T21:58:32.7440983Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-20T21:58:32.7549909Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-20T21:58:33.1802034Z Installing collected packages: ninja 2024-08-20T21:58:33.1903633Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-20T21:58:33.1905272Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-20T21:58:33.1955885Z Successfully installed ninja-1.10.2 2024-08-20T21:58:33.2632338Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T21:58:33.2635063Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T21:58:33.2636946Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-08-20T21:58:33.2637532Z + install_tlparse 2024-08-20T21:58:33.2638010Z + pip_install --user tlparse==0.3.25 2024-08-20T21:58:33.2638717Z + pip install --progress-bar off --user tlparse==0.3.25 2024-08-20T21:58:33.7597220Z Collecting tlparse==0.3.25 2024-08-20T21:58:33.7798915Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-08-20T21:58:33.7931497Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-20T21:58:34.2497206Z Installing collected packages: tlparse 2024-08-20T21:58:34.2915665Z Successfully installed tlparse-0.3.25 2024-08-20T21:58:34.3619940Z ++ python -m site --user-base 2024-08-20T21:58:34.3946261Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T21:58:34.3948627Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *asan* ]] 2024-08-20T21:58:34.3949391Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-debug* ]] 2024-08-20T21:58:34.3950146Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-08-20T21:58:34.3951468Z + echo 'We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass' 2024-08-20T21:58:34.3952880Z We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass 2024-08-20T21:58:34.3953747Z + cd test 2024-08-20T21:58:34.3954419Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-20T21:58:36.5202398Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-20T21:58:36.5203101Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-20T21:58:36.5207772Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-20T21:58:36.5208468Z + [[ distributed == *pr_time_benchmarks* ]] 2024-08-20T21:58:36.5209042Z + [[ distributed == *dynamo_eager* ]] 2024-08-20T21:58:36.5209574Z + [[ distributed == *aot_eager* ]] 2024-08-20T21:58:36.5210067Z + [[ distributed == *aot_inductor* ]] 2024-08-20T21:58:36.5210582Z + [[ distributed == *inductor* ]] 2024-08-20T21:58:36.5211075Z + [[ distributed == *dynamic* ]] 2024-08-20T21:58:36.5211536Z + [[ distributed == *cpu* ]] 2024-08-20T21:58:36.5212266Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-20T21:58:36.5250880Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-08-20T21:58:36.5251668Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-bazel-* ]] 2024-08-20T21:58:36.5254181Z + cd test 2024-08-20T21:58:36.5254887Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-20T21:58:38.4190243Z PyTorch built with: 2024-08-20T21:58:38.4191496Z - GCC 9.4 2024-08-20T21:58:38.4191932Z - C++ Version: 201703 2024-08-20T21:58:38.4193425Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T21:58:38.4194881Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T21:58:38.4195714Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T21:58:38.4196378Z - LAPACK is enabled (usually provided by MKL) 2024-08-20T21:58:38.4196994Z - NNPACK is enabled 2024-08-20T21:58:38.4197456Z - CPU capability usage: AVX2 2024-08-20T21:58:38.4197986Z - CUDA Runtime 11.8 2024-08-20T21:58:38.4198659Z - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52 2024-08-20T21:58:38.4199337Z - CuDNN 90.1 2024-08-20T21:58:38.4199737Z - Magma 2.6.1 2024-08-20T21:58:38.4208838Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-20T21:58:38.4217139Z 2024-08-20T21:58:38.6869441Z + cd test 2024-08-20T21:58:38.6870902Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-20T21:58:40.5540985Z ATen/Parallel: 2024-08-20T21:58:40.5541969Z at::get_num_threads() : 16 2024-08-20T21:58:40.5542543Z at::get_num_interop_threads() : 16 2024-08-20T21:58:40.5543073Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T21:58:40.5543561Z omp_get_max_threads() : 16 2024-08-20T21:58:40.5545223Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T21:58:40.5546282Z mkl_get_max_threads() : 16 2024-08-20T21:58:40.5547107Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T21:58:40.5547885Z std::thread::hardware_concurrency() : 32 2024-08-20T21:58:40.5548417Z Environment variables: 2024-08-20T21:58:40.5548848Z OMP_NUM_THREADS : [not set] 2024-08-20T21:58:40.5549286Z MKL_NUM_THREADS : [not set] 2024-08-20T21:58:40.5549745Z ATen parallel backend: OpenMP 2024-08-20T21:58:40.5550060Z 2024-08-20T21:58:40.8038599Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-08-20T21:58:40.8040072Z + [[ distributed == *backward* ]] 2024-08-20T21:58:40.8040589Z + [[ distributed == *xla* ]] 2024-08-20T21:58:40.8041051Z + [[ distributed == *executorch* ]] 2024-08-20T21:58:40.8041594Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-20T21:58:40.8042368Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-08-20T21:58:40.8042986Z + [[ distributed == distributed ]] 2024-08-20T21:58:40.8043492Z + test_distributed 2024-08-20T21:58:40.8044010Z + echo 'Testing distributed python tests' 2024-08-20T21:58:40.8044551Z Testing distributed python tests 2024-08-20T21:58:40.8045335Z + python test/run_test.py --distributed-tests --shard 1 3 --verbose 2024-08-20T21:58:40.9369049Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T21:58:40.9370478Z import pkg_resources 2024-08-20T21:58:45.5676680Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T21:58:45.6178221Z Ignoring disabled issues: [''] 2024-08-20T21:58:45.6322502Z Found test times from artifacts 2024-08-20T21:58:45.6874263Z Found test times from artifacts 2024-08-20T21:58:45.6891131Z Running 25% of tests based on TD 2024-08-20T21:58:45.7014727Z Running parallel tests on 2 processes 2024-08-20T21:58:45.7015852Z Name: tests to run (est. time: 11.21min) 2024-08-20T21:58:45.7016665Z Serial tests (11): 2024-08-20T21:58:45.7017642Z distributed/test_inductor_collectives 1/1 2024-08-20T21:58:45.7018555Z distributed/test_data_parallel 1/1 2024-08-20T21:58:45.7019194Z distributed/_composable/test_replicate_with_compiler 1/1 2024-08-20T21:58:45.7020078Z distributed/_tensor/test_dtensor_ops 1/1 2024-08-20T21:58:45.7021568Z distributed/_composable/fsdp/test_fully_shard_compile 1/1 2024-08-20T21:58:45.7022714Z distributed/pipelining/test_schedule_multiproc 1/1 2024-08-20T21:58:45.7023390Z distributed/_tensor/test_xla_integration 1/1 2024-08-20T21:58:45.7024015Z distributed/pipelining/test_unflatten 1/1 2024-08-20T21:58:45.7024631Z distributed/_tensor/test_convolution_ops 1/1 2024-08-20T21:58:45.7025200Z distributed/launcher/test_run 1/1 2024-08-20T21:58:45.7025714Z distributed/test_c10d_gloo 1/1 2024-08-20T21:58:45.7026195Z Parallel tests (0): 2024-08-20T21:58:45.7026614Z Name: excluded (est. time: 76.52min) 2024-08-20T21:58:45.7027108Z Serial tests (54): 2024-08-20T21:58:45.7027553Z distributed/test_distributed_spawn 1/9 2024-08-20T21:58:45.7028108Z distributed/test_distributed_spawn 4/9 2024-08-20T21:58:45.7028675Z distributed/test_distributed_spawn 7/9 2024-08-20T21:58:45.7029217Z distributed/test_c10d_nccl 1/2 2024-08-20T21:58:45.7029737Z distributed/fsdp/test_fsdp_state_dict 1/2 2024-08-20T21:58:45.7030357Z distributed/rpc/cuda/test_tensorpipe_agent 1/2 2024-08-20T21:58:45.7031383Z distributed/_shard/sharded_tensor/test_sharded_tensor 1/1 2024-08-20T21:58:45.7032013Z distributed/test_c10d_pypg 1/1 2024-08-20T21:58:45.7032563Z distributed/checkpoint/test_state_dict 1/1 2024-08-20T21:58:45.7033130Z distributed/fsdp/test_wrap 1/1 2024-08-20T21:58:45.7033614Z distributed/test_c10d_common 1/1 2024-08-20T21:58:45.7034132Z distributed/test_pg_wrapper 1/1 2024-08-20T21:58:45.7034861Z distributed/_composable/test_replicate 1/1 2024-08-20T21:58:45.7035452Z distributed/test_functional_api 1/1 2024-08-20T21:58:45.7036032Z distributed/_tensor/test_redistribute 1/1 2024-08-20T21:58:45.7036608Z distributed/_tensor/test_utils 1/1 2024-08-20T21:58:45.7037125Z distributed/fsdp/test_fsdp_comm 1/1 2024-08-20T21:58:45.7037648Z distributed/test_store 1/1 2024-08-20T21:58:45.7038165Z distributed/_composable/test_compose 1/1 2024-08-20T21:58:45.7038734Z distributed/_tensor/test_matrix_ops 1/1 2024-08-20T21:58:45.7039326Z distributed/fsdp/test_fsdp_exec_order 1/1 2024-08-20T21:58:45.7040043Z distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 2024-08-20T21:58:45.7040842Z distributed/_composable/fully_shard/test_fully_shard_init 1/1 2024-08-20T21:58:45.7041549Z distributed/fsdp/test_fsdp_fine_tune 1/1 2024-08-20T21:58:45.7042129Z distributed/_tensor/test_view_ops 1/1 2024-08-20T21:58:45.7042662Z distributed/test_c10d_spawn_ucc 1/1 2024-08-20T21:58:45.7043297Z distributed/_tensor/debug/test_comm_mode_features 1/1 2024-08-20T21:58:45.7044005Z distributed/checkpoint/test_state_dict_utils 1/1 2024-08-20T21:58:45.7044599Z distributed/test_c10d_logger 1/1 2024-08-20T21:58:45.7045189Z distributed/checkpoint/test_hsdp_checkpoint 1/1 2024-08-20T21:58:45.7045874Z distributed/elastic/timer/local_timer_example 1/1 2024-08-20T21:58:45.7046548Z distributed/_shard/sharded_tensor/ops/test_init 1/1 2024-08-20T21:58:45.7047278Z distributed/_shard/sharding_plan/test_sharding_plan 1/1 2024-08-20T21:58:45.7048149Z distributed/_composable/fully_shard/test_fully_shard_optim_checkpoint 1/1 2024-08-20T21:58:45.7048967Z distributed/checkpoint/test_tp_checkpoint 1/1 2024-08-20T21:58:45.7049554Z distributed/fsdp/test_fsdp_input 1/1 2024-08-20T21:58:45.7050109Z distributed/_shard/test_sharder 1/1 2024-08-20T21:58:45.7050904Z distributed/_shard/sharded_tensor/ops/test_embedding 1/1 2024-08-20T21:58:45.7051596Z distributed/elastic/test_control_plane 1/1 2024-08-20T21:58:45.7052230Z distributed/elastic/timer/local_timer_test 1/1 2024-08-20T21:58:45.7052917Z distributed/checkpoint/test_dtensor_checkpoint 1/1 2024-08-20T21:58:45.7053751Z distributed/_composable/fully_shard/test_fully_shard_mixed_precision 1/1 2024-08-20T21:58:45.7054566Z distributed/rpc/test_share_memory 1/1 2024-08-20T21:58:45.7055130Z distributed/pipelining/test_stage 1/1 2024-08-20T21:58:45.7055652Z distributed/test_launcher 1/1 2024-08-20T21:58:45.7056193Z distributed/_tensor/test_op_strategy 1/1 2024-08-20T21:58:45.7056807Z distributed/_tools/test_memory_tracker 1/1 2024-08-20T21:58:45.7057398Z distributed/elastic/utils/logging_test 1/1 2024-08-20T21:58:45.7058019Z distributed/_composable/test_checkpoint 1/1 2024-08-20T21:58:45.7058680Z distributed/fsdp/test_distributed_checkpoint 1/1 2024-08-20T21:58:45.7059384Z distributed/_composable/fsdp/test_fully_shard_state 1/1 2024-08-20T21:58:45.7060135Z distributed/optim/test_apply_optimizer_in_backward 1/1 2024-08-20T21:58:45.7060978Z distributed/_composable/test_composability/test_pp_composability 1/1 2024-08-20T21:58:45.7061719Z distributed/elastic/timer/api_test 1/1 2024-08-20T21:58:45.7062259Z Parallel tests (0): 2024-08-20T21:58:45.7093392Z Running distributed/test_inductor_collectives 1/1 ... [2024-08-20 21:58:45.708968] 2024-08-20T21:58:45.7094230Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T21:58:45.7100465Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_inductor_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 21:58:45.709569] 2024-08-20T21:58:58.3009913Z 2024-08-20T21:58:58.3012210Z distributed/test_inductor_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_inductor_collectives_1.1_198b1b3df32a9df6_.log 2024-08-20T21:58:58.3048067Z Running 47 items in this shard: test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor_split_sizes_none, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_contiguous_input, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_into_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_output_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor_cudagraph_trees, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_input_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_broadcast_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_c10d_functional_tagged_pt2_compliant, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_eager_allreduce_inductor_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_inductor_allreduce_eager_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_permute_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_reduce_scatter_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_backwards, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source_GroupMember_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source__get_default_group, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source_group_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_graphbreaks_unsupported_async_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_pg_var, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_args_match, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_list, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_to_all_single, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_kwargs, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_kwargs_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_positional, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_positional_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_unspecified, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op0, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op1, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op2, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op3, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op4, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_reduce_scatter, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_support_collective_op_with_async_op_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor_pg, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allgather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allreduce, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_reduce_scatter_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_all_gather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_doesnt_mutate_shared, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_reduce_scatter_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_single_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_steal_buffer, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_meta 2024-08-20T21:58:58.3081228Z 2024-08-20T21:58:58.3081862Z Running distributed/test_data_parallel 1/1 ... [2024-08-20 21:58:58.301511] 2024-08-20T21:58:58.3082643Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T21:58:58.3084784Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_data_parallel.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 21:58:58.301996] 2024-08-20T21:59:05.1302222Z 2024-08-20T21:59:05.1304072Z distributed/test_data_parallel 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_data_parallel_1.1_8ac38b7594174bfb_.log 2024-08-20T21:59:05.1334758Z Running 46 items in this shard: test/distributed/test_data_parallel.py::TestDataParallel::test_autocast, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_buffers_requiring_grad, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_complex, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_device_args, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_function_deletion, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_lazy_linear, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_model_device, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_model_no_refcycles, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_module_zero_inputs, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_multiple_input, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_nested_input, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_nested_output, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_no_grad, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_rnn, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_small_back, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_sparse, test/distributed/test_data_parallel.py::TestDataParallel::test_gather_cpu, test/distributed/test_data_parallel.py::TestDataParallel::test_gather_different_len_dicts, test/distributed/test_data_parallel.py::TestDataParallel::test_gather_gpu, test/distributed/test_data_parallel.py::TestDataParallel::test_parallel_apply, test/distributed/test_data_parallel.py::TestDataParallel::test_parallel_apply_autocast, test/distributed/test_data_parallel.py::TestDataParallel::test_parallel_apply_passes_exception, test/distributed/test_data_parallel.py::TestDataParallel::test_parameter_list_dict_replica, test/distributed/test_data_parallel.py::TestDataParallel::test_replicate, test/distributed/test_data_parallel.py::TestDataParallel::test_replicate_buffers, test/distributed/test_data_parallel.py::TestDataParallel::test_save_replica_module, test/distributed/test_data_parallel.py::TestDataParallel::test_scatter_cpu, test/distributed/test_data_parallel.py::TestDataParallel::test_scatter_gpu, test/distributed/test_data_parallel.py::TestDataParallel::test_strided_grad_layout, test/distributed/test_data_parallel.py::TestDataParallel::test_zero_grad, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_dict_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_dict_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_dict_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_list_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_list_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_list_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_tuple_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_tuple_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_tuple_cuda_float64 2024-08-20T21:59:05.1363501Z 2024-08-20T21:59:05.1364238Z Running distributed/_composable/test_replicate_with_compiler 1/1 ... [2024-08-20 21:59:05.130592] 2024-08-20T21:59:05.1365148Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T21:59:05.1367473Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_replicate_with_compiler.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 21:59:05.131066] 2024-08-20T22:00:35.2780095Z 2024-08-20T22:00:35.2784859Z distributed/_composable/test_replicate_with_compiler 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_replicate_with_compiler_1.1_7aff2b5b0a111dc7_.log 2024-08-20T22:00:35.2794886Z Running 10 items in this shard: test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_bucketing_coalesced_op, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_bucketing_concat_op, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_backward_only, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_bf16, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_cpu, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_cpu_no_sync, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_fp16, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_gpu, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_gpu_ac, test/distributed/_composable/test_replicate_with_compiler.py::DDP_TP_Test::test_ddp_tp 2024-08-20T22:00:35.2802614Z 2024-08-20T22:00:35.2803397Z Running distributed/_tensor/test_dtensor_ops 1/1 ... [2024-08-20 22:00:35.278130] 2024-08-20T22:00:35.2804436Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:00:35.2807139Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:00:35.278711] 2024-08-20T22:00:40.1876255Z 2024-08-20T22:00:40.1878213Z distributed/_tensor/test_dtensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_ops_1.1_f9c3963ab3395b46_.log 2024-08-20T22:00:40.1879727Z Running 0 items in this shard: 2024-08-20T22:00:40.1880045Z 2024-08-20T22:00:40.1883666Z Running distributed/_composable/fsdp/test_fully_shard_compile 1/1 ... [2024-08-20 22:00:40.188008] 2024-08-20T22:00:40.1884560Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:00:40.1890039Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:00:40.188551] 2024-08-20T22:00:54.5300845Z 2024-08-20T22:00:54.5302997Z distributed/_composable/fsdp/test_fully_shard_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_compile_1.1_7f5a642b63bd1abc_.log 2024-08-20T22:00:54.5314229Z Running 12 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompileCompute::test_disable_compiling_hooks, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_dynamo_trace_use_training_state, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_aot_eager, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_aot_eager_decomp_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_inductor, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_simple_mlp_fullgraph_backend_aot_eager, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_simple_mlp_fullgraph_backend_aot_eager_decomp_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_simple_mlp_fullgraph_backend_inductor, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_trace_fsdp_set_, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_aot_eager, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_aot_eager_decomp_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_inductor 2024-08-20T22:00:54.5324492Z 2024-08-20T22:00:54.5325149Z Running distributed/pipelining/test_schedule_multiproc 1/1 ... [2024-08-20 22:00:54.530509] 2024-08-20T22:00:54.5326028Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:00:54.5329932Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_schedule_multiproc.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:00:54.531009] 2024-08-20T22:01:07.1697573Z 2024-08-20T22:01:07.1699997Z distributed/pipelining/test_schedule_multiproc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_schedule_multiproc_1.1_f4c0d3cf796ea20c_.log 2024-08-20T22:01:07.1713762Z Running 16 items in this shard: test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_forward_only_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass0_ModelClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass1_ModelClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_weight_update_mlp_e2e_ScheduleClass0 2024-08-20T22:01:07.1738754Z Running 16 items in this shard: test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_forward_only_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass0_ModelClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass1_ModelClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_weight_update_mlp_e2e_ScheduleClass0 2024-08-20T22:01:07.1751188Z 2024-08-20T22:01:07.1751838Z Running distributed/_tensor/test_xla_integration 1/1 ... [2024-08-20 22:01:07.170190] 2024-08-20T22:01:07.1752665Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:01:07.1754901Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_xla_integration.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:01:07.170725] 2024-08-20T22:01:10.9925755Z 2024-08-20T22:01:10.9929522Z distributed/_tensor/test_xla_integration 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_xla_integration_1.1_2cfd52c7790d739c_.log 2024-08-20T22:01:10.9936787Z Running 3 items in this shard: test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_1d_replicate, test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_1d_shard, test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_2d 2024-08-20T22:01:10.9941609Z 2024-08-20T22:01:10.9942813Z Running distributed/pipelining/test_unflatten 1/1 ... [2024-08-20 22:01:10.993171] 2024-08-20T22:01:10.9944211Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:01:10.9946515Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_unflatten.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:01:10.993848] 2024-08-20T22:01:15.3664780Z 2024-08-20T22:01:15.3666773Z distributed/pipelining/test_unflatten 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_unflatten_1.1_98580f375335cca4_.log 2024-08-20T22:01:15.3668831Z Running 1 items in this shard: test/distributed/pipelining/test_unflatten.py::UnflattenTests::test_unflatten 2024-08-20T22:01:15.3669668Z 2024-08-20T22:01:15.3671760Z Running distributed/_tensor/test_convolution_ops 1/1 ... [2024-08-20 22:01:15.366868] 2024-08-20T22:01:15.3672591Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:01:15.3677346Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_convolution_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:01:15.367332] 2024-08-20T22:01:53.9100715Z 2024-08-20T22:01:53.9102715Z distributed/_tensor/test_convolution_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_convolution_ops_1.1_48e5ae552bff8d51_.log 2024-08-20T22:01:53.9106080Z Running 2 items in this shard: test/distributed/_tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_depthwise_convolution, test/distributed/_tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_downsampling_convolution 2024-08-20T22:01:53.9107855Z 2024-08-20T22:01:53.9108462Z Running distributed/launcher/test_run 1/1 ... [2024-08-20 22:01:53.910334] 2024-08-20T22:01:53.9109232Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:01:53.9111955Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/launcher/test_run.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:01:53.910777] 2024-08-20T22:02:35.7632937Z 2024-08-20T22:02:35.7634752Z distributed/launcher/test_run 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.launcher.test_run_1.1_5c6cba50912ef819_.log 2024-08-20T22:02:35.7652599Z Running 24 items in this shard: test/distributed/launcher/test_run.py::ElasticLaunchTest::test_capture_logs_using_default_logs_specs, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_init_method_env_with_torchelastic, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_init_method_tcp_with_torchelastic, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_is_not_torchelastic_launched, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_is_torchelastic_launched, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_is_torchelastic_launched_with_logs_spec_defined, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_elastic, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_elastic_agent_raise_exception, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_elastic_multiple_agents, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_elastic_worker_raise_exception, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_run_path, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_shutdown, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_standalone, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_user_script_bash, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_user_script_default_nproc, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_user_script_python, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_user_script_python_caffe2_bc, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_with_env_vars, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_logs_logs_spec_entrypoint_must_be_defined, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_min_max_nodes_parse, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_nproc_gpu_launch_configurations, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_nproc_launch_auto_configurations, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_nproc_launch_number_configurations, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_nproc_launch_unknown_configurations 2024-08-20T22:02:35.7667307Z 2024-08-20T22:02:35.7667844Z Running distributed/test_c10d_gloo 1/1 ... [2024-08-20 22:02:35.764046] 2024-08-20T22:02:35.7668589Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:02:35.7670818Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_gloo.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:02:35.764483] 2024-08-20T22:17:46.9919438Z 2024-08-20T22:17:46.9921326Z distributed/test_c10d_gloo 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_gloo_1.1_c27234943ab0c488_.log 2024-08-20T22:17:47.0003281Z Running 120 items in this shard: test/distributed/test_c10d_gloo.py::RendezvousEnvTest::test_logging_init, test/distributed/test_c10d_gloo.py::TimeoutTest::test_default_store_timeout_gloo, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_async, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_into_tensor_coalesced, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_noncontiguous_input, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_async, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_barrier_implies_wait, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_empty_tensors, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_noncontiguous_input, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_multi_device_constructor, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_send_recv_all_to_all, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_set_gloo_pg_timeout, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_cuda_dispatched, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output_unused_param, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_cpu, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_gloo, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_register_just_once, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_sparse_gradients, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_init, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_return_type, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_find_unused_parameters_when_unused_parameters_empty, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_grad_is_view, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_static_graph, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_integer_list, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_torch_device_list, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_2gpu_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_4gpu_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module_grad_is_view, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output_with_unused_parameters, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_sharded_tensor, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_invalid_powerSGD_state, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_save_load_checkpoint, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients_grad_is_view, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward, test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_optimizer, test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_unused_parameters, test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_multi_bucket, test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_single_bucket, test/distributed/test_c10d_gloo.py::ReducerTest::test_single_dtype_single_bucket, test/distributed/test_c10d_gloo.py::CommTest::test_bool_tensors, test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cpu, test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cuda, test/distributed/test_c10d_gloo.py::CommTest::test_gloo_rank_membership, test/distributed/test_c10d_gloo.py::CommTest::test_gloo_warn_not_in_group, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_default, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_subgroup, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_default_pg_gloo, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_gloo_new_group, test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_complex, test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_mismatch, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allgather_coalesced, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_collectives, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_monitored_barrier, test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync, test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_duplicate_pg, test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_sanity_check 2024-08-20T22:17:47.0073923Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::RendezvousEnvTest::test_logging_init 2024-08-20T22:17:47.0075432Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::TimeoutTest::test_default_store_timeout_gloo 2024-08-20T22:17:47.0076891Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics 2024-08-20T22:17:47.0078361Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics_cuda 2024-08-20T22:17:47.0079859Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_checks 2024-08-20T22:17:47.0081366Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_async 2024-08-20T22:17:47.0082936Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_checks 2024-08-20T22:17:47.0084538Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_into_tensor_coalesced 2024-08-20T22:17:47.0086172Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_noncontiguous_input 2024-08-20T22:17:47.0087708Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress 2024-08-20T22:17:47.0089160Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress_cuda 2024-08-20T22:17:47.0090811Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics 2024-08-20T22:17:47.0092283Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics_cuda 2024-08-20T22:17:47.0094475Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_checks 2024-08-20T22:17:47.0097792Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_async 2024-08-20T22:17:47.0099426Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_basics 2024-08-20T22:17:47.0100999Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks 2024-08-20T22:17:47.0102597Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks_cuda 2024-08-20T22:17:47.0104195Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_stress 2024-08-20T22:17:47.0105704Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress 2024-08-20T22:17:47.0107178Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress_cuda 2024-08-20T22:17:47.0108669Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_barrier_implies_wait 2024-08-20T22:17:47.0110141Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics 2024-08-20T22:17:47.0111623Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics_cuda 2024-08-20T22:17:47.0113094Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_checks 2024-08-20T22:17:47.0114521Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress 2024-08-20T22:17:47.0116000Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress_cuda 2024-08-20T22:17:47.0117449Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_empty_tensors 2024-08-20T22:17:47.0118863Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics 2024-08-20T22:17:47.0120286Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics_cuda 2024-08-20T22:17:47.0121727Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_checks 2024-08-20T22:17:47.0123221Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_noncontiguous_input 2024-08-20T22:17:47.0124686Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress 2024-08-20T22:17:47.0126133Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress_cuda 2024-08-20T22:17:47.0127641Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_multi_device_constructor 2024-08-20T22:17:47.0129125Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics 2024-08-20T22:17:47.0130545Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics_cuda 2024-08-20T22:17:47.0131974Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_checks 2024-08-20T22:17:47.0133428Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor 2024-08-20T22:17:47.0135005Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor_coalesced 2024-08-20T22:17:47.0136675Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress 2024-08-20T22:17:47.0138161Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress_cuda 2024-08-20T22:17:47.0139605Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics 2024-08-20T22:17:47.0141122Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics_cuda 2024-08-20T22:17:47.0142588Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_checks 2024-08-20T22:17:47.0143999Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress 2024-08-20T22:17:47.0145445Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress_cuda 2024-08-20T22:17:47.0146928Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_send_recv_all_to_all 2024-08-20T22:17:47.0148406Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_set_gloo_pg_timeout 2024-08-20T22:17:47.0149908Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics 2024-08-20T22:17:47.0151490Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics_cuda 2024-08-20T22:17:47.0153037Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_checks 2024-08-20T22:17:47.0154635Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_cuda_dispatched 2024-08-20T22:17:47.0156237Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output 2024-08-20T22:17:47.0157867Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output_unused_param 2024-08-20T22:17:47.0159601Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module 2024-08-20T22:17:47.0161410Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing 2024-08-20T22:17:47.0163276Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False 2024-08-20T22:17:47.0165122Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True 2024-08-20T22:17:47.0167074Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False 2024-08-20T22:17:47.0169115Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True 2024-08-20T22:17:47.0171072Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False 2024-08-20T22:17:47.0172934Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True 2024-08-20T22:17:47.0174791Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing 2024-08-20T22:17:47.0176696Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False 2024-08-20T22:17:47.0178668Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True 2024-08-20T22:17:47.0180744Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False 2024-08-20T22:17:47.0182735Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True 2024-08-20T22:17:47.0184674Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_cpu 2024-08-20T22:17:47.0186473Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_gloo 2024-08-20T22:17:47.0188232Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_register_just_once 2024-08-20T22:17:47.0189963Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_sparse_gradients 2024-08-20T22:17:47.0191671Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_init 2024-08-20T22:17:47.0194085Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_return_type 2024-08-20T22:17:47.0195950Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_find_unused_parameters_when_unused_parameters_empty 2024-08-20T22:17:47.0197811Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad 2024-08-20T22:17:47.0199648Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_grad_is_view 2024-08-20T22:17:47.0201595Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_static_graph 2024-08-20T22:17:47.0203517Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_integer_list 2024-08-20T22:17:47.0205469Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_torch_device_list 2024-08-20T22:17:47.0207290Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_2gpu_module 2024-08-20T22:17:47.0208937Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_4gpu_module 2024-08-20T22:17:47.0210566Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module 2024-08-20T22:17:47.0212279Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module_grad_is_view 2024-08-20T22:17:47.0213944Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output 2024-08-20T22:17:47.0215612Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output_with_unused_parameters 2024-08-20T22:17:47.0217319Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_sharded_tensor 2024-08-20T22:17:47.0218946Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_invalid_powerSGD_state 2024-08-20T22:17:47.0220559Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_save_load_checkpoint 2024-08-20T22:17:47.0222122Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients 2024-08-20T22:17:47.0223756Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients_grad_is_view 2024-08-20T22:17:47.0225729Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input 2024-08-20T22:17:47.0227445Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input 2024-08-20T22:17:47.0229080Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward 2024-08-20T22:17:47.0230479Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_optimizer 2024-08-20T22:17:47.0231961Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_unused_parameters 2024-08-20T22:17:47.0233431Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_multi_bucket 2024-08-20T22:17:47.0234835Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_single_bucket 2024-08-20T22:17:47.0236275Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_single_dtype_single_bucket 2024-08-20T22:17:47.0237637Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_bool_tensors 2024-08-20T22:17:47.0238958Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cpu 2024-08-20T22:17:47.0240386Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cuda 2024-08-20T22:17:47.0241765Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_gloo_rank_membership 2024-08-20T22:17:47.0243104Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_gloo_warn_not_in_group 2024-08-20T22:17:47.0244532Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_default 2024-08-20T22:17:47.0246078Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_subgroup 2024-08-20T22:17:47.0247579Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_default_pg_gloo 2024-08-20T22:17:47.0249039Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_gloo_new_group 2024-08-20T22:17:47.0250422Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_complex 2024-08-20T22:17:47.0251763Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_mismatch 2024-08-20T22:17:47.0253391Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single 2024-08-20T22:17:47.0255264Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allgather_coalesced 2024-08-20T22:17:47.0257174Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced 2024-08-20T22:17:47.0259020Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_collectives 2024-08-20T22:17:47.0260978Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2024-08-20T22:17:47.0263056Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2024-08-20T22:17:47.0265049Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_monitored_barrier 2024-08-20T22:17:47.0266680Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync 2024-08-20T22:17:47.0268306Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_duplicate_pg 2024-08-20T22:17:47.0269840Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_sanity_check 2024-08-20T22:17:47.0270724Z 2024-08-20T22:17:47.5132850Z Running test batch 'tests to run' cost 1141.81 seconds 2024-08-20T22:17:47.9279243Z 2024-08-20T22:17:47.9280219Z real 19m7.124s 2024-08-20T22:17:47.9280662Z user 39m4.012s 2024-08-20T22:17:47.9281039Z sys 14m4.006s 2024-08-20T22:17:47.9281374Z + assert_git_not_dirty 2024-08-20T22:17:47.9282167Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-08-20T22:17:47.9282913Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *xla* ]] 2024-08-20T22:17:47.9286716Z ++ git status --porcelain 2024-08-20T22:17:47.9287241Z ++ grep -v '?? third_party' 2024-08-20T22:17:50.8870722Z ++ true 2024-08-20T22:17:50.8872030Z + git_status= 2024-08-20T22:17:50.8872701Z + [[ -n '' ]] 2024-08-20T22:17:50.8873634Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-08-20T22:17:50.8874231Z + [[ 1 == 1 ]] 2024-08-20T22:17:50.8874712Z + echo 'Testing distributed C++ tests' 2024-08-20T22:17:50.8875225Z Testing distributed C++ tests 2024-08-20T22:17:50.8880641Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T22:17:50.8895132Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T22:17:50.8906289Z + export CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T22:17:50.8907461Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T22:17:50.8908480Z + python test/run_test.py --cpp --verbose -i cpp/FileStoreTest 2024-08-20T22:17:51.0258306Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:17:51.0259729Z import pkg_resources 2024-08-20T22:17:55.4793994Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:17:55.4927880Z Found test times from artifacts 2024-08-20T22:17:55.5479780Z Found test times from artifacts 2024-08-20T22:17:55.5497214Z Running 25% of tests based on TD 2024-08-20T22:17:55.5500861Z Running parallel tests on 2 processes 2024-08-20T22:17:55.5501430Z Name: tests to run (est. time: 0.0min) 2024-08-20T22:17:55.5501940Z Serial tests (1): 2024-08-20T22:17:55.5502328Z cpp/FileStoreTest 1/1 2024-08-20T22:17:55.5502744Z Parallel tests (0): 2024-08-20T22:17:55.5503190Z Name: excluded (est. time: 0.0min) 2024-08-20T22:17:55.5503676Z Serial tests (0): 2024-08-20T22:17:55.5504045Z Parallel tests (0): 2024-08-20T22:17:55.5570693Z Running cpp/FileStoreTest 1/1 ... [2024-08-20 22:17:55.556723] 2024-08-20T22:17:55.5571406Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:17:55.5578718Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/FileStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-14e0155c163f92bf.xml', '-x', '--reruns=2'] ... [2024-08-20 22:17:55.557410] 2024-08-20T22:17:59.4305027Z 2024-08-20T22:17:59.4306868Z cpp/FileStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.FileStoreTest_1.1_a37c0d78006c6547_.log 2024-08-20T22:17:59.4308000Z 2024-08-20T22:17:59.9369539Z Running test batch 'tests to run' cost 4.39 seconds 2024-08-20T22:18:00.3613774Z + python test/run_test.py --cpp --verbose -i cpp/HashStoreTest 2024-08-20T22:18:00.4971838Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:18:00.4973310Z import pkg_resources 2024-08-20T22:18:04.9736681Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:18:04.9871204Z Found test times from artifacts 2024-08-20T22:18:05.0418358Z Found test times from artifacts 2024-08-20T22:18:05.0435238Z Running 25% of tests based on TD 2024-08-20T22:18:05.0438468Z Running parallel tests on 2 processes 2024-08-20T22:18:05.0439290Z Name: tests to run (est. time: 0.0min) 2024-08-20T22:18:05.0439851Z Serial tests (1): 2024-08-20T22:18:05.0440257Z cpp/HashStoreTest 1/1 2024-08-20T22:18:05.0440673Z Parallel tests (0): 2024-08-20T22:18:05.0441098Z Name: excluded (est. time: 0.0min) 2024-08-20T22:18:05.0441573Z Serial tests (0): 2024-08-20T22:18:05.0441961Z Parallel tests (0): 2024-08-20T22:18:05.0504099Z Running cpp/HashStoreTest 1/1 ... [2024-08-20 22:18:05.050087] 2024-08-20T22:18:05.0504782Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:18:05.0511215Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/HashStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-a6ee4ffed2c43e83.xml', '-x', '--reruns=2'] ... [2024-08-20 22:18:05.050671] 2024-08-20T22:18:08.5226459Z 2024-08-20T22:18:08.5228149Z cpp/HashStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.HashStoreTest_1.1_74536d14da227845_.log 2024-08-20T22:18:08.5229271Z 2024-08-20T22:18:09.0370616Z Running test batch 'tests to run' cost 3.99 seconds 2024-08-20T22:18:09.4596927Z + python test/run_test.py --cpp --verbose -i cpp/TCPStoreTest 2024-08-20T22:18:09.5901698Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:18:09.5903132Z import pkg_resources 2024-08-20T22:18:14.0334511Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:18:14.0468586Z Found test times from artifacts 2024-08-20T22:18:14.1015207Z Found test times from artifacts 2024-08-20T22:18:14.1031903Z Running 25% of tests based on TD 2024-08-20T22:18:14.1035106Z Running parallel tests on 2 processes 2024-08-20T22:18:14.1035699Z Name: tests to run (est. time: 0.0min) 2024-08-20T22:18:14.1036204Z Serial tests (1): 2024-08-20T22:18:14.1036589Z cpp/TCPStoreTest 1/1 2024-08-20T22:18:14.1037020Z Parallel tests (0): 2024-08-20T22:18:14.1037450Z Name: excluded (est. time: 0.0min) 2024-08-20T22:18:14.1037911Z Serial tests (0): 2024-08-20T22:18:14.1038296Z Parallel tests (0): 2024-08-20T22:18:14.1100809Z Running cpp/TCPStoreTest 1/1 ... [2024-08-20 22:18:14.109757] 2024-08-20T22:18:14.1101487Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:18:14.1107860Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/TCPStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-50c33a98d8dddec0.xml', '-x', '--reruns=2'] ... [2024-08-20 22:18:14.110352] 2024-08-20T22:18:29.9596472Z 2024-08-20T22:18:29.9598448Z cpp/TCPStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.TCPStoreTest_1.1_d2d2cb0eaefd9add_.log 2024-08-20T22:18:29.9599888Z 2024-08-20T22:18:30.4784398Z Running test batch 'tests to run' cost 16.37 seconds 2024-08-20T22:18:30.8981358Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-08-20T22:18:30.8984593Z ++ command -v mpiexec 2024-08-20T22:18:30.8989763Z + MPIEXEC=/usr/bin/mpiexec 2024-08-20T22:18:30.8990327Z + [[ -n /usr/bin/mpiexec ]] 2024-08-20T22:18:30.8991836Z + MPICMD='/usr/bin/mpiexec -np 2 /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupMPITest' 2024-08-20T22:18:30.8994201Z + eval '/usr/bin/mpiexec -np 2 /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupMPITest' 2024-08-20T22:18:30.8995860Z ++ /usr/bin/mpiexec -np 2 /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupMPITest 2024-08-20T22:18:34.1115996Z Test successful 2024-08-20T22:18:34.1167569Z Test successful 2024-08-20T22:18:34.2769232Z + python test/run_test.py --cpp --verbose -i cpp/ProcessGroupGlooTest 2024-08-20T22:18:34.4072992Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:18:34.4074431Z import pkg_resources 2024-08-20T22:18:38.8213042Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:18:38.8346943Z Found test times from artifacts 2024-08-20T22:18:38.8888532Z Found test times from artifacts 2024-08-20T22:18:38.8906043Z Running 25% of tests based on TD 2024-08-20T22:18:38.8909782Z Running parallel tests on 2 processes 2024-08-20T22:18:38.8910361Z Name: tests to run (est. time: 0.0min) 2024-08-20T22:18:38.8910856Z Serial tests (1): 2024-08-20T22:18:38.8911270Z cpp/ProcessGroupGlooTest 1/1 2024-08-20T22:18:38.8911763Z Parallel tests (0): 2024-08-20T22:18:38.8912176Z Name: excluded (est. time: 0.0min) 2024-08-20T22:18:38.8912658Z Serial tests (0): 2024-08-20T22:18:38.8913075Z Parallel tests (0): 2024-08-20T22:18:38.8973590Z Running cpp/ProcessGroupGlooTest 1/1 ... [2024-08-20 22:18:38.897021] 2024-08-20T22:18:38.8974348Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:18:38.8981137Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupGlooTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-615d683de6fcfca3.xml', '-x', '--reruns=2'] ... [2024-08-20 22:18:38.897645] 2024-08-20T22:18:48.1807991Z 2024-08-20T22:18:48.1809789Z cpp/ProcessGroupGlooTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.ProcessGroupGlooTest_1.1_08b1816f659103f7_.log 2024-08-20T22:18:48.1811019Z 2024-08-20T22:18:48.7012479Z Running test batch 'tests to run' cost 9.81 seconds 2024-08-20T22:18:49.1223274Z + python test/run_test.py --cpp --verbose -i cpp/ProcessGroupNCCLTest 2024-08-20T22:18:49.2575161Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:18:49.2576606Z import pkg_resources 2024-08-20T22:18:53.7505452Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:18:53.7638887Z Found test times from artifacts 2024-08-20T22:18:53.8183967Z Found test times from artifacts 2024-08-20T22:18:53.8201184Z Running 25% of tests based on TD 2024-08-20T22:18:53.8204109Z Running parallel tests on 2 processes 2024-08-20T22:18:53.8204814Z Name: tests to run (est. time: 0.0min) 2024-08-20T22:18:53.8205316Z Serial tests (1): 2024-08-20T22:18:53.8205767Z cpp/ProcessGroupNCCLTest 1/1 2024-08-20T22:18:53.8206246Z Parallel tests (0): 2024-08-20T22:18:53.8206681Z Name: excluded (est. time: 0.0min) 2024-08-20T22:18:53.8207135Z Serial tests (0): 2024-08-20T22:18:53.8207504Z Parallel tests (0): 2024-08-20T22:18:53.8269450Z Running cpp/ProcessGroupNCCLTest 1/1 ... [2024-08-20 22:18:53.826599] 2024-08-20T22:18:53.8270209Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:18:53.8276644Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-b7cf2c5ef4882e51.xml', '-x', '--reruns=2'] ... [2024-08-20 22:18:53.827185] 2024-08-20T22:19:19.3383934Z 2024-08-20T22:19:19.3385729Z cpp/ProcessGroupNCCLTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.ProcessGroupNCCLTest_1.1_cecc5d7b3fbcf249_.log 2024-08-20T22:19:19.3388493Z 2024-08-20T22:19:19.8588223Z Running test batch 'tests to run' cost 26.04 seconds 2024-08-20T22:19:20.2781935Z + python test/run_test.py --cpp --verbose -i cpp/ProcessGroupNCCLErrorsTest 2024-08-20T22:19:20.4102269Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:19:20.4103756Z import pkg_resources 2024-08-20T22:19:24.8339065Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:19:24.8472543Z Found test times from artifacts 2024-08-20T22:19:24.9015250Z Found test times from artifacts 2024-08-20T22:19:24.9031471Z Running 25% of tests based on TD 2024-08-20T22:19:24.9034879Z Running parallel tests on 2 processes 2024-08-20T22:19:24.9035456Z Name: tests to run (est. time: 0.0min) 2024-08-20T22:19:24.9035962Z Serial tests (1): 2024-08-20T22:19:24.9036390Z cpp/ProcessGroupNCCLErrorsTest 1/1 2024-08-20T22:19:24.9038715Z Parallel tests (0): 2024-08-20T22:19:24.9039216Z Name: excluded (est. time: 0.0min) 2024-08-20T22:19:24.9039696Z Serial tests (0): 2024-08-20T22:19:24.9040097Z Parallel tests (0): 2024-08-20T22:19:24.9098671Z Running cpp/ProcessGroupNCCLErrorsTest 1/1 ... [2024-08-20 22:19:24.909510] 2024-08-20T22:19:24.9099540Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:19:24.9105616Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLErrorsTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-83c98233bdd5be50.xml', '-x', '--reruns=2'] ... [2024-08-20 22:19:24.910114] 2024-08-20T22:19:37.4504710Z 2024-08-20T22:19:37.4506574Z cpp/ProcessGroupNCCLErrorsTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.ProcessGroupNCCLErrorsTest_1.1_b439d669bee9584f_.log 2024-08-20T22:19:37.4508866Z 2024-08-20T22:19:37.9534778Z Running test batch 'tests to run' cost 13.05 seconds 2024-08-20T22:19:38.3708454Z + [[ 1 == 1 ]] 2024-08-20T22:19:38.3708985Z + test_rpc 2024-08-20T22:19:38.3709597Z + echo 'Testing RPC C++ tests' 2024-08-20T22:19:38.3710073Z Testing RPC C++ tests 2024-08-20T22:19:38.3715462Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T22:19:38.3730684Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T22:19:38.3743380Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T22:19:38.3744796Z + python test/run_test.py --cpp --verbose -i cpp/test_cpp_rpc 2024-08-20T22:19:38.5058358Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:19:38.5059807Z import pkg_resources 2024-08-20T22:19:42.9585362Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:19:42.9719472Z Found test times from artifacts 2024-08-20T22:19:43.0269305Z Found test times from artifacts 2024-08-20T22:19:43.0285965Z Running 25% of tests based on TD 2024-08-20T22:19:43.0289157Z Running parallel tests on 2 processes 2024-08-20T22:19:43.0289745Z Name: tests to run (est. time: 0.0min) 2024-08-20T22:19:43.0290219Z Serial tests (1): 2024-08-20T22:19:43.0290617Z cpp/test_cpp_rpc 1/1 2024-08-20T22:19:43.0291043Z Parallel tests (0): 2024-08-20T22:19:43.0291472Z Name: excluded (est. time: 0.0min) 2024-08-20T22:19:43.0294374Z Serial tests (0): 2024-08-20T22:19:43.0294833Z Parallel tests (0): 2024-08-20T22:19:43.0354742Z Running cpp/test_cpp_rpc 1/1 ... [2024-08-20 22:19:43.035151] 2024-08-20T22:19:43.0355433Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:19:43.0361736Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/test_cpp_rpc', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-512eddac4c5ea4ae.xml', '-x', '--reruns=2'] ... [2024-08-20 22:19:43.035742] 2024-08-20T22:19:49.4638901Z 2024-08-20T22:19:49.4640793Z cpp/test_cpp_rpc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.test_cpp_rpc_1.1_3922cf9f8a40b9eb_.log 2024-08-20T22:19:49.4641896Z 2024-08-20T22:19:49.9883315Z Running test batch 'tests to run' cost 6.96 seconds 2024-08-20T22:19:50.4068917Z + cleanup_workspace 2024-08-20T22:19:50.4070255Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-20T22:19:50.4071635Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-20T22:19:50.4072818Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-20T22:19:50.4073601Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-20T22:19:50.4074656Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-20T22:19:50.4075780Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-20T22:19:50.4076649Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-20T22:19:51.0233833Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-20T22:19:51.0234518Z cat test/**/*_toprint.log || true 2024-08-20T22:19:51.0245651Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:19:51.0246273Z env: 2024-08-20T22:19:51.0246699Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:51.0247244Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:19:51.0248173Z DOCKER_CONTAINER_ID: 9afe50b973f3f5524a846944b5650c0e68adccbc8ed115dca00a11ff567068c4 2024-08-20T22:19:51.0248983Z ##[endgroup] 2024-08-20T22:19:51.0338828Z cat: 'test/**/*_toprint.log': No such file or directory 2024-08-20T22:19:51.0379091Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-20T22:19:51.0379706Z kill "$MONITOR_SCRIPT_PID" 2024-08-20T22:19:51.0386489Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:19:51.0387115Z env: 2024-08-20T22:19:51.0387454Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:51.0387997Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:19:51.0388926Z DOCKER_CONTAINER_ID: 9afe50b973f3f5524a846944b5650c0e68adccbc8ed115dca00a11ff567068c4 2024-08-20T22:19:51.0389764Z MONITOR_SCRIPT_PID: 379156 2024-08-20T22:19:51.0390187Z ##[endgroup] 2024-08-20T22:19:51.0634073Z Prepare all required actions 2024-08-20T22:19:51.0634861Z Getting action download info 2024-08-20T22:19:51.2939826Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-20T22:19:51.4592380Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-20T22:19:51.4593540Z with: 2024-08-20T22:19:51.4594199Z file-suffix: test-distributed-1-3-amz2023.linux.8xlarge.nvidia.gpu_29025850427 2024-08-20T22:19:51.4595025Z s3-bucket: gha-artifacts 2024-08-20T22:19:51.4595417Z env: 2024-08-20T22:19:51.4595744Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:51.4596295Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:19:51.4597198Z DOCKER_CONTAINER_ID: 9afe50b973f3f5524a846944b5650c0e68adccbc8ed115dca00a11ff567068c4 2024-08-20T22:19:51.4598007Z ##[endgroup] 2024-08-20T22:19:51.4632517Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-20T22:19:51.4634671Z # Remove any previous test jsons if they exist 2024-08-20T22:19:51.4635307Z rm -f test-jsons-*.zip 2024-08-20T22:19:51.4635955Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-20T22:19:51.4642681Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:19:51.4643329Z env: 2024-08-20T22:19:51.4643664Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:51.4644221Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:19:51.4645144Z DOCKER_CONTAINER_ID: 9afe50b973f3f5524a846944b5650c0e68adccbc8ed115dca00a11ff567068c4 2024-08-20T22:19:51.4646260Z FILE_SUFFIX: test-distributed-1-3-amz2023.linux.8xlarge.nvidia.gpu_29025850427 2024-08-20T22:19:51.4647104Z ##[endgroup] 2024-08-20T22:19:51.4908769Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-20T22:19:51.4943521Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-20T22:19:51.4944371Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-20T22:19:51.4951881Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-20T22:19:51.4958527Z adding: test/slow_tests.json (deflated 82%) 2024-08-20T22:19:51.4961423Z adding: test/test-reports/td_exclusions-4715b68bbb6f0125675e.json (deflated 86%) 2024-08-20T22:19:51.4962629Z adding: test/test-reports/td_exclusions-b6168c3719b08052adb0.json (deflated 15%) 2024-08-20T22:19:51.4963795Z adding: test/test-reports/td_exclusions-0ded5e5e8c005e26ba19.json (deflated 14%) 2024-08-20T22:19:51.4964993Z adding: test/test-reports/td_exclusions-44c9fe9ba873dbc7c3ba.json (deflated 14%) 2024-08-20T22:19:51.4966185Z adding: test/test-reports/td_exclusions-93c9a9a47836a91dbe84.json (deflated 12%) 2024-08-20T22:19:51.4967361Z adding: test/test-reports/td_exclusions-b01e2eafc7dc30ed590e.json (deflated 12%) 2024-08-20T22:19:51.4968552Z adding: test/test-reports/td_exclusions-3068bcb6594d522b4429.json (deflated 11%) 2024-08-20T22:19:51.4969740Z adding: test/test-reports/td_exclusions-dc7b36636ffe49d20030.json (deflated 18%) 2024-08-20T22:19:51.4977340Z adding: test/.pytorch-disabled-tests.json (deflated 88%) 2024-08-20T22:19:51.5013345Z ##[group]Run # Remove any previous test reports if they exist 2024-08-20T22:19:51.5014156Z # Remove any previous test reports if they exist 2024-08-20T22:19:51.5014808Z rm -f test-reports-*.zip 2024-08-20T22:19:51.5015537Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-20T22:19:51.5022417Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:19:51.5023034Z env: 2024-08-20T22:19:51.5023373Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:51.5023913Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:19:51.5024838Z DOCKER_CONTAINER_ID: 9afe50b973f3f5524a846944b5650c0e68adccbc8ed115dca00a11ff567068c4 2024-08-20T22:19:51.5025970Z FILE_SUFFIX: test-distributed-1-3-amz2023.linux.8xlarge.nvidia.gpu_29025850427 2024-08-20T22:19:51.5026749Z ##[endgroup] 2024-08-20T22:19:51.5268482Z adding: test/test-reports/python-pytest/distributed.test_inductor_collectives/distributed.test_inductor_collectives-14a28b0f03e6f371.xml (deflated 89%) 2024-08-20T22:19:51.5270829Z adding: test/test-reports/python-pytest/distributed.test_data_parallel/distributed.test_data_parallel-e231aa7220493244.xml (deflated 85%) 2024-08-20T22:19:51.5273149Z adding: test/test-reports/python-pytest/distributed._composable.test_replicate_with_compiler/distributed._composable.test_replicate_with_compiler-bd80e10f11125359.xml (deflated 84%) 2024-08-20T22:19:51.5275487Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor_ops/distributed._tensor.test_dtensor_ops-f9d747bdec1721ca.xml (deflated 28%) 2024-08-20T22:19:51.5277883Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_compile/distributed._composable.fsdp.test_fully_shard_compile-c3d360f41bf0ec72.xml (deflated 88%) 2024-08-20T22:19:51.5280452Z adding: test/test-reports/python-pytest/distributed.pipelining.test_schedule_multiproc/distributed.pipelining.test_schedule_multiproc-a016a5fc796cc22a.xml (deflated 84%) 2024-08-20T22:19:51.5282943Z adding: test/test-reports/python-pytest/distributed.pipelining.test_schedule_multiproc/distributed.pipelining.test_schedule_multiproc-2f8b88e19b4d58ac.xml (deflated 84%) 2024-08-20T22:19:51.5285323Z adding: test/test-reports/python-pytest/distributed._tensor.test_xla_integration/distributed._tensor.test_xla_integration-46a5891428c142f8.xml (deflated 74%) 2024-08-20T22:19:51.5287542Z adding: test/test-reports/python-pytest/distributed.pipelining.test_unflatten/distributed.pipelining.test_unflatten-6d120581489080c7.xml (deflated 41%) 2024-08-20T22:19:51.5289789Z adding: test/test-reports/python-pytest/distributed._tensor.test_convolution_ops/distributed._tensor.test_convolution_ops-a51a0c546e343433.xml (deflated 50%) 2024-08-20T22:19:51.5291910Z adding: test/test-reports/python-pytest/distributed.launcher.test_run/distributed.launcher.test_run-9d97c63124fe3b75.xml (deflated 84%) 2024-08-20T22:19:51.5294516Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-0c10f7a01066811c.xml (deflated 36%) 2024-08-20T22:19:51.5296398Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-f8748d2c71594a3b.xml (deflated 38%) 2024-08-20T22:19:51.5298271Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-d474a09e8989f601.xml (deflated 36%) 2024-08-20T22:19:51.5300190Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-31aaddf316ba0597.xml (deflated 36%) 2024-08-20T22:19:51.5302069Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-ee90c98ce5ffcbb7.xml (deflated 36%) 2024-08-20T22:19:51.5303948Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-66f387b555fa8e7a.xml (deflated 36%) 2024-08-20T22:19:51.5305815Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-ac31c1ef63d22421.xml (deflated 36%) 2024-08-20T22:19:51.5307695Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-666c7c967855eef0.xml (deflated 36%) 2024-08-20T22:19:51.5309564Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-3df9285ffa470b9e.xml (deflated 36%) 2024-08-20T22:19:51.5311436Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2b2e90cd852cc6f8.xml (deflated 36%) 2024-08-20T22:19:51.5313290Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-af384ac8ae2e7305.xml (deflated 36%) 2024-08-20T22:19:51.5315166Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-3a927bf08158a796.xml (deflated 36%) 2024-08-20T22:19:51.5317202Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-f194960c63655c5b.xml (deflated 36%) 2024-08-20T22:19:51.5319210Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-bc38bd3fbb9d3155.xml (deflated 36%) 2024-08-20T22:19:51.5321068Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-a5094ff744052e78.xml (deflated 36%) 2024-08-20T22:19:51.5322926Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2829c1029719ae62.xml (deflated 36%) 2024-08-20T22:19:51.5324783Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-6147009a11882a1b.xml (deflated 36%) 2024-08-20T22:19:51.5326625Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-6e5933853c452271.xml (deflated 36%) 2024-08-20T22:19:51.5328494Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-3cac02ea4fdfcea2.xml (deflated 36%) 2024-08-20T22:19:51.5330381Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2e405e211a86ad8b.xml (deflated 36%) 2024-08-20T22:19:51.5332258Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7c9a90c16255dbf7.xml (deflated 36%) 2024-08-20T22:19:51.5334108Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2e45db9307f053e3.xml (deflated 36%) 2024-08-20T22:19:51.5335977Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-5b75df98bf2c851d.xml (deflated 37%) 2024-08-20T22:19:51.5337849Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-94d41a7ad530df36.xml (deflated 37%) 2024-08-20T22:19:51.5339721Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-8499ba36d1a0ec3a.xml (deflated 36%) 2024-08-20T22:19:51.5341576Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-4e2dbc0b338360ed.xml (deflated 37%) 2024-08-20T22:19:51.5343459Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-d6bad6ba3fa81d6a.xml (deflated 36%) 2024-08-20T22:19:51.5345601Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-d63f900619eb4e49.xml (deflated 36%) 2024-08-20T22:19:51.5347528Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-6a199e2baa83c600.xml (deflated 36%) 2024-08-20T22:19:51.5349382Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-c1d5a4314fa86a9f.xml (deflated 36%) 2024-08-20T22:19:51.5351250Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-73316b4bf351f2bc.xml (deflated 36%) 2024-08-20T22:19:51.5353127Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-cb3872cab6802bed.xml (deflated 36%) 2024-08-20T22:19:51.5355010Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-d77bf5b1a11ccc72.xml (deflated 36%) 2024-08-20T22:19:51.5356864Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-df6d983946805e62.xml (deflated 36%) 2024-08-20T22:19:51.5358724Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7989c4c80bced8a8.xml (deflated 36%) 2024-08-20T22:19:51.5360584Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-6fc2c14ae1141306.xml (deflated 36%) 2024-08-20T22:19:51.5362443Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-0620ba95893f54d4.xml (deflated 36%) 2024-08-20T22:19:51.5364298Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-efd1e1a31592e19e.xml (deflated 36%) 2024-08-20T22:19:51.5366269Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-bed2f445a7cb4110.xml (deflated 36%) 2024-08-20T22:19:51.5368236Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-18bf6f4e316f0343.xml (deflated 36%) 2024-08-20T22:19:51.5370084Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-64bb8538a6d1f5f9.xml (deflated 36%) 2024-08-20T22:19:51.5371957Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-dc416fbc5ab6e6ec.xml (deflated 36%) 2024-08-20T22:19:51.5373826Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2efa169514f06cf9.xml (deflated 36%) 2024-08-20T22:19:51.5375685Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-6364254670ff70a6.xml (deflated 36%) 2024-08-20T22:19:51.5377529Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-61343574ccb87dcc.xml (deflated 36%) 2024-08-20T22:19:51.5379411Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-36fe1988b8585713.xml (deflated 36%) 2024-08-20T22:19:51.5381266Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2aa083d50516726f.xml (deflated 47%) 2024-08-20T22:19:51.5383132Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-ed4dbfc19afd77a8.xml (deflated 36%) 2024-08-20T22:19:51.5385051Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-6bf52d421679ad9f.xml (deflated 59%) 2024-08-20T22:19:51.5386888Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-5e1b91437640d7d1.xml (deflated 36%) 2024-08-20T22:19:51.5388751Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-47263323ee7935d7.xml (deflated 36%) 2024-08-20T22:19:51.5390625Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-d85d7be3f6937b60.xml (deflated 36%) 2024-08-20T22:19:51.5392913Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2d74f28f6e8ac11e.xml (deflated 36%) 2024-08-20T22:19:51.5394952Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-cf952dbafd420d7b.xml (deflated 38%) 2024-08-20T22:19:51.5396826Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-82acbba4d79713d8.xml (deflated 38%) 2024-08-20T22:19:51.5398694Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-06d0d99c590c409b.xml (deflated 36%) 2024-08-20T22:19:51.5400548Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2cb0266d91a0ab9f.xml (deflated 36%) 2024-08-20T22:19:51.5402416Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-b6ced80fe62f4d20.xml (deflated 36%) 2024-08-20T22:19:51.5404291Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-60b0d25c64de39c5.xml (deflated 36%) 2024-08-20T22:19:51.5406157Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-d6302c1257f3842d.xml (deflated 36%) 2024-08-20T22:19:51.5408019Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-8b199cbcd1e4d9f8.xml (deflated 36%) 2024-08-20T22:19:51.5409864Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-3d96f73e9d09e7d7.xml (deflated 36%) 2024-08-20T22:19:51.5411725Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9ba61dfee6f70b47.xml (deflated 36%) 2024-08-20T22:19:51.5413745Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-313051320a55ec8c.xml (deflated 36%) 2024-08-20T22:19:51.5415757Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-3dbce36e36b17cc1.xml (deflated 37%) 2024-08-20T22:19:51.5417610Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-62c86e77031ab297.xml (deflated 37%) 2024-08-20T22:19:51.5419474Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9fb26afd22e25153.xml (deflated 36%) 2024-08-20T22:19:51.5421341Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-cc88b4a74732a1dd.xml (deflated 36%) 2024-08-20T22:19:51.5423185Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-b23196560953d4a0.xml (deflated 36%) 2024-08-20T22:19:51.5425062Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-e799b0939cf99206.xml (deflated 37%) 2024-08-20T22:19:51.5426933Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2e80a0caeebb1bb5.xml (deflated 37%) 2024-08-20T22:19:51.5428807Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-dc9c9075de8a01e4.xml (deflated 36%) 2024-08-20T22:19:51.5430647Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-60453956503c8a64.xml (deflated 37%) 2024-08-20T22:19:51.5432508Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-5d9a23aeae2649c1.xml (deflated 36%) 2024-08-20T22:19:51.5434381Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-1b33d7a3cab56627.xml (deflated 38%) 2024-08-20T22:19:51.5436247Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-8c0ab4ab2bf8ed7f.xml (deflated 37%) 2024-08-20T22:19:51.5438106Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-4b6ac4e847326798.xml (deflated 37%) 2024-08-20T22:19:51.5439973Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-3653570cd9ab1ff5.xml (deflated 37%) 2024-08-20T22:19:51.5441838Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-483e1aa7c816112b.xml (deflated 37%) 2024-08-20T22:19:51.5443698Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-f593d0ec9236e5c7.xml (deflated 37%) 2024-08-20T22:19:51.5445557Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-583b195ad764d0c6.xml (deflated 50%) 2024-08-20T22:19:51.5447474Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-06520c86eea7b786.xml (deflated 50%) 2024-08-20T22:19:51.5449342Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-a4f66381685f2952.xml (deflated 37%) 2024-08-20T22:19:51.5451208Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-4ff053b903665b0d.xml (deflated 36%) 2024-08-20T22:19:51.5453058Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9df976af38a5f12f.xml (deflated 37%) 2024-08-20T22:19:51.5454921Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-f94691621798dde8.xml (deflated 37%) 2024-08-20T22:19:51.5456782Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-5841e850f461086c.xml (deflated 37%) 2024-08-20T22:19:51.5458629Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-a3ca419b82cc27a0.xml (deflated 36%) 2024-08-20T22:19:51.5460502Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-b6bf6e0e23e40232.xml (deflated 36%) 2024-08-20T22:19:51.5462456Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-6bb2d53a65ab6929.xml (deflated 37%) 2024-08-20T22:19:51.5464419Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-c74d081eda72173d.xml (deflated 37%) 2024-08-20T22:19:51.5466264Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-71b77ca2f0643494.xml (deflated 36%) 2024-08-20T22:19:51.5468130Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-1ab9884d450966c2.xml (deflated 36%) 2024-08-20T22:19:51.5469998Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-d1d6ce73a6a70fa4.xml (deflated 36%) 2024-08-20T22:19:51.5471869Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-b27051b22a5b90e3.xml (deflated 35%) 2024-08-20T22:19:51.5473723Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2a1e9e17b8715cee.xml (deflated 36%) 2024-08-20T22:19:51.5475604Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-b332e06cd1cb95dd.xml (deflated 36%) 2024-08-20T22:19:51.5477470Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-bada8077736c14f0.xml (deflated 36%) 2024-08-20T22:19:51.5479344Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-b699c29f031e2dc3.xml (deflated 37%) 2024-08-20T22:19:51.5481192Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7174cb29de4acb4d.xml (deflated 36%) 2024-08-20T22:19:51.5483056Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-b82f202e48d1700e.xml (deflated 37%) 2024-08-20T22:19:51.5484938Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-f4ade3d8095aa08c.xml (deflated 37%) 2024-08-20T22:19:51.5486813Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-cec5919e30d8e423.xml (deflated 36%) 2024-08-20T22:19:51.5488677Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-dcf68fa4f797a861.xml (deflated 36%) 2024-08-20T22:19:51.5490552Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-fbd812a7d231a93f.xml (deflated 37%) 2024-08-20T22:19:51.5492424Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-b4df324d6be3da01.xml (deflated 49%) 2024-08-20T22:19:51.5494927Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-dc054b9382a28c27.xml (deflated 36%) 2024-08-20T22:19:51.5496784Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-b9b78d5279f56fa0.xml (deflated 36%) 2024-08-20T22:19:51.5498653Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-530abb905e048feb.xml (deflated 36%) 2024-08-20T22:19:51.5524788Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-2663aff0006e5cda.xml (deflated 36%) 2024-08-20T22:19:51.5526675Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-7d6a2d286531b066.xml (deflated 35%) 2024-08-20T22:19:51.5528546Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-bea196bd6791d8ca.xml (deflated 35%) 2024-08-20T22:19:51.5530409Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-c6fa81dfe8761345.xml (deflated 35%) 2024-08-20T22:19:51.5532240Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-5093693898f9bb21.xml (deflated 36%) 2024-08-20T22:19:51.5534630Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9311cf8bff361138.xml (deflated 35%) 2024-08-20T22:19:51.5536639Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-c0aee9f860627ee0.xml (deflated 36%) 2024-08-20T22:19:51.5538490Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-b0a1ab2bba2bd955.xml (deflated 35%) 2024-08-20T22:19:51.5540348Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-9faa67acc02ad646.xml (deflated 35%) 2024-08-20T22:19:51.5542213Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-11defdcc2818050a.xml (deflated 35%) 2024-08-20T22:19:51.5544079Z adding: test/test-reports/python-pytest/distributed.test_c10d_gloo/distributed.test_c10d_gloo-450ba1898c073fb9.xml (deflated 35%) 2024-08-20T22:19:51.5545746Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-14e0155c163f92bf.xml (deflated 60%) 2024-08-20T22:19:51.5547317Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-a6ee4ffed2c43e83.xml (deflated 60%) 2024-08-20T22:19:51.5548821Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-50c33a98d8dddec0.xml (deflated 69%) 2024-08-20T22:19:51.5550293Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-615d683de6fcfca3.xml (deflated 78%) 2024-08-20T22:19:51.5551785Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-b7cf2c5ef4882e51.xml (deflated 74%) 2024-08-20T22:19:51.5553292Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-83c98233bdd5be50.xml (deflated 76%) 2024-08-20T22:19:51.5554791Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-512eddac4c5ea4ae.xml (deflated 64%) 2024-08-20T22:19:51.5587529Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-20T22:19:51.5588303Z # Remove any previous usage logs if they exist 2024-08-20T22:19:51.5588908Z rm -f logs-*.zip 2024-08-20T22:19:51.5589741Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-20T22:19:51.5590709Z # so check to see if the file exists first 2024-08-20T22:19:51.5591303Z if [ -f 'usage_log.txt' ]; then 2024-08-20T22:19:51.5591938Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-20T22:19:51.5592926Z fi 2024-08-20T22:19:51.5593548Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-20T22:19:51.5594256Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-20T22:19:51.5594850Z fi 2024-08-20T22:19:51.5601319Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:19:51.5601929Z env: 2024-08-20T22:19:51.5602258Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:51.5602796Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:19:51.5603721Z DOCKER_CONTAINER_ID: 9afe50b973f3f5524a846944b5650c0e68adccbc8ed115dca00a11ff567068c4 2024-08-20T22:19:51.5604862Z FILE_SUFFIX: test-distributed-1-3-amz2023.linux.8xlarge.nvidia.gpu_29025850427 2024-08-20T22:19:51.5605657Z ##[endgroup] 2024-08-20T22:19:51.5669665Z adding: usage_log.txt (deflated 92%) 2024-08-20T22:19:51.5919542Z adding: test/test-reports/distributed.test_inductor_collectives_1.1_198b1b3df32a9df6_.log (deflated 87%) 2024-08-20T22:19:51.5921203Z adding: test/test-reports/distributed.test_data_parallel_1.1_8ac38b7594174bfb_.log (deflated 88%) 2024-08-20T22:19:51.5923187Z adding: test/test-reports/distributed._composable.test_replicate_with_compiler_1.1_7aff2b5b0a111dc7_.log (deflated 83%) 2024-08-20T22:19:51.5924830Z adding: test/test-reports/distributed._tensor.test_dtensor_ops_1.1_f9c3963ab3395b46_.log (deflated 52%) 2024-08-20T22:19:51.5926435Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_compile_1.1_7f5a642b63bd1abc_.log (deflated 82%) 2024-08-20T22:19:51.5928358Z adding: test/test-reports/distributed.pipelining.test_schedule_multiproc_1.1_f4c0d3cf796ea20c_.log (deflated 90%) 2024-08-20T22:19:51.5930104Z adding: test/test-reports/distributed._tensor.test_xla_integration_1.1_2cfd52c7790d739c_.log (deflated 66%) 2024-08-20T22:19:51.5931625Z adding: test/test-reports/distributed.pipelining.test_unflatten_1.1_98580f375335cca4_.log (deflated 54%) 2024-08-20T22:19:51.5933142Z adding: test/test-reports/distributed._tensor.test_convolution_ops_1.1_48e5ae552bff8d51_.log (deflated 59%) 2024-08-20T22:19:51.5934608Z adding: test/test-reports/distributed.launcher.test_run_1.1_5c6cba50912ef819_.log (deflated 88%) 2024-08-20T22:19:51.5956111Z adding: test/test-reports/distributed.test_c10d_gloo_1.1_c27234943ab0c488_.log (deflated 96%) 2024-08-20T22:19:51.5957414Z adding: test/test-reports/cpp.FileStoreTest_1.1_a37c0d78006c6547_.log (deflated 72%) 2024-08-20T22:19:51.5958614Z adding: test/test-reports/cpp.HashStoreTest_1.1_74536d14da227845_.log (deflated 71%) 2024-08-20T22:19:51.5959854Z adding: test/test-reports/cpp.TCPStoreTest_1.1_d2d2cb0eaefd9add_.log (deflated 79%) 2024-08-20T22:19:51.5961180Z adding: test/test-reports/cpp.ProcessGroupGlooTest_1.1_08b1816f659103f7_.log (deflated 84%) 2024-08-20T22:19:51.5962488Z adding: test/test-reports/cpp.ProcessGroupNCCLTest_1.1_cecc5d7b3fbcf249_.log (deflated 82%) 2024-08-20T22:19:51.5963893Z adding: test/test-reports/cpp.ProcessGroupNCCLErrorsTest_1.1_b439d669bee9584f_.log (deflated 78%) 2024-08-20T22:19:51.5965205Z adding: test/test-reports/cpp.test_cpp_rpc_1.1_3922cf9f8a40b9eb_.log (deflated 79%) 2024-08-20T22:19:51.5998463Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-20T22:19:51.5999355Z # Remove any previous debugging artifacts if they exist 2024-08-20T22:19:51.6000034Z rm -f debug-*.zip 2024-08-20T22:19:51.6000496Z if [ -d 'test/debug' ]; then 2024-08-20T22:19:51.6001096Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-20T22:19:51.6001670Z fi 2024-08-20T22:19:51.6008243Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:19:51.6008869Z env: 2024-08-20T22:19:51.6009205Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:51.6009747Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:19:51.6010668Z DOCKER_CONTAINER_ID: 9afe50b973f3f5524a846944b5650c0e68adccbc8ed115dca00a11ff567068c4 2024-08-20T22:19:51.6011801Z FILE_SUFFIX: test-distributed-1-3-amz2023.linux.8xlarge.nvidia.gpu_29025850427 2024-08-20T22:19:51.6012598Z ##[endgroup] 2024-08-20T22:19:51.6116868Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:19:51.6117389Z with: 2024-08-20T22:19:51.6117723Z s3-bucket: gha-artifacts 2024-08-20T22:19:51.6118249Z s3-prefix: pytorch/pytorch/10479309237/1/artifact 2024-08-20T22:19:51.6118810Z retention-days: 14 2024-08-20T22:19:51.6119209Z if-no-files-found: warn 2024-08-20T22:19:51.6119636Z path: test-jsons-*.zip 2024-08-20T22:19:51.6120025Z name: artifact 2024-08-20T22:19:51.6120386Z region: us-east-1 2024-08-20T22:19:51.6120751Z env: 2024-08-20T22:19:51.6121068Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:51.6121617Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:19:51.6122534Z DOCKER_CONTAINER_ID: 9afe50b973f3f5524a846944b5650c0e68adccbc8ed115dca00a11ff567068c4 2024-08-20T22:19:51.6123318Z ##[endgroup] 2024-08-20T22:19:52.0538761Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T22:19:52.0539684Z With the provided path, there will be 1 file uploaded 2024-08-20T22:19:52.0540462Z Uploading to s3 prefix: pytorch/pytorch/10479309237/1/artifact 2024-08-20T22:19:52.0593495Z Starting upload of test-jsons-test-distributed-1-3-amz2023.linux.8xlarge.nvidia.gpu_29025850427.zip 2024-08-20T22:19:52.2591456Z Finished upload of test-jsons-test-distributed-1-3-amz2023.linux.8xlarge.nvidia.gpu_29025850427.zip 2024-08-20T22:19:52.2790068Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:19:52.2790606Z with: 2024-08-20T22:19:52.2790947Z s3-bucket: gha-artifacts 2024-08-20T22:19:52.2791695Z s3-prefix: pytorch/pytorch/10479309237/1/artifact 2024-08-20T22:19:52.2792437Z retention-days: 14 2024-08-20T22:19:52.2793197Z if-no-files-found: error 2024-08-20T22:19:52.2793644Z path: test-reports-*.zip 2024-08-20T22:19:52.2794051Z name: artifact 2024-08-20T22:19:52.2794414Z region: us-east-1 2024-08-20T22:19:52.2794775Z env: 2024-08-20T22:19:52.2795084Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:52.2795637Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:19:52.2796560Z DOCKER_CONTAINER_ID: 9afe50b973f3f5524a846944b5650c0e68adccbc8ed115dca00a11ff567068c4 2024-08-20T22:19:52.2797351Z ##[endgroup] 2024-08-20T22:19:52.7259874Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T22:19:52.7260819Z With the provided path, there will be 1 file uploaded 2024-08-20T22:19:52.7261572Z Uploading to s3 prefix: pytorch/pytorch/10479309237/1/artifact 2024-08-20T22:19:52.7313817Z Starting upload of test-reports-test-distributed-1-3-amz2023.linux.8xlarge.nvidia.gpu_29025850427.zip 2024-08-20T22:19:52.8361463Z Finished upload of test-reports-test-distributed-1-3-amz2023.linux.8xlarge.nvidia.gpu_29025850427.zip 2024-08-20T22:19:52.8568262Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:19:52.8568799Z with: 2024-08-20T22:19:52.8569143Z s3-bucket: gha-artifacts 2024-08-20T22:19:52.8569682Z s3-prefix: pytorch/pytorch/10479309237/1/artifact 2024-08-20T22:19:52.8570250Z retention-days: 14 2024-08-20T22:19:52.8570668Z if-no-files-found: ignore 2024-08-20T22:19:52.8571112Z path: logs-*.zip 2024-08-20T22:19:52.8571471Z name: artifact 2024-08-20T22:19:52.8571838Z region: us-east-1 2024-08-20T22:19:52.8572201Z env: 2024-08-20T22:19:52.8572514Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:52.8573073Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:19:52.8574005Z DOCKER_CONTAINER_ID: 9afe50b973f3f5524a846944b5650c0e68adccbc8ed115dca00a11ff567068c4 2024-08-20T22:19:52.8574801Z ##[endgroup] 2024-08-20T22:19:53.2999285Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T22:19:53.3000222Z With the provided path, there will be 1 file uploaded 2024-08-20T22:19:53.3001000Z Uploading to s3 prefix: pytorch/pytorch/10479309237/1/artifact 2024-08-20T22:19:53.3052446Z Starting upload of logs-test-distributed-1-3-amz2023.linux.8xlarge.nvidia.gpu_29025850427.zip 2024-08-20T22:19:53.4162456Z Finished upload of logs-test-distributed-1-3-amz2023.linux.8xlarge.nvidia.gpu_29025850427.zip 2024-08-20T22:19:53.4356187Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:19:53.4356699Z with: 2024-08-20T22:19:53.4357041Z s3-bucket: gha-artifacts 2024-08-20T22:19:53.4357564Z s3-prefix: pytorch/pytorch/10479309237/1/artifact 2024-08-20T22:19:53.4358120Z retention-days: 14 2024-08-20T22:19:53.4358526Z if-no-files-found: ignore 2024-08-20T22:19:53.4358959Z path: debug-*.zip 2024-08-20T22:19:53.4359316Z name: artifact 2024-08-20T22:19:53.4359683Z region: us-east-1 2024-08-20T22:19:53.4360041Z env: 2024-08-20T22:19:53.4360360Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:53.4360932Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:19:53.4361851Z DOCKER_CONTAINER_ID: 9afe50b973f3f5524a846944b5650c0e68adccbc8ed115dca00a11ff567068c4 2024-08-20T22:19:53.4362637Z ##[endgroup] 2024-08-20T22:19:53.8683495Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-20T22:19:53.8873543Z ##[group]Run # shellcheck disable=SC2156 2024-08-20T22:19:53.8874169Z # shellcheck disable=SC2156 2024-08-20T22:19:53.8875173Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-20T22:19:53.8882746Z shell: /usr/bin/bash -e {0} 2024-08-20T22:19:53.8883177Z env: 2024-08-20T22:19:53.8883506Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:53.8884045Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:19:53.8884977Z DOCKER_CONTAINER_ID: 9afe50b973f3f5524a846944b5650c0e68adccbc8ed115dca00a11ff567068c4 2024-08-20T22:19:53.8885779Z ##[endgroup] 2024-08-20T22:19:54.1679238Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-20T22:19:54.1679948Z with: 2024-08-20T22:19:54.1680257Z env: 2024-08-20T22:19:54.1680590Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:54.1681135Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:19:54.1682066Z DOCKER_CONTAINER_ID: 9afe50b973f3f5524a846944b5650c0e68adccbc8ed115dca00a11ff567068c4 2024-08-20T22:19:54.1682879Z ##[endgroup] 2024-08-20T22:19:54.1709244Z ##[group]Run set -eou pipefail 2024-08-20T22:19:54.1709748Z set -eou pipefail 2024-08-20T22:19:54.1710175Z  2024-08-20T22:19:54.1710801Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-20T22:19:54.1711574Z for _ in $(seq 1440); do 2024-08-20T22:19:54.1712154Z  # Break if no ssh session exists anymore 2024-08-20T22:19:54.1712768Z  if [ "$(who)" = "" ]; then 2024-08-20T22:19:54.1713249Z  break 2024-08-20T22:19:54.1713704Z  fi 2024-08-20T22:19:54.1714057Z  echo "." 2024-08-20T22:19:54.1714455Z  sleep 5 2024-08-20T22:19:54.1714836Z done 2024-08-20T22:19:54.1721712Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:19:54.1722330Z env: 2024-08-20T22:19:54.1722666Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:54.1723208Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:19:54.1724137Z DOCKER_CONTAINER_ID: 9afe50b973f3f5524a846944b5650c0e68adccbc8ed115dca00a11ff567068c4 2024-08-20T22:19:54.1724948Z ##[endgroup] 2024-08-20T22:19:54.1753792Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-20T22:19:54.1829992Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:19:54.1830944Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:19:54.1831668Z # shellcheck disable=SC2046 2024-08-20T22:19:54.1832235Z docker stop $(docker ps -q) || true 2024-08-20T22:19:54.1832816Z # Prune all of the docker images 2024-08-20T22:19:54.1833364Z docker system prune -af 2024-08-20T22:19:54.1840083Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:19:54.1840677Z env: 2024-08-20T22:19:54.1841006Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:54.1841558Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:19:54.1842460Z DOCKER_CONTAINER_ID: 9afe50b973f3f5524a846944b5650c0e68adccbc8ed115dca00a11ff567068c4 2024-08-20T22:19:54.1843263Z ##[endgroup] 2024-08-20T22:19:54.9167104Z 9afe50b973f3 2024-08-20T22:19:55.4272519Z Deleted Containers: 2024-08-20T22:19:55.4273232Z 9afe50b973f3f5524a846944b5650c0e68adccbc8ed115dca00a11ff567068c4 2024-08-20T22:19:55.4273790Z 2024-08-20T22:20:01.5260544Z Deleted Images: 2024-08-20T22:20:01.5262321Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:20:01.5264882Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9@sha256:0b3aa0836d4f18656c98dcf355b46cbae2123243424ebbed0929206c3d798126 2024-08-20T22:20:01.5266651Z deleted: sha256:35f4945dfeab4592c8657de7107f96a9fd9a51999f582a0064937246789c0825 2024-08-20T22:20:01.5267689Z deleted: sha256:5aeec7923eaf143c8c860346742d9e9c0a627a72c06c4a8574f03d7359497d8a 2024-08-20T22:20:01.5268729Z deleted: sha256:8c2e3c359e6f8f863100522d9503272ad2a22af4d3245cb32988cf7f75dff9bf 2024-08-20T22:20:01.5269771Z deleted: sha256:b9c8bef934615ea09715897fa2252d4ff9a33ae4b0b99d65f4314edb98a08508 2024-08-20T22:20:01.5270786Z deleted: sha256:7325e907409be83e224427131a7e02928ab8ca8880e53b975b309a17c9887b3d 2024-08-20T22:20:01.5271824Z deleted: sha256:93c3c0295ffeb06e6b45fd31a03c7b9de49d6a6404a889643dcb02a29956a46d 2024-08-20T22:20:01.5272871Z deleted: sha256:75434bb2c96c820ddef9e3cd15729787375a35257f82c07e9a2e49a63ccd258b 2024-08-20T22:20:01.5274287Z deleted: sha256:2b55c8d634372339b274a7068c1e8058ff9ca9b828c020d97462adddfbbab68e 2024-08-20T22:20:01.5275532Z deleted: sha256:f4242713ef5734c93522c188d00d66c11c2ea30e1edc576f7557feec4a71fb44 2024-08-20T22:20:01.5276562Z deleted: sha256:e3f0028f9a27ad44915c7ce616833089a5ac41163b1f3fd441c29448c9967bd6 2024-08-20T22:20:01.5277634Z deleted: sha256:157d6992f3a76ca9422961a443176f6c953d6d2609d8f26abc95e283177d8838 2024-08-20T22:20:01.5278674Z deleted: sha256:80725860ab404ed4a3e2d4f1c9179efb9628ddaaead438f0e2dfd45860821659 2024-08-20T22:20:01.5279729Z deleted: sha256:12612899d16f7cab2f6bcce2c8c2ce9a6aabe0f8c57d6a9ec3df95aebc24eb66 2024-08-20T22:20:01.5280794Z deleted: sha256:f892c2afba839178fe4f4480ffa99124c5788c76959cef3c751d31ffbc56532e 2024-08-20T22:20:01.5281838Z deleted: sha256:3783df37fa2f17df24597d0c9b48925712f7ec5cdb75993e04a750249242d945 2024-08-20T22:20:01.5282851Z deleted: sha256:03700104a8d899077ca0cf2dcca4902d103f736f5605fc2dcb1228b4fbb01cf8 2024-08-20T22:20:01.5283895Z deleted: sha256:c22e6b22d7d1206421c03a90e2a0d202a5b0293700ea39051d3062ab4e02212f 2024-08-20T22:20:01.5284951Z deleted: sha256:6a5fd513d78286d2a97e6118bdfaf09d345513de1afe00f7214930aab4e182ce 2024-08-20T22:20:01.5286112Z deleted: sha256:12d6046ca585372ab7221c75bf1ff9fbad344c584bfac3bf48490124569b09ce 2024-08-20T22:20:01.5287784Z deleted: sha256:11a342b5bdf2902b76dc6b741fa09aeada64dc62a4634c30e0b92dd49ecaa3e6 2024-08-20T22:20:01.5288840Z deleted: sha256:9c19056af41bd6f7d0316312696e9fb6ec9e16b9b2916bebb487ed1302846fca 2024-08-20T22:20:01.5289892Z deleted: sha256:dff9ad772a710e9d7a74de2fb2e60c8d87630855c5a75c7348b20d817de188c9 2024-08-20T22:20:01.5290922Z deleted: sha256:21911a7fa8e0ff3a30aa732134f3d24cb0e339bc36725f9980a7687e520e356a 2024-08-20T22:20:01.5291967Z deleted: sha256:f40a21fc1dad1aa6a566142c92a08afe87864068e6c360b6ade83ec521803286 2024-08-20T22:20:01.5293669Z deleted: sha256:617a6a309e07100390ac08e5ea2680a47e1f79fa72907e8b08ff46309a46f1ba 2024-08-20T22:20:01.5294717Z deleted: sha256:8d26fabf19fef49bc3a331f43fdaa661678b87d53109fc7540bfb9b44ce810bf 2024-08-20T22:20:01.5295790Z deleted: sha256:a2d242ed225d8f09315e3ee4597c3feb02f088337f59c95ddf440784d2be4f52 2024-08-20T22:20:01.5296832Z deleted: sha256:e10875c7907612ac7bd831be770f133f6eccaa347118b765c5d9ec776512f0b2 2024-08-20T22:20:01.5297865Z deleted: sha256:b5fa471f84592d75a2cf8625dcf9a285b9d76a7e7edccff81d385b66a46f9532 2024-08-20T22:20:01.5298919Z deleted: sha256:e42f988c8d3d938d4a1e3975407f64e4f962f3cabc1122a8fe98828492e34657 2024-08-20T22:20:01.5299950Z deleted: sha256:9507391f7f3611eb7c6f95061219d63ee3ecfd4cc81b3de44126c6d3c4667551 2024-08-20T22:20:01.5300973Z deleted: sha256:5e72605653a900048cf413ca242bb695bc9932595763521ed44f8fbe7166166a 2024-08-20T22:20:01.5302015Z deleted: sha256:435005331c11e4210ce1b81221a7054aee7ccaeb202bb2e09b45f2510f6ab254 2024-08-20T22:20:01.5303047Z deleted: sha256:a2f04d898ab3e490e7247db9696a9726e1f1f85012b46115a89aedab38930710 2024-08-20T22:20:01.5304065Z deleted: sha256:07b864569ca9924353696ae1560e08d337c4867cb398ce0e552e04c37d4cf6e5 2024-08-20T22:20:01.5305079Z deleted: sha256:907fa71a16fcb35456269eb13d0128ce7763243eaf3449c44cdd36b5b49312e1 2024-08-20T22:20:01.5306137Z deleted: sha256:f79790dfbd9f1c7e3fc5d45ffd290c41176ec11dffaf7a8c63c2c58b797fcdb6 2024-08-20T22:20:01.5307183Z deleted: sha256:406d461e2aa914e38d6eef5723bb633966284960223c93a97cd00be8a1cc0faf 2024-08-20T22:20:01.5308197Z deleted: sha256:119a4673c9c39860d87ebbc4b72e29b12aa2f3242fabcf48cb24a5c8cfcd1704 2024-08-20T22:20:01.5309232Z deleted: sha256:624417752cf64b2297d09ab2e429cc9c87774134cc1b34bf2e5c2b79cd34ecc4 2024-08-20T22:20:01.5310277Z deleted: sha256:2bcfc72dea6a6bc48e2290c98fa34228cfe20302176ff01ce6b84971d811079a 2024-08-20T22:20:01.5311342Z deleted: sha256:d4aee3900a65d7e4f9a427d9bae22e836280b85f289fd5b80adc4ed2cd3aff53 2024-08-20T22:20:01.5312364Z deleted: sha256:2a255798af4e419932e17c11a82527d26470216c9bc57ab8807a735dbd2c93ae 2024-08-20T22:20:01.5313387Z deleted: sha256:fa19e724b62cc0bbe219419c371baf2471725f88a55292672c1aa24358db4a9b 2024-08-20T22:20:01.5314645Z deleted: sha256:9c5a64873ca39cb9da2e93ad3ec1eb31211c8c088c3a90be1541ec5ac891ac4e 2024-08-20T22:20:01.5315806Z deleted: sha256:8421b1b55b2087ef547bb0133e57f5859efb3c9429d011de45772b63c923dd90 2024-08-20T22:20:01.5316849Z deleted: sha256:66066e4af5674b8211099bfebb5a5ab99b3d9181fcfc1dc67b54a0eff5ded07a 2024-08-20T22:20:01.5317902Z deleted: sha256:dad58a2cd6e9c8234f343c2e5ca4cd2a317b729f8048049d34151b15c83b0dae 2024-08-20T22:20:01.5318932Z deleted: sha256:3bc136283d22fe55b48989019c8484e37c075a7a98fae0a5b7f827623cb2fd5b 2024-08-20T22:20:01.5319973Z deleted: sha256:2543cfb072205917a252ea669db6e7fe93ac6bf424262d6d67b6bf24c1edfe55 2024-08-20T22:20:01.5321009Z deleted: sha256:a0abc6809e0930ad53392ef3443966cb3f6bc7645d3e1968f2d18cb0648d1240 2024-08-20T22:20:01.5322052Z deleted: sha256:0068bc6311b72b957fcc8bec156923eef76c142d408a73e201ff797dcf126f5a 2024-08-20T22:20:01.5323085Z deleted: sha256:3bb2c93e3ca7da8a708900f677778ff0abe6bcb9666d9480bdad78acd6793a59 2024-08-20T22:20:01.5324180Z deleted: sha256:d3a2bdf80e0add5b55bffcb4cf9bd55c993adad4fb66a4ec8864e0d8a118bb0a 2024-08-20T22:20:01.5325250Z deleted: sha256:c021994940e99e271b90c5bf290f821f73a14768aa34bfae3755c04685c88251 2024-08-20T22:20:01.5326248Z deleted: sha256:688ff5904c74620c1b2bd34c392425856fe1f759f94bf0419d051997c4fb23ad 2024-08-20T22:20:01.5327283Z deleted: sha256:d397bdc2172f15f11a5f57d5f57064aee1b038d71c3ca9e967ebf47dcbd5dec0 2024-08-20T22:20:01.5328335Z deleted: sha256:a70ac2bc18340667dfc9895cf91a48cc3c2b5a6c2acdc898bae02810b111c7d7 2024-08-20T22:20:01.5329376Z deleted: sha256:0fbec293f1fa0d21f414351f21bd86a114facc2f3bc74715393f4f33c9b2c89e 2024-08-20T22:20:01.5330425Z deleted: sha256:4a26099a629bbf706fb951dcb1381af8361bf5f3e736e82ea839a4a27788d0b3 2024-08-20T22:20:01.5331467Z deleted: sha256:b6284de21252ed91fd7dea71b02196645a8e239d6188ecd9f080d9b4a2026b77 2024-08-20T22:20:01.5332510Z deleted: sha256:adca9f30c5b25b7904016386497a34e3699d12b61be1cabda8703a4f6f51adaf 2024-08-20T22:20:01.5333518Z deleted: sha256:b1007278b864200bed03e3738e6a554a7913b41e9246c86925126b02f19d2512 2024-08-20T22:20:01.5334562Z deleted: sha256:ac9c37335cbc79df52a9de95491fe9ffa89b9fe730c29153ae1d8c93a046bda9 2024-08-20T22:20:01.5335629Z deleted: sha256:d0e212b9d05b634be5d1abfbfcc4fd090277954332f2b8e519c695a979bfd0dd 2024-08-20T22:20:01.5336677Z deleted: sha256:abcbe99d7f9c9df7e3c594587238e23d28f96a5fc92c2a9dd1d183ff7af4800c 2024-08-20T22:20:01.5337727Z deleted: sha256:7740288981d55c53dc1d781eb5a229ccbd5c81c56584c4d4217d7fbb512de04b 2024-08-20T22:20:01.5338756Z deleted: sha256:020ed2c4861bb111c010888e7c4ca3b2c1125b9d7b8a29148de859f786ecea66 2024-08-20T22:20:01.5339762Z deleted: sha256:89edf8c906678c217b9a88160e1561991f53909c3fd27a4a5e1f0ca0f7ff7888 2024-08-20T22:20:01.5340814Z deleted: sha256:d5ba67f8f9b0c2b6da06d2d5d9d5fedc9f17cbe1a5e13a1a4033a4619f4528ba 2024-08-20T22:20:01.5341860Z deleted: sha256:5d61311e1a293c0c9642e34871719525dfeec7028e4b45e08344a469e3fe24bf 2024-08-20T22:20:01.5342888Z deleted: sha256:b89bb70f745a94c0edd5e974aee294e160735f76fd666a1a06a0b9b1be3caf63 2024-08-20T22:20:01.5343928Z deleted: sha256:9e5477f1f21a38c3dfe21a56a7d77d1b1a42453aed3e11729afcbb9d5731770b 2024-08-20T22:20:01.5345005Z deleted: sha256:b76f4af91cffe2bb7b3f09960f5bdbb26f741419defda65c451c4407fc5b0a92 2024-08-20T22:20:01.5346061Z deleted: sha256:3f8748a00ebd394c0e9d9d481972d764dff58896841ef0150b3f7efaac0f4a4a 2024-08-20T22:20:01.5347080Z deleted: sha256:13989c771e3ba4d40f618344e55e5cb7062a35041b5efd69e2f740ba819f46ee 2024-08-20T22:20:01.5348179Z deleted: sha256:9ebb27837f6c67ad8c17e9af4e63c008faae03fa226c3641dd8bbbfeefb106c4 2024-08-20T22:20:01.5349222Z deleted: sha256:8c16907d735950f8093582bc3cf392b70e002456bf9c4224de5a5b86b78b3ea2 2024-08-20T22:20:01.5350229Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-08-20T22:20:01.5350874Z 2024-08-20T22:20:01.5351055Z Total reclaimed space: 34.72GB 2024-08-20T22:20:01.5439933Z Post job cleanup. 2024-08-20T22:20:01.5507275Z Post job cleanup. 2024-08-20T22:20:01.6682992Z [command]/usr/bin/git version 2024-08-20T22:20:01.6743680Z git version 2.40.1 2024-08-20T22:20:01.6804956Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/2d32a0d6-9d69-485e-aca7-d32e38e12dc4' before making global git config changes 2024-08-20T22:20:01.6806792Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T22:20:01.6812557Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:20:01.6857343Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T22:20:01.6899611Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T22:20:01.7252312Z Entering 'android/libs/fbjni' 2024-08-20T22:20:01.7317383Z Entering 'third_party/FP16' 2024-08-20T22:20:01.7382040Z Entering 'third_party/FXdiv' 2024-08-20T22:20:01.7447216Z Entering 'third_party/NNPACK' 2024-08-20T22:20:01.7510669Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:20:01.7574554Z Entering 'third_party/XNNPACK' 2024-08-20T22:20:01.7657671Z Entering 'third_party/benchmark' 2024-08-20T22:20:01.7723325Z Entering 'third_party/cpp-httplib' 2024-08-20T22:20:01.7786258Z Entering 'third_party/cpuinfo' 2024-08-20T22:20:01.7851990Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:20:01.7916720Z Entering 'third_party/cutlass' 2024-08-20T22:20:01.7989898Z Entering 'third_party/eigen' 2024-08-20T22:20:01.8057063Z Entering 'third_party/fbgemm' 2024-08-20T22:20:01.8121964Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:20:01.8184490Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:20:01.8247750Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:20:01.8319526Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:20:01.8381386Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:20:01.8445909Z Entering 'third_party/flatbuffers' 2024-08-20T22:20:01.8514985Z Entering 'third_party/fmt' 2024-08-20T22:20:01.8578555Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:20:01.8643015Z Entering 'third_party/gloo' 2024-08-20T22:20:01.8707191Z Entering 'third_party/googletest' 2024-08-20T22:20:01.8771036Z Entering 'third_party/ideep' 2024-08-20T22:20:01.8834537Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:20:01.8906402Z Entering 'third_party/ittapi' 2024-08-20T22:20:01.8970528Z Entering 'third_party/kineto' 2024-08-20T22:20:01.9034357Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:20:01.9098211Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:20:01.9163038Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:20:01.9226634Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:20:01.9289260Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:20:01.9351924Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:20:01.9416928Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:20:01.9480603Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:20:01.9544920Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:20:01.9610015Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:20:01.9675269Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:20:01.9739820Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:20:01.9805675Z Entering 'third_party/mimalloc' 2024-08-20T22:20:01.9868834Z Entering 'third_party/nccl/nccl' 2024-08-20T22:20:01.9933473Z Entering 'third_party/nlohmann' 2024-08-20T22:20:01.9999485Z Entering 'third_party/onnx' 2024-08-20T22:20:02.0080371Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:20:02.0145512Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:20:02.0211158Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:20:02.0274632Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:20:02.0336999Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:20:02.0400181Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:20:02.0462202Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:20:02.0526904Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:20:02.0588545Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:20:02.0651333Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:20:02.0713566Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:20:02.0777445Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:20:02.0843181Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:20:02.0929572Z Entering 'third_party/pocketfft' 2024-08-20T22:20:02.0992069Z Entering 'third_party/protobuf' 2024-08-20T22:20:02.1058772Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:20:02.1121311Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:20:02.1185322Z Entering 'third_party/psimd' 2024-08-20T22:20:02.1248957Z Entering 'third_party/pthreadpool' 2024-08-20T22:20:02.1311352Z Entering 'third_party/pybind11' 2024-08-20T22:20:02.1374715Z Entering 'third_party/python-peachpy' 2024-08-20T22:20:02.1438786Z Entering 'third_party/sleef' 2024-08-20T22:20:02.1502634Z Entering 'third_party/tensorpipe' 2024-08-20T22:20:02.1564563Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:20:02.1628281Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:20:02.1689478Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:20:02.1753166Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:20:02.1814902Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:20:02.1901173Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T22:20:02.1931408Z http.https://github.com/.extraheader 2024-08-20T22:20:02.1942907Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-20T22:20:02.1984507Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T22:20:02.2326298Z Entering 'android/libs/fbjni' 2024-08-20T22:20:02.2367775Z http.https://github.com/.extraheader 2024-08-20T22:20:02.2406634Z Entering 'third_party/FP16' 2024-08-20T22:20:02.2450125Z http.https://github.com/.extraheader 2024-08-20T22:20:02.2488398Z Entering 'third_party/FXdiv' 2024-08-20T22:20:02.2531133Z http.https://github.com/.extraheader 2024-08-20T22:20:02.2568683Z Entering 'third_party/NNPACK' 2024-08-20T22:20:02.2615462Z http.https://github.com/.extraheader 2024-08-20T22:20:02.2653260Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:20:02.2696476Z http.https://github.com/.extraheader 2024-08-20T22:20:02.2734607Z Entering 'third_party/XNNPACK' 2024-08-20T22:20:02.2777064Z http.https://github.com/.extraheader 2024-08-20T22:20:02.2835145Z Entering 'third_party/benchmark' 2024-08-20T22:20:02.2877249Z http.https://github.com/.extraheader 2024-08-20T22:20:02.2917545Z Entering 'third_party/cpp-httplib' 2024-08-20T22:20:02.2960147Z http.https://github.com/.extraheader 2024-08-20T22:20:02.2998257Z Entering 'third_party/cpuinfo' 2024-08-20T22:20:02.3041325Z http.https://github.com/.extraheader 2024-08-20T22:20:02.3080098Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:20:02.3122928Z http.https://github.com/.extraheader 2024-08-20T22:20:02.3161080Z Entering 'third_party/cutlass' 2024-08-20T22:20:02.3204262Z http.https://github.com/.extraheader 2024-08-20T22:20:02.3252003Z Entering 'third_party/eigen' 2024-08-20T22:20:02.3295765Z http.https://github.com/.extraheader 2024-08-20T22:20:02.3336233Z Entering 'third_party/fbgemm' 2024-08-20T22:20:02.3378728Z http.https://github.com/.extraheader 2024-08-20T22:20:02.3418041Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:20:02.3459910Z http.https://github.com/.extraheader 2024-08-20T22:20:02.3498691Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:20:02.3540565Z http.https://github.com/.extraheader 2024-08-20T22:20:02.3578643Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:20:02.3621693Z http.https://github.com/.extraheader 2024-08-20T22:20:02.3667255Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:20:02.3710702Z http.https://github.com/.extraheader 2024-08-20T22:20:02.3748507Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:20:02.3790285Z http.https://github.com/.extraheader 2024-08-20T22:20:02.3829403Z Entering 'third_party/flatbuffers' 2024-08-20T22:20:02.3872283Z http.https://github.com/.extraheader 2024-08-20T22:20:02.3914488Z Entering 'third_party/fmt' 2024-08-20T22:20:02.3956722Z http.https://github.com/.extraheader 2024-08-20T22:20:02.3995119Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:20:02.4038034Z http.https://github.com/.extraheader 2024-08-20T22:20:02.4076489Z Entering 'third_party/gloo' 2024-08-20T22:20:02.4119913Z http.https://github.com/.extraheader 2024-08-20T22:20:02.4157891Z Entering 'third_party/googletest' 2024-08-20T22:20:02.4201134Z http.https://github.com/.extraheader 2024-08-20T22:20:02.4239293Z Entering 'third_party/ideep' 2024-08-20T22:20:02.4282501Z http.https://github.com/.extraheader 2024-08-20T22:20:02.4324281Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:20:02.4365351Z http.https://github.com/.extraheader 2024-08-20T22:20:02.4411823Z Entering 'third_party/ittapi' 2024-08-20T22:20:02.4453721Z http.https://github.com/.extraheader 2024-08-20T22:20:02.4491215Z Entering 'third_party/kineto' 2024-08-20T22:20:02.4534599Z http.https://github.com/.extraheader 2024-08-20T22:20:02.4572344Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:20:02.4614808Z http.https://github.com/.extraheader 2024-08-20T22:20:02.4653721Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:20:02.4696863Z http.https://github.com/.extraheader 2024-08-20T22:20:02.4738162Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:20:02.4779781Z http.https://github.com/.extraheader 2024-08-20T22:20:02.4818988Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:20:02.4861118Z http.https://github.com/.extraheader 2024-08-20T22:20:02.4901074Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:20:02.4942613Z http.https://github.com/.extraheader 2024-08-20T22:20:02.4980355Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:20:02.5022875Z http.https://github.com/.extraheader 2024-08-20T22:20:02.5063424Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:20:02.5106076Z http.https://github.com/.extraheader 2024-08-20T22:20:02.5144917Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:20:02.5186374Z http.https://github.com/.extraheader 2024-08-20T22:20:02.5225715Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:20:02.5267820Z http.https://github.com/.extraheader 2024-08-20T22:20:02.5308419Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:20:02.5349947Z http.https://github.com/.extraheader 2024-08-20T22:20:02.5389973Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:20:02.5432390Z http.https://github.com/.extraheader 2024-08-20T22:20:02.5470244Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:20:02.5511771Z http.https://github.com/.extraheader 2024-08-20T22:20:02.5551068Z Entering 'third_party/mimalloc' 2024-08-20T22:20:02.5593845Z http.https://github.com/.extraheader 2024-08-20T22:20:02.5631681Z Entering 'third_party/nccl/nccl' 2024-08-20T22:20:02.5674977Z http.https://github.com/.extraheader 2024-08-20T22:20:02.5712990Z Entering 'third_party/nlohmann' 2024-08-20T22:20:02.5755734Z http.https://github.com/.extraheader 2024-08-20T22:20:02.5794981Z Entering 'third_party/onnx' 2024-08-20T22:20:02.5837296Z http.https://github.com/.extraheader 2024-08-20T22:20:02.5893530Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:20:02.5935860Z http.https://github.com/.extraheader 2024-08-20T22:20:02.5973851Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:20:02.6016554Z http.https://github.com/.extraheader 2024-08-20T22:20:02.6056328Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:20:02.6099450Z http.https://github.com/.extraheader 2024-08-20T22:20:02.6139212Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:20:02.6180883Z http.https://github.com/.extraheader 2024-08-20T22:20:02.6219439Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:20:02.6260562Z http.https://github.com/.extraheader 2024-08-20T22:20:02.6298623Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:20:02.6340971Z http.https://github.com/.extraheader 2024-08-20T22:20:02.6378414Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:20:02.6421114Z http.https://github.com/.extraheader 2024-08-20T22:20:02.6460756Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:20:02.6502539Z http.https://github.com/.extraheader 2024-08-20T22:20:02.6540444Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:20:02.6581657Z http.https://github.com/.extraheader 2024-08-20T22:20:02.6620377Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:20:02.6661995Z http.https://github.com/.extraheader 2024-08-20T22:20:02.6700605Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:20:02.6742172Z http.https://github.com/.extraheader 2024-08-20T22:20:02.6782825Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:20:02.6824956Z http.https://github.com/.extraheader 2024-08-20T22:20:02.6864442Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:20:02.6907177Z http.https://github.com/.extraheader 2024-08-20T22:20:02.6968971Z Entering 'third_party/pocketfft' 2024-08-20T22:20:02.7011434Z http.https://github.com/.extraheader 2024-08-20T22:20:02.7048918Z Entering 'third_party/protobuf' 2024-08-20T22:20:02.7091270Z http.https://github.com/.extraheader 2024-08-20T22:20:02.7133162Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:20:02.7174690Z http.https://github.com/.extraheader 2024-08-20T22:20:02.7212922Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:20:02.7255701Z http.https://github.com/.extraheader 2024-08-20T22:20:02.7296727Z Entering 'third_party/psimd' 2024-08-20T22:20:02.7338210Z http.https://github.com/.extraheader 2024-08-20T22:20:02.7375330Z Entering 'third_party/pthreadpool' 2024-08-20T22:20:02.7418889Z http.https://github.com/.extraheader 2024-08-20T22:20:02.7456570Z Entering 'third_party/pybind11' 2024-08-20T22:20:02.7500535Z http.https://github.com/.extraheader 2024-08-20T22:20:02.7538692Z Entering 'third_party/python-peachpy' 2024-08-20T22:20:02.7581294Z http.https://github.com/.extraheader 2024-08-20T22:20:02.7619825Z Entering 'third_party/sleef' 2024-08-20T22:20:02.7662116Z http.https://github.com/.extraheader 2024-08-20T22:20:02.7700247Z Entering 'third_party/tensorpipe' 2024-08-20T22:20:02.7742921Z http.https://github.com/.extraheader 2024-08-20T22:20:02.7781518Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:20:02.7824294Z http.https://github.com/.extraheader 2024-08-20T22:20:02.7861726Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:20:02.7904721Z http.https://github.com/.extraheader 2024-08-20T22:20:02.7942279Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:20:02.7984051Z http.https://github.com/.extraheader 2024-08-20T22:20:02.8025125Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:20:02.8066969Z http.https://github.com/.extraheader 2024-08-20T22:20:02.8104709Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:20:02.8147639Z http.https://github.com/.extraheader 2024-08-20T22:20:02.8295097Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-20T22:20:02.8320218Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-20T22:20:02.8326572Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:02.8327211Z ##[endgroup] 2024-08-20T22:20:10.9295083Z Cleaning up orphan processes