2024-06-26T05:21:23.8133714Z Current runner version: '2.317.0' 2024-06-26T05:21:23.8139679Z Runner name: 'i-0a8c115e2cb171c32' 2024-06-26T05:21:23.8140452Z Runner group name: 'Default' 2024-06-26T05:21:23.8141328Z Machine name: 'ip-10-0-37-235' 2024-06-26T05:21:23.8146177Z ##[group]GITHUB_TOKEN Permissions 2024-06-26T05:21:23.8148151Z Actions: read 2024-06-26T05:21:23.8148704Z Attestations: read 2024-06-26T05:21:23.8149167Z Checks: read 2024-06-26T05:21:23.8149634Z Contents: read 2024-06-26T05:21:23.8150109Z Deployments: read 2024-06-26T05:21:23.8150595Z Discussions: read 2024-06-26T05:21:23.8151064Z Issues: read 2024-06-26T05:21:23.8151561Z Metadata: read 2024-06-26T05:21:23.8152023Z Packages: read 2024-06-26T05:21:23.8152481Z Pages: read 2024-06-26T05:21:23.8152945Z PullRequests: read 2024-06-26T05:21:23.8153464Z RepositoryProjects: read 2024-06-26T05:21:23.8154086Z SecurityEvents: read 2024-06-26T05:21:23.8154558Z Statuses: read 2024-06-26T05:21:23.8155016Z ##[endgroup] 2024-06-26T05:21:23.8158200Z Secret source: Actions 2024-06-26T05:21:23.8158970Z Prepare workflow directory 2024-06-26T05:21:24.1879388Z Prepare all required actions 2024-06-26T05:21:24.2058229Z Getting action download info 2024-06-26T05:21:24.3746070Z Download action repository 'pytorch/test-infra@main' (SHA:43a2ce341cc31288e9a38b65ce600a7f43021bd5) 2024-06-26T05:21:24.7411764Z Download action repository 'pytorch/pytorch@main' (SHA:6181e65cd81725efc6bc5d64ef3be607b0aa3ca1) 2024-06-26T05:21:27.4487536Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-06-26T05:21:27.5940986Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-06-26T05:21:27.8175859Z Getting action download info 2024-06-26T05:21:27.9052378Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-06-26T05:21:28.0815517Z Getting action download info 2024-06-26T05:21:28.1740476Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-06-26T05:21:28.3150820Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/129470 (b8c4c54d347aa776934c60784e35936878ef18dc) 2024-06-26T05:21:28.3153019Z ##[group] Inputs 2024-06-26T05:21:28.3153533Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T05:21:28.3162338Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T05:21:28.3172023Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T05:21:28.3173245Z sync-tag: 2024-06-26T05:21:28.3174128Z timeout-minutes: 240 2024-06-26T05:21:28.3174489Z use-gha: 2024-06-26T05:21:28.3174810Z dashboard-tag: 2024-06-26T05:21:28.3175170Z s3-bucket: gha-artifacts 2024-06-26T05:21:28.3175541Z aws-role-to-assume: 2024-06-26T05:21:28.3175897Z ##[endgroup] 2024-06-26T05:21:28.3176813Z Complete job name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu) 2024-06-26T05:21:28.3862922Z A job started hook has been configured by the self-hosted runner administrator 2024-06-26T05:21:28.4004514Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T05:21:28.4015191Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:21:28.4015743Z ##[endgroup] 2024-06-26T05:21:29.0334847Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-06-26T05:21:29.0335459Z with: 2024-06-26T05:21:29.0336245Z github-secret: *** 2024-06-26T05:21:29.0337241Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-06-26T05:21:29.0338271Z activate-with-label: false 2024-06-26T05:21:29.0338654Z label: with-ssh 2024-06-26T05:21:29.0338999Z remove-existing-keys: true 2024-06-26T05:21:29.0339381Z fail-silently: true 2024-06-26T05:21:29.0339716Z env: 2024-06-26T05:21:29.0340024Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:21:29.0340381Z ##[endgroup] 2024-06-26T05:21:29.1228245Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-06-26T05:21:29.1229304Z ciflow reference detected, attempting to extract PR number 2024-06-26T05:21:29.3897806Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-06-26T05:21:29.4482307Z No SSH keys found for user pytorch-bot[bot] 2024-06-26T05:21:29.4483039Z Grabbing public ssh keys from https://github.com/leslie-fang-intel.keys 2024-06-26T05:21:29.5160551Z No SSH keys found for user leslie-fang-intel 2024-06-26T05:21:29.5448388Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-06-26T05:21:29.5448989Z with: 2024-06-26T05:21:29.5449287Z submodules: recursive 2024-06-26T05:21:29.5449645Z fetch-depth: 0 2024-06-26T05:21:29.5449988Z env: 2024-06-26T05:21:29.5450301Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:21:29.5450699Z ##[endgroup] 2024-06-26T05:21:29.5649513Z ##[group]Run retry () { 2024-06-26T05:21:29.5649948Z retry () { 2024-06-26T05:21:29.5650486Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-06-26T05:21:29.5651060Z } 2024-06-26T05:21:29.5651392Z echo "${GITHUB_WORKSPACE}" 2024-06-26T05:21:29.5651830Z if [ -z "${NO_SUDO}" ]; then 2024-06-26T05:21:29.5652336Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T05:21:29.5652814Z else 2024-06-26T05:21:29.5653177Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T05:21:29.5653619Z fi 2024-06-26T05:21:29.5653986Z mkdir "${GITHUB_WORKSPACE}" 2024-06-26T05:21:29.5662102Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:21:29.5662666Z env: 2024-06-26T05:21:29.5662969Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:21:29.5663359Z NO_SUDO: 2024-06-26T05:21:29.5663649Z ##[endgroup] 2024-06-26T05:21:29.5686278Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T05:21:32.4893913Z ##[group]Run malfet/checkout@silent-checkout 2024-06-26T05:21:32.4894371Z with: 2024-06-26T05:21:32.4894710Z ref: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T05:21:32.4895171Z fetch-depth: 0 2024-06-26T05:21:32.4895507Z submodules: recursive 2024-06-26T05:21:32.4895860Z quiet-checkout: true 2024-06-26T05:21:32.4896236Z repository: pytorch/pytorch 2024-06-26T05:21:32.4896740Z token: *** 2024-06-26T05:21:32.4897052Z ssh-strict: true 2024-06-26T05:21:32.4897392Z persist-credentials: true 2024-06-26T05:21:32.4897771Z clean: true 2024-06-26T05:21:32.4898127Z sparse-checkout-cone-mode: true 2024-06-26T05:21:32.4898540Z lfs: false 2024-06-26T05:21:32.4898854Z set-safe-directory: true 2024-06-26T05:21:32.4899222Z env: 2024-06-26T05:21:32.4899507Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:21:32.4899861Z ##[endgroup] 2024-06-26T05:21:32.5839366Z Syncing repository: pytorch/pytorch 2024-06-26T05:21:32.5840935Z ##[group]Getting Git version info 2024-06-26T05:21:32.5841718Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T05:21:32.5842602Z [command]/usr/bin/git version 2024-06-26T05:21:32.5842997Z git version 2.40.1 2024-06-26T05:21:32.5844377Z ##[endgroup] 2024-06-26T05:21:32.5857257Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/684c89ef-7ecd-4657-bce1-5a3b8c19e447' before making global git config changes 2024-06-26T05:21:32.5858545Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T05:21:32.5862549Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T05:21:32.5898001Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T05:21:32.5901885Z ##[group]Initializing the repository 2024-06-26T05:21:32.5905055Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T05:21:32.5929799Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-06-26T05:21:32.5930820Z hint: is subject to change. To configure the initial branch name to use in all 2024-06-26T05:21:32.5931653Z hint: of your new repositories, which will suppress this warning, call: 2024-06-26T05:21:32.5932246Z hint: 2024-06-26T05:21:32.5932881Z hint: git config --global init.defaultBranch 2024-06-26T05:21:32.5933367Z hint: 2024-06-26T05:21:32.5933897Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-06-26T05:21:32.5934760Z hint: 'development'. The just-created branch can be renamed via this command: 2024-06-26T05:21:32.5935556Z hint: 2024-06-26T05:21:32.5935890Z hint: git branch -m 2024-06-26T05:21:32.5936654Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-06-26T05:21:32.5938695Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-06-26T05:21:32.5959282Z ##[endgroup] 2024-06-26T05:21:32.5959887Z ##[group]Disabling automatic garbage collection 2024-06-26T05:21:32.5961937Z [command]/usr/bin/git config --local gc.auto 0 2024-06-26T05:21:32.5987070Z ##[endgroup] 2024-06-26T05:21:32.5987623Z ##[group]Setting up auth 2024-06-26T05:21:32.5993013Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T05:21:32.6018900Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T05:21:32.6233400Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T05:21:32.6259362Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T05:21:32.6477882Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T05:21:32.6519191Z ##[endgroup] 2024-06-26T05:21:32.6519791Z ##[group]Fetching the repository 2024-06-26T05:21:32.6525493Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-06-26T05:21:34.7913936Z remote: Enumerating objects: 987778 2024-06-26T05:21:34.7914513Z remote: Enumerating objects: 990060, done. 2024-06-26T05:21:34.7917293Z remote: Counting objects: 0% (1/2282) 2024-06-26T05:21:34.7917991Z remote: Counting objects: 1% (23/2282) 2024-06-26T05:21:34.7918650Z remote: Counting objects: 2% (46/2282) 2024-06-26T05:21:34.7919528Z remote: Counting objects: 3% (69/2282) 2024-06-26T05:21:34.7920165Z remote: Counting objects: 4% (92/2282) 2024-06-26T05:21:34.7920793Z remote: Counting objects: 5% (115/2282) 2024-06-26T05:21:34.7921322Z remote: Counting objects: 6% (137/2282) 2024-06-26T05:21:34.7921865Z remote: Counting objects: 7% (160/2282) 2024-06-26T05:21:34.7923293Z remote: Counting objects: 8% (183/2282) 2024-06-26T05:21:34.7924010Z remote: Counting objects: 9% (206/2282) 2024-06-26T05:21:34.7924528Z remote: Counting objects: 10% (229/2282) 2024-06-26T05:21:34.7925150Z remote: Counting objects: 11% (252/2282) 2024-06-26T05:21:34.7925849Z remote: Counting objects: 12% (274/2282) 2024-06-26T05:21:34.7926485Z remote: Counting objects: 13% (297/2282) 2024-06-26T05:21:34.7927010Z remote: Counting objects: 14% (320/2282) 2024-06-26T05:21:34.7929454Z remote: Counting objects: 15% (343/2282) 2024-06-26T05:21:34.7930172Z remote: Counting objects: 16% (366/2282) 2024-06-26T05:21:34.7930823Z remote: Counting objects: 17% (388/2282) 2024-06-26T05:21:34.7931344Z remote: Counting objects: 18% (411/2282) 2024-06-26T05:21:34.7931871Z remote: Counting objects: 19% (434/2282) 2024-06-26T05:21:34.7932574Z remote: Counting objects: 20% (457/2282) 2024-06-26T05:21:34.7933150Z remote: Counting objects: 21% (480/2282) 2024-06-26T05:21:34.7933660Z remote: Counting objects: 22% (503/2282) 2024-06-26T05:21:34.7934233Z remote: Counting objects: 23% (525/2282) 2024-06-26T05:21:34.7934938Z remote: Counting objects: 24% (548/2282) 2024-06-26T05:21:34.7935487Z remote: Counting objects: 25% (571/2282) 2024-06-26T05:21:34.7936002Z remote: Counting objects: 26% (594/2282) 2024-06-26T05:21:34.7936506Z remote: Counting objects: 27% (617/2282) 2024-06-26T05:21:34.7937215Z remote: Counting objects: 28% (639/2282) 2024-06-26T05:21:34.7937734Z remote: Counting objects: 29% (662/2282) 2024-06-26T05:21:34.7938247Z remote: Counting objects: 30% (685/2282) 2024-06-26T05:21:34.7938750Z remote: Counting objects: 31% (708/2282) 2024-06-26T05:21:34.7939271Z remote: Counting objects: 32% (731/2282) 2024-06-26T05:21:34.7939806Z remote: Counting objects: 33% (754/2282) 2024-06-26T05:21:34.7940323Z remote: Counting objects: 34% (776/2282) 2024-06-26T05:21:34.7940825Z remote: Counting objects: 35% (799/2282) 2024-06-26T05:21:34.7941340Z remote: Counting objects: 36% (822/2282) 2024-06-26T05:21:34.7941854Z remote: Counting objects: 37% (845/2282) 2024-06-26T05:21:34.7942368Z remote: Counting objects: 38% (868/2282) 2024-06-26T05:21:34.7942875Z remote: Counting objects: 39% (890/2282) 2024-06-26T05:21:34.7943394Z remote: Counting objects: 40% (913/2282) 2024-06-26T05:21:34.7943923Z remote: Counting objects: 41% (936/2282) 2024-06-26T05:21:34.7944791Z remote: Counting objects: 42% (959/2282) 2024-06-26T05:21:34.7945303Z remote: Counting objects: 43% (982/2282) 2024-06-26T05:21:34.7945830Z remote: Counting objects: 44% (1005/2282) 2024-06-26T05:21:34.7946372Z remote: Counting objects: 45% (1027/2282) 2024-06-26T05:21:34.7947018Z remote: Counting objects: 46% (1050/2282) 2024-06-26T05:21:34.7947547Z remote: Counting objects: 47% (1073/2282) 2024-06-26T05:21:34.7948097Z remote: Counting objects: 48% (1096/2282) 2024-06-26T05:21:34.7948630Z remote: Counting objects: 49% (1119/2282) 2024-06-26T05:21:34.7949168Z remote: Counting objects: 50% (1141/2282) 2024-06-26T05:21:34.7949693Z remote: Counting objects: 51% (1164/2282) 2024-06-26T05:21:34.7950222Z remote: Counting objects: 52% (1187/2282) 2024-06-26T05:21:34.7950760Z remote: Counting objects: 53% (1210/2282) 2024-06-26T05:21:34.7951298Z remote: Counting objects: 54% (1233/2282) 2024-06-26T05:21:34.7951832Z remote: Counting objects: 55% (1256/2282) 2024-06-26T05:21:34.7952360Z remote: Counting objects: 56% (1278/2282) 2024-06-26T05:21:34.7952887Z remote: Counting objects: 57% (1301/2282) 2024-06-26T05:21:34.7953418Z remote: Counting objects: 58% (1324/2282) 2024-06-26T05:21:34.7953954Z remote: Counting objects: 59% (1347/2282) 2024-06-26T05:21:34.7954489Z remote: Counting objects: 60% (1370/2282) 2024-06-26T05:21:34.7955021Z remote: Counting objects: 61% (1393/2282) 2024-06-26T05:21:34.7955553Z remote: Counting objects: 62% (1415/2282) 2024-06-26T05:21:34.7956098Z remote: Counting objects: 63% (1438/2282) 2024-06-26T05:21:34.7956623Z remote: Counting objects: 64% (1461/2282) 2024-06-26T05:21:34.7957162Z remote: Counting objects: 65% (1484/2282) 2024-06-26T05:21:34.7957695Z remote: Counting objects: 66% (1507/2282) 2024-06-26T05:21:34.7958223Z remote: Counting objects: 67% (1529/2282) 2024-06-26T05:21:34.7958740Z remote: Counting objects: 68% (1552/2282) 2024-06-26T05:21:34.7959266Z remote: Counting objects: 69% (1575/2282) 2024-06-26T05:21:34.7959796Z remote: Counting objects: 70% (1598/2282) 2024-06-26T05:21:34.7960339Z remote: Counting objects: 71% (1621/2282) 2024-06-26T05:21:34.7960861Z remote: Counting objects: 72% (1644/2282) 2024-06-26T05:21:34.7961398Z remote: Counting objects: 73% (1666/2282) 2024-06-26T05:21:34.7961930Z remote: Counting objects: 74% (1689/2282) 2024-06-26T05:21:34.7962461Z remote: Counting objects: 75% (1712/2282) 2024-06-26T05:21:34.7963006Z remote: Counting objects: 76% (1735/2282) 2024-06-26T05:21:34.7963536Z remote: Counting objects: 77% (1758/2282) 2024-06-26T05:21:34.7964079Z remote: Counting objects: 78% (1780/2282) 2024-06-26T05:21:34.7964692Z remote: Counting objects: 79% (1803/2282) 2024-06-26T05:21:34.7965306Z remote: Counting objects: 80% (1826/2282) 2024-06-26T05:21:34.7965837Z remote: Counting objects: 81% (1849/2282) 2024-06-26T05:21:34.7966375Z remote: Counting objects: 82% (1872/2282) 2024-06-26T05:21:34.7966915Z remote: Counting objects: 83% (1895/2282) 2024-06-26T05:21:34.7967498Z remote: Counting objects: 84% (1917/2282) 2024-06-26T05:21:34.7968039Z remote: Counting objects: 85% (1940/2282) 2024-06-26T05:21:34.7968584Z remote: Counting objects: 86% (1963/2282) 2024-06-26T05:21:34.7969134Z remote: Counting objects: 87% (1986/2282) 2024-06-26T05:21:34.7969657Z remote: Counting objects: 88% (2009/2282) 2024-06-26T05:21:34.7970197Z remote: Counting objects: 89% (2031/2282) 2024-06-26T05:21:34.7970745Z remote: Counting objects: 90% (2054/2282) 2024-06-26T05:21:34.7971287Z remote: Counting objects: 91% (2077/2282) 2024-06-26T05:21:34.7971817Z remote: Counting objects: 92% (2100/2282) 2024-06-26T05:21:34.7972364Z remote: Counting objects: 93% (2123/2282) 2024-06-26T05:21:34.7972898Z remote: Counting objects: 94% (2146/2282) 2024-06-26T05:21:34.7973444Z remote: Counting objects: 95% (2168/2282) 2024-06-26T05:21:34.7973971Z remote: Counting objects: 96% (2191/2282) 2024-06-26T05:21:34.7974594Z remote: Counting objects: 97% (2214/2282) 2024-06-26T05:21:34.7975134Z remote: Counting objects: 98% (2237/2282) 2024-06-26T05:21:34.7975672Z remote: Counting objects: 99% (2260/2282) 2024-06-26T05:21:34.7976197Z remote: Counting objects: 100% (2282/2282) 2024-06-26T05:21:34.7976776Z remote: Counting objects: 100% (2282/2282), done. 2024-06-26T05:21:34.8079908Z remote: Compressing objects: 0% (1/949) 2024-06-26T05:21:34.8393428Z remote: Compressing objects: 1% (10/949) 2024-06-26T05:21:34.8696730Z remote: Compressing objects: 2% (19/949) 2024-06-26T05:21:34.9053363Z remote: Compressing objects: 3% (29/949) 2024-06-26T05:21:34.9287021Z remote: Compressing objects: 4% (38/949) 2024-06-26T05:21:34.9438648Z remote: Compressing objects: 5% (48/949) 2024-06-26T05:21:34.9687136Z remote: Compressing objects: 6% (57/949) 2024-06-26T05:21:34.9854872Z remote: Compressing objects: 7% (67/949) 2024-06-26T05:21:35.0034749Z remote: Compressing objects: 8% (76/949) 2024-06-26T05:21:35.0402681Z remote: Compressing objects: 9% (86/949) 2024-06-26T05:21:35.1207360Z remote: Compressing objects: 10% (95/949) 2024-06-26T05:21:35.2162281Z remote: Compressing objects: 11% (105/949) 2024-06-26T05:21:35.2716723Z remote: Compressing objects: 12% (114/949) 2024-06-26T05:21:35.2963099Z remote: Compressing objects: 13% (124/949) 2024-06-26T05:21:35.3128864Z remote: Compressing objects: 14% (133/949) 2024-06-26T05:21:35.3131272Z remote: Compressing objects: 15% (143/949) 2024-06-26T05:21:35.3302628Z remote: Compressing objects: 16% (152/949) 2024-06-26T05:21:35.3338925Z remote: Compressing objects: 17% (162/949) 2024-06-26T05:21:35.3368600Z remote: Compressing objects: 18% (171/949) 2024-06-26T05:21:35.3372645Z remote: Compressing objects: 19% (181/949) 2024-06-26T05:21:35.3395257Z remote: Compressing objects: 20% (190/949) 2024-06-26T05:21:35.3410793Z remote: Compressing objects: 21% (200/949) 2024-06-26T05:21:35.3423770Z remote: Compressing objects: 22% (209/949) 2024-06-26T05:21:35.3431689Z remote: Compressing objects: 23% (219/949) 2024-06-26T05:21:35.3452026Z remote: Compressing objects: 24% (228/949) 2024-06-26T05:21:35.3460139Z remote: Compressing objects: 25% (238/949) 2024-06-26T05:21:35.3469152Z remote: Compressing objects: 26% (247/949) 2024-06-26T05:21:35.3473990Z remote: Compressing objects: 27% (257/949) 2024-06-26T05:21:35.3483328Z remote: Compressing objects: 28% (266/949) 2024-06-26T05:21:35.3483891Z remote: Compressing objects: 29% (276/949) 2024-06-26T05:21:35.3491156Z remote: Compressing objects: 30% (285/949) 2024-06-26T05:21:35.3491714Z remote: Compressing objects: 31% (295/949) 2024-06-26T05:21:35.3498887Z remote: Compressing objects: 32% (304/949) 2024-06-26T05:21:35.3504920Z remote: Compressing objects: 33% (314/949) 2024-06-26T05:21:35.3511340Z remote: Compressing objects: 34% (323/949) 2024-06-26T05:21:35.3514128Z remote: Compressing objects: 35% (333/949) 2024-06-26T05:21:35.3525584Z remote: Compressing objects: 36% (342/949) 2024-06-26T05:21:35.3535289Z remote: Compressing objects: 37% (352/949) 2024-06-26T05:21:35.3546514Z remote: Compressing objects: 38% (361/949) 2024-06-26T05:21:35.3564160Z remote: Compressing objects: 39% (371/949) 2024-06-26T05:21:35.3570338Z remote: Compressing objects: 40% (380/949) 2024-06-26T05:21:35.3582892Z remote: Compressing objects: 41% (390/949) 2024-06-26T05:21:35.3595920Z remote: Compressing objects: 42% (399/949) 2024-06-26T05:21:35.3606166Z remote: Compressing objects: 43% (409/949) 2024-06-26T05:21:35.3620685Z remote: Compressing objects: 44% (418/949) 2024-06-26T05:21:35.3633298Z remote: Compressing objects: 45% (428/949) 2024-06-26T05:21:35.3646780Z remote: Compressing objects: 46% (437/949) 2024-06-26T05:21:35.3662886Z remote: Compressing objects: 47% (447/949) 2024-06-26T05:21:35.3665734Z remote: Compressing objects: 48% (456/949) 2024-06-26T05:21:35.3670000Z remote: Compressing objects: 49% (466/949) 2024-06-26T05:21:35.3675979Z remote: Compressing objects: 50% (475/949) 2024-06-26T05:21:35.3681061Z remote: Compressing objects: 51% (484/949) 2024-06-26T05:21:35.3699270Z remote: Compressing objects: 52% (494/949) 2024-06-26T05:21:35.3710574Z remote: Compressing objects: 53% (503/949) 2024-06-26T05:21:35.3717276Z remote: Compressing objects: 54% (513/949) 2024-06-26T05:21:35.3726566Z remote: Compressing objects: 55% (522/949) 2024-06-26T05:21:35.3744319Z remote: Compressing objects: 56% (532/949) 2024-06-26T05:21:35.3755854Z remote: Compressing objects: 57% (541/949) 2024-06-26T05:21:35.3765737Z remote: Compressing objects: 58% (551/949) 2024-06-26T05:21:35.3773752Z remote: Compressing objects: 59% (560/949) 2024-06-26T05:21:35.3782892Z remote: Compressing objects: 60% (570/949) 2024-06-26T05:21:35.3791216Z remote: Compressing objects: 61% (579/949) 2024-06-26T05:21:35.3801242Z remote: Compressing objects: 62% (589/949) 2024-06-26T05:21:35.3813357Z remote: Compressing objects: 63% (598/949) 2024-06-26T05:21:35.3819723Z remote: Compressing objects: 64% (608/949) 2024-06-26T05:21:35.3830125Z remote: Compressing objects: 65% (617/949) 2024-06-26T05:21:35.3839795Z remote: Compressing objects: 66% (627/949) 2024-06-26T05:21:35.3846379Z remote: Compressing objects: 67% (636/949) 2024-06-26T05:21:35.3853340Z remote: Compressing objects: 68% (646/949) 2024-06-26T05:21:35.3863589Z remote: Compressing objects: 69% (655/949) 2024-06-26T05:21:35.3871347Z remote: Compressing objects: 70% (665/949) 2024-06-26T05:21:35.3879961Z remote: Compressing objects: 71% (674/949) 2024-06-26T05:21:35.3885810Z remote: Compressing objects: 72% (684/949) 2024-06-26T05:21:35.3890867Z remote: Compressing objects: 73% (693/949) 2024-06-26T05:21:35.3894679Z remote: Compressing objects: 74% (703/949) 2024-06-26T05:21:35.3900189Z remote: Compressing objects: 75% (712/949) 2024-06-26T05:21:35.3900751Z remote: Compressing objects: 76% (722/949) 2024-06-26T05:21:35.3901287Z remote: Compressing objects: 77% (731/949) 2024-06-26T05:21:35.3901946Z remote: Compressing objects: 78% (741/949) 2024-06-26T05:21:35.3902497Z remote: Compressing objects: 79% (750/949) 2024-06-26T05:21:35.3903044Z remote: Compressing objects: 80% (760/949) 2024-06-26T05:21:35.3907699Z remote: Compressing objects: 81% (769/949) 2024-06-26T05:21:35.3911351Z remote: Compressing objects: 82% (779/949) 2024-06-26T05:21:35.3914924Z remote: Compressing objects: 83% (788/949) 2024-06-26T05:21:35.3917831Z remote: Compressing objects: 84% (798/949) 2024-06-26T05:21:35.3920260Z remote: Compressing objects: 85% (807/949) 2024-06-26T05:21:35.3922821Z remote: Compressing objects: 86% (817/949) 2024-06-26T05:21:35.3925295Z remote: Compressing objects: 87% (826/949) 2024-06-26T05:21:35.3930505Z remote: Compressing objects: 88% (836/949) 2024-06-26T05:21:35.3931058Z remote: Compressing objects: 89% (845/949) 2024-06-26T05:21:35.3935205Z remote: Compressing objects: 90% (855/949) 2024-06-26T05:21:35.3935765Z remote: Compressing objects: 91% (864/949) 2024-06-26T05:21:35.3936319Z remote: Compressing objects: 92% (874/949) 2024-06-26T05:21:35.3936863Z remote: Compressing objects: 93% (883/949) 2024-06-26T05:21:35.3937874Z remote: Compressing objects: 94% (893/949) 2024-06-26T05:21:35.3938431Z remote: Compressing objects: 95% (902/949) 2024-06-26T05:21:35.3940692Z remote: Compressing objects: 96% (912/949) 2024-06-26T05:21:35.3941241Z remote: Compressing objects: 97% (921/949) 2024-06-26T05:21:35.3943164Z remote: Compressing objects: 98% (931/949) 2024-06-26T05:21:35.3943712Z remote: Compressing objects: 99% (940/949) 2024-06-26T05:21:35.3944367Z remote: Compressing objects: 100% (949/949) 2024-06-26T05:21:35.3944963Z remote: Compressing objects: 100% (949/949), done. 2024-06-26T05:21:55.5306984Z remote: Total 990060 (delta 1836), reused 1629 (delta 1332), pack-reused 987778 2024-06-26T05:22:19.8181165Z [command]/usr/bin/git rev-parse --verify --quiet b8c4c54d347aa776934c60784e35936878ef18dc^{object} 2024-06-26T05:22:19.8206234Z b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T05:22:19.8209699Z ##[endgroup] 2024-06-26T05:22:19.8210280Z ##[group]Determining the checkout info 2024-06-26T05:22:19.8213007Z ##[endgroup] 2024-06-26T05:22:19.8213595Z ##[group]Checking out the ref 2024-06-26T05:22:19.8215665Z [command]/usr/bin/git checkout --quiet --force b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T05:22:21.0452092Z ##[endgroup] 2024-06-26T05:22:21.0452776Z ##[group]Setting up auth for fetching submodules 2024-06-26T05:22:21.0457432Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T05:22:21.0505664Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-06-26T05:22:21.0531972Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-06-26T05:22:21.0558431Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-06-26T05:22:21.0580841Z ##[endgroup] 2024-06-26T05:22:21.0581392Z ##[group]Fetching submodules 2024-06-26T05:22:21.0585468Z [command]/usr/bin/git submodule sync --recursive 2024-06-26T05:22:21.0824813Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-06-26T05:22:21.1052625Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-06-26T05:22:21.1054208Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-06-26T05:22:21.1055799Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-06-26T05:22:21.1057278Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-06-26T05:22:21.1059516Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-06-26T05:22:21.1061380Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-06-26T05:22:21.1063082Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-06-26T05:22:21.1065800Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-06-26T05:22:21.1067884Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-06-26T05:22:21.1070473Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-06-26T05:22:21.1072285Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-06-26T05:22:21.1074945Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-06-26T05:22:21.1077360Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-06-26T05:22:21.1079992Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-06-26T05:22:21.1082458Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-06-26T05:22:21.1085101Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-06-26T05:22:21.1088166Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-06-26T05:22:21.1090647Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-06-26T05:22:21.1093624Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-06-26T05:22:21.1096342Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-06-26T05:22:21.1099393Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-06-26T05:22:21.1102425Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-06-26T05:22:21.1105859Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-06-26T05:22:21.1109016Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-06-26T05:22:21.1113105Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-06-26T05:22:21.1115986Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-06-26T05:22:21.1119490Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-06-26T05:22:21.1122660Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-06-26T05:22:21.1126250Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-06-26T05:22:21.1129832Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-06-26T05:22:21.1133512Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-06-26T05:22:21.1137221Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-06-26T05:22:21.1140892Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-06-26T05:22:21.1145559Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-06-26T05:22:21.1150532Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-06-26T05:22:21.1171294Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-06-26T05:22:21.3284192Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-06-26T05:22:21.4992457Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-06-26T05:22:21.6553923Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-06-26T05:22:21.8703595Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-06-26T05:22:23.7596014Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-06-26T05:22:32.5721371Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-06-26T05:22:32.9513749Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-06-26T05:22:33.4449499Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-06-26T05:22:33.9881775Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-06-26T05:22:35.1221435Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-06-26T05:22:36.8714433Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-06-26T05:22:42.7762697Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-06-26T05:22:44.0046663Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-06-26T05:22:45.6298517Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-06-26T05:22:46.7585725Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-06-26T05:22:46.9099299Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-06-26T05:22:47.2699111Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-06-26T05:22:47.5798866Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-06-26T05:22:48.4665275Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-06-26T05:22:48.7680341Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-06-26T05:22:48.9837625Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-06-26T05:22:50.5718051Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-06-26T05:22:51.3691908Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-06-26T05:22:52.0797928Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-06-26T05:22:57.2804619Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-06-26T05:22:59.4193279Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-06-26T05:23:04.2193727Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-06-26T05:23:04.4039704Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-06-26T05:23:12.1458247Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-06-26T05:23:12.3014800Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-06-26T05:23:12.4976034Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-06-26T05:23:13.4995898Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-06-26T05:23:13.7777758Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-06-26T05:23:14.3719302Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-06-26T05:23:14.7797691Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-06-26T05:23:14.7886947Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-06-26T05:23:14.7953656Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-06-26T05:23:14.8140226Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-06-26T05:23:14.8449792Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-06-26T05:23:15.6243317Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-06-26T05:23:15.6422507Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-06-26T05:23:15.6784621Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-06-26T05:23:15.7643155Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-06-26T05:23:15.7916828Z Submodule path 'third_party/cudnn_frontend': checked out 'aa3abd4bc689d6412979c7f55f9cd132848c9c6a' 2024-06-26T05:23:16.2233803Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-06-26T05:23:16.4377092Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-06-26T05:23:16.5023986Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-06-26T05:23:16.5035344Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-06-26T05:23:16.5038459Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T05:23:16.5039998Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-06-26T05:23:16.5041526Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-06-26T05:23:16.5043213Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T05:23:16.5060984Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-06-26T05:23:17.3614704Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-06-26T05:23:17.9309108Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-06-26T05:23:19.6246451Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-06-26T05:23:20.5010847Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-06-26T05:23:20.8086798Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-06-26T05:23:20.8935803Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-06-26T05:23:21.2370654Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-06-26T05:23:21.2924084Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-06-26T05:23:21.3014362Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-06-26T05:23:21.3943747Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-06-26T05:23:21.4233869Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-06-26T05:23:21.4303793Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-06-26T05:23:21.4619987Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-06-26T05:23:21.4818825Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-06-26T05:23:21.5216974Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T05:23:21.5321033Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-06-26T05:23:21.5332727Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-06-26T05:23:21.5350910Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-06-26T05:23:32.9449095Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-06-26T05:23:32.9589266Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-06-26T05:23:33.0381128Z Submodule path 'third_party/kineto': checked out '8681ff11e1fa54da39023076c5c43eddd87b7a8a' 2024-06-26T05:23:33.0392657Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T05:23:33.0394410Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T05:23:33.0396123Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T05:23:33.0416760Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-06-26T05:23:33.5441029Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-06-26T05:23:34.6317182Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-06-26T05:23:35.5746763Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-06-26T05:23:35.5758864Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T05:23:35.5760947Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T05:23:35.5763050Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T05:23:35.5765260Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T05:23:35.5767343Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T05:23:35.5769469Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T05:23:35.5771818Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T05:23:35.5773673Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T05:23:35.5791304Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-06-26T05:23:36.5555576Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-06-26T05:23:36.8764129Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-06-26T05:23:37.9708111Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-06-26T05:23:38.2311360Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-06-26T05:23:38.6930248Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-06-26T05:23:39.5552507Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-06-26T05:23:44.7805200Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-06-26T05:23:45.1232553Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-06-26T05:23:45.1377458Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-06-26T05:23:45.1676861Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-06-26T05:23:45.1782432Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-06-26T05:23:45.1793685Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T05:23:45.1812112Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-06-26T05:23:45.4573320Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-06-26T05:23:45.4722428Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-06-26T05:23:45.5072578Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-06-26T05:23:45.5897222Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-06-26T05:23:45.6030630Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-06-26T05:23:45.6331642Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-06-26T05:23:45.6834041Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-06-26T05:23:45.7145298Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-06-26T05:23:45.7348669Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-06-26T05:23:45.8213477Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-06-26T05:23:46.1001810Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-06-26T05:23:46.1029603Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-06-26T05:23:46.1032766Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-06-26T05:23:46.1046981Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-06-26T05:23:46.4901267Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-06-26T05:23:47.6158524Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-06-26T05:23:47.6451390Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-06-26T05:23:47.6987354Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-06-26T05:23:47.7003241Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T05:23:47.7009487Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T05:23:47.7012262Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T05:23:47.7014277Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T05:23:47.7016550Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T05:23:47.7018597Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T05:23:47.7020502Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T05:23:47.7022133Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T05:23:47.7037428Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-06-26T05:23:48.2073222Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-06-26T05:23:49.1238703Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-06-26T05:23:49.4235355Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-06-26T05:23:54.9466239Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-06-26T05:23:55.3181674Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-06-26T05:23:55.5985345Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-06-26T05:23:55.9009416Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-06-26T05:24:03.4180344Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-06-26T05:24:03.4524252Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-06-26T05:24:03.4659270Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-06-26T05:24:03.5515902Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-06-26T05:24:03.5616785Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-06-26T05:24:03.5732490Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-06-26T05:24:03.5851123Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-06-26T05:24:03.5862023Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T05:24:03.5864013Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T05:24:03.5882390Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-06-26T05:24:05.0982591Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-06-26T05:24:06.1767570Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-06-26T05:24:06.2143634Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T05:24:06.5856783Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-06-26T05:24:06.5947961Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-06-26T05:24:06.8132822Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-06-26T05:24:06.8146879Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-06-26T05:24:06.8148507Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-06-26T05:24:06.8167215Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-06-26T05:24:07.1791399Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-06-26T05:24:08.0801871Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-06-26T05:24:08.1440243Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-06-26T05:24:08.1507575Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-06-26T05:24:08.1603744Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-06-26T05:24:08.1889304Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-06-26T05:24:08.2117414Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-06-26T05:24:08.2451960Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-06-26T05:24:08.2661773Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-06-26T05:24:08.2675017Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-06-26T05:24:08.2676611Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-06-26T05:24:08.2678214Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-06-26T05:24:08.2679985Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T05:24:08.2698328Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-06-26T05:24:09.1208423Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-06-26T05:24:09.3458052Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-06-26T05:24:10.3247349Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-06-26T05:24:11.3548260Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-06-26T05:24:11.3669038Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-06-26T05:24:11.4244745Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-06-26T05:24:11.4486150Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-06-26T05:24:11.4496534Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T05:24:11.4515254Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-06-26T05:24:11.6324136Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-06-26T05:24:11.6351329Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-06-26T05:24:11.6577055Z Entering 'android/libs/fbjni' 2024-06-26T05:24:11.6607152Z Entering 'third_party/FP16' 2024-06-26T05:24:11.6638440Z Entering 'third_party/FXdiv' 2024-06-26T05:24:11.6668666Z Entering 'third_party/NNPACK' 2024-06-26T05:24:11.6699017Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T05:24:11.6729412Z Entering 'third_party/XNNPACK' 2024-06-26T05:24:11.6770973Z Entering 'third_party/benchmark' 2024-06-26T05:24:11.6801754Z Entering 'third_party/cpp-httplib' 2024-06-26T05:24:11.6832314Z Entering 'third_party/cpuinfo' 2024-06-26T05:24:11.6862496Z Entering 'third_party/cudnn_frontend' 2024-06-26T05:24:11.6892262Z Entering 'third_party/cutlass' 2024-06-26T05:24:11.6928869Z Entering 'third_party/eigen' 2024-06-26T05:24:11.6960952Z Entering 'third_party/fbgemm' 2024-06-26T05:24:11.6992030Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T05:24:11.7020475Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T05:24:11.7049634Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T05:24:11.7083519Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T05:24:11.7111791Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T05:24:11.7141006Z Entering 'third_party/flatbuffers' 2024-06-26T05:24:11.7174856Z Entering 'third_party/fmt' 2024-06-26T05:24:11.7205757Z Entering 'third_party/foxi' 2024-06-26T05:24:11.7235797Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T05:24:11.7266277Z Entering 'third_party/gloo' 2024-06-26T05:24:11.7295923Z Entering 'third_party/googletest' 2024-06-26T05:24:11.7326905Z Entering 'third_party/ideep' 2024-06-26T05:24:11.7356801Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T05:24:11.7391283Z Entering 'third_party/ittapi' 2024-06-26T05:24:11.7420199Z Entering 'third_party/kineto' 2024-06-26T05:24:11.7450920Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T05:24:11.7480316Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T05:24:11.7511064Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T05:24:11.7539763Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T05:24:11.7574460Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T05:24:11.7602202Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T05:24:11.7633566Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T05:24:11.7662008Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T05:24:11.7692250Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T05:24:11.7722336Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T05:24:11.7757286Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T05:24:11.7786884Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T05:24:11.7817786Z Entering 'third_party/mimalloc' 2024-06-26T05:24:11.7848316Z Entering 'third_party/nccl/nccl' 2024-06-26T05:24:11.7878936Z Entering 'third_party/nlohmann' 2024-06-26T05:24:11.7911567Z Entering 'third_party/onnx' 2024-06-26T05:24:11.7952777Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T05:24:11.7982836Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T05:24:11.8014141Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T05:24:11.8047725Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T05:24:11.8078391Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T05:24:11.8108781Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T05:24:11.8137847Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T05:24:11.8167779Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T05:24:11.8196930Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T05:24:11.8225624Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T05:24:11.8257455Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T05:24:11.8288523Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T05:24:11.8319394Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T05:24:11.8363107Z Entering 'third_party/pocketfft' 2024-06-26T05:24:11.8392387Z Entering 'third_party/protobuf' 2024-06-26T05:24:11.8423233Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T05:24:11.8451556Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T05:24:11.8482517Z Entering 'third_party/psimd' 2024-06-26T05:24:11.8515443Z Entering 'third_party/pthreadpool' 2024-06-26T05:24:11.8546247Z Entering 'third_party/pybind11' 2024-06-26T05:24:11.8577626Z Entering 'third_party/python-peachpy' 2024-06-26T05:24:11.8608594Z Entering 'third_party/sleef' 2024-06-26T05:24:11.8638398Z Entering 'third_party/tensorpipe' 2024-06-26T05:24:11.8675516Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T05:24:11.8704653Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T05:24:11.8734229Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T05:24:11.8764730Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T05:24:11.8793183Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T05:24:11.8839153Z ##[endgroup] 2024-06-26T05:24:11.8841641Z ##[group]Persisting credentials for submodules 2024-06-26T05:24:11.8844122Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-06-26T05:24:11.9076286Z Entering 'android/libs/fbjni' 2024-06-26T05:24:11.9113457Z Entering 'third_party/FP16' 2024-06-26T05:24:11.9155147Z Entering 'third_party/FXdiv' 2024-06-26T05:24:11.9195330Z Entering 'third_party/NNPACK' 2024-06-26T05:24:11.9233387Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T05:24:11.9274838Z Entering 'third_party/XNNPACK' 2024-06-26T05:24:11.9324686Z Entering 'third_party/benchmark' 2024-06-26T05:24:11.9364473Z Entering 'third_party/cpp-httplib' 2024-06-26T05:24:11.9402279Z Entering 'third_party/cpuinfo' 2024-06-26T05:24:11.9441717Z Entering 'third_party/cudnn_frontend' 2024-06-26T05:24:11.9480644Z Entering 'third_party/cutlass' 2024-06-26T05:24:11.9534322Z Entering 'third_party/eigen' 2024-06-26T05:24:11.9574353Z Entering 'third_party/fbgemm' 2024-06-26T05:24:11.9613414Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T05:24:11.9651200Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T05:24:11.9688729Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T05:24:11.9734010Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T05:24:11.9772035Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T05:24:11.9809921Z Entering 'third_party/flatbuffers' 2024-06-26T05:24:11.9851268Z Entering 'third_party/fmt' 2024-06-26T05:24:11.9889866Z Entering 'third_party/foxi' 2024-06-26T05:24:11.9929124Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T05:24:11.9967987Z Entering 'third_party/gloo' 2024-06-26T05:24:12.0007552Z Entering 'third_party/googletest' 2024-06-26T05:24:12.0045071Z Entering 'third_party/ideep' 2024-06-26T05:24:12.0081876Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T05:24:12.0123327Z Entering 'third_party/ittapi' 2024-06-26T05:24:12.0162709Z Entering 'third_party/kineto' 2024-06-26T05:24:12.0200515Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T05:24:12.0236230Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T05:24:12.0274806Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T05:24:12.0312417Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T05:24:12.0351686Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T05:24:12.0388270Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T05:24:12.0426800Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T05:24:12.0463661Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T05:24:12.0500911Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T05:24:12.0543056Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T05:24:12.0580021Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T05:24:12.0618275Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T05:24:12.0656796Z Entering 'third_party/mimalloc' 2024-06-26T05:24:12.0696454Z Entering 'third_party/nccl/nccl' 2024-06-26T05:24:12.0734323Z Entering 'third_party/nlohmann' 2024-06-26T05:24:12.0774338Z Entering 'third_party/onnx' 2024-06-26T05:24:12.0824016Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T05:24:12.0862798Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T05:24:12.0903061Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T05:24:12.0943867Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T05:24:12.0980651Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T05:24:12.1020216Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T05:24:12.1058886Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T05:24:12.1098144Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T05:24:12.1134159Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T05:24:12.1171312Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T05:24:12.1208162Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T05:24:12.1248422Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T05:24:12.1291593Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T05:24:12.1345547Z Entering 'third_party/pocketfft' 2024-06-26T05:24:12.1386217Z Entering 'third_party/protobuf' 2024-06-26T05:24:12.1427364Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T05:24:12.1464129Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T05:24:12.1503485Z Entering 'third_party/psimd' 2024-06-26T05:24:12.1544642Z Entering 'third_party/pthreadpool' 2024-06-26T05:24:12.1588205Z Entering 'third_party/pybind11' 2024-06-26T05:24:12.1628004Z Entering 'third_party/python-peachpy' 2024-06-26T05:24:12.1666314Z Entering 'third_party/sleef' 2024-06-26T05:24:12.1710254Z Entering 'third_party/tensorpipe' 2024-06-26T05:24:12.1749003Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T05:24:12.1786177Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T05:24:12.1822941Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T05:24:12.1859601Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T05:24:12.1896143Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T05:24:12.1950201Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-06-26T05:24:12.2183475Z Entering 'android/libs/fbjni' 2024-06-26T05:24:12.2217658Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-06-26T05:24:12.2233073Z Entering 'third_party/FP16' 2024-06-26T05:24:12.2268327Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-06-26T05:24:12.2282819Z Entering 'third_party/FXdiv' 2024-06-26T05:24:12.2320305Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-06-26T05:24:12.2334912Z Entering 'third_party/NNPACK' 2024-06-26T05:24:12.2369021Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-06-26T05:24:12.2383849Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T05:24:12.2418461Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-06-26T05:24:12.2433819Z Entering 'third_party/XNNPACK' 2024-06-26T05:24:12.2468374Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-06-26T05:24:12.2493787Z Entering 'third_party/benchmark' 2024-06-26T05:24:12.2528134Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-06-26T05:24:12.2542486Z Entering 'third_party/cpp-httplib' 2024-06-26T05:24:12.2577895Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-06-26T05:24:12.2592266Z Entering 'third_party/cpuinfo' 2024-06-26T05:24:12.2626602Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T05:24:12.2641163Z Entering 'third_party/cudnn_frontend' 2024-06-26T05:24:12.2676781Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-06-26T05:24:12.2691190Z Entering 'third_party/cutlass' 2024-06-26T05:24:12.2727640Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-06-26T05:24:12.2748183Z Entering 'third_party/eigen' 2024-06-26T05:24:12.2780996Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-06-26T05:24:12.2800208Z Entering 'third_party/fbgemm' 2024-06-26T05:24:12.2839755Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-06-26T05:24:12.2855404Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T05:24:12.2888643Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-06-26T05:24:12.2902550Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T05:24:12.2937476Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T05:24:12.2952129Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T05:24:12.2987277Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-06-26T05:24:12.3006181Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T05:24:12.3039907Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-06-26T05:24:12.3053733Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T05:24:12.3089112Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-06-26T05:24:12.3105442Z Entering 'third_party/flatbuffers' 2024-06-26T05:24:12.3139776Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-06-26T05:24:12.3157176Z Entering 'third_party/fmt' 2024-06-26T05:24:12.3190944Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-06-26T05:24:12.3205371Z Entering 'third_party/foxi' 2024-06-26T05:24:12.3238847Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-06-26T05:24:12.3253774Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T05:24:12.3289112Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-06-26T05:24:12.3303766Z Entering 'third_party/gloo' 2024-06-26T05:24:12.3338188Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-06-26T05:24:12.3353260Z Entering 'third_party/googletest' 2024-06-26T05:24:12.3390187Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-06-26T05:24:12.3403892Z Entering 'third_party/ideep' 2024-06-26T05:24:12.3438215Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-06-26T05:24:12.3452139Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T05:24:12.3486990Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-06-26T05:24:12.3506633Z Entering 'third_party/ittapi' 2024-06-26T05:24:12.3541056Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-06-26T05:24:12.3558295Z Entering 'third_party/kineto' 2024-06-26T05:24:12.3596329Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-06-26T05:24:12.3609688Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T05:24:12.3644692Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-06-26T05:24:12.3658768Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T05:24:12.3695673Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-06-26T05:24:12.3711144Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T05:24:12.3746086Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-06-26T05:24:12.3760124Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T05:24:12.3794467Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-06-26T05:24:12.3807624Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T05:24:12.3847303Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-06-26T05:24:12.3861367Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T05:24:12.3896559Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-06-26T05:24:12.3912166Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T05:24:12.3945674Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-06-26T05:24:12.3960330Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T05:24:12.3998065Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-06-26T05:24:12.4011408Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T05:24:12.4045517Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-06-26T05:24:12.4060879Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T05:24:12.4097320Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-06-26T05:24:12.4113229Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T05:24:12.4148058Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-06-26T05:24:12.4162207Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T05:24:12.4197822Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-06-26T05:24:12.4213074Z Entering 'third_party/mimalloc' 2024-06-26T05:24:12.4248377Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-06-26T05:24:12.4263292Z Entering 'third_party/nccl/nccl' 2024-06-26T05:24:12.4297970Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-06-26T05:24:12.4313375Z Entering 'third_party/nlohmann' 2024-06-26T05:24:12.4348404Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-06-26T05:24:12.4363805Z Entering 'third_party/onnx' 2024-06-26T05:24:12.4398104Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-06-26T05:24:12.4424900Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T05:24:12.4462522Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-06-26T05:24:12.4477455Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T05:24:12.4511688Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-06-26T05:24:12.4527894Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T05:24:12.4562078Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-06-26T05:24:12.4577927Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T05:24:12.4612668Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-06-26T05:24:12.4627016Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T05:24:12.4659945Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-06-26T05:24:12.4674591Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T05:24:12.4712719Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-06-26T05:24:12.4727197Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T05:24:12.4762408Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-06-26T05:24:12.4777102Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T05:24:12.4812457Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-06-26T05:24:12.4826318Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T05:24:12.4862842Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-06-26T05:24:12.4877223Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T05:24:12.4912357Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-06-26T05:24:12.4926005Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T05:24:12.4964390Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-06-26T05:24:12.4979693Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T05:24:12.5016978Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-06-26T05:24:12.5033684Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T05:24:12.5068005Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-06-26T05:24:12.5098169Z Entering 'third_party/pocketfft' 2024-06-26T05:24:12.5133317Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-06-26T05:24:12.5148591Z Entering 'third_party/protobuf' 2024-06-26T05:24:12.5182397Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-06-26T05:24:12.5199571Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T05:24:12.5237967Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-06-26T05:24:12.5251887Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T05:24:12.5286412Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-06-26T05:24:12.5301884Z Entering 'third_party/psimd' 2024-06-26T05:24:12.5336482Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-06-26T05:24:12.5351852Z Entering 'third_party/pthreadpool' 2024-06-26T05:24:12.5387581Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-06-26T05:24:12.5401158Z Entering 'third_party/pybind11' 2024-06-26T05:24:12.5435038Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-06-26T05:24:12.5454551Z Entering 'third_party/python-peachpy' 2024-06-26T05:24:12.5490893Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-06-26T05:24:12.5503795Z Entering 'third_party/sleef' 2024-06-26T05:24:12.5537862Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-06-26T05:24:12.5552361Z Entering 'third_party/tensorpipe' 2024-06-26T05:24:12.5587944Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-06-26T05:24:12.5600648Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T05:24:12.5637387Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-06-26T05:24:12.5651047Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T05:24:12.5685326Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-06-26T05:24:12.5699436Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T05:24:12.5733346Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-06-26T05:24:12.5748103Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T05:24:12.5782566Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-06-26T05:24:12.5795572Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T05:24:12.5834763Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-06-26T05:24:12.6556372Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-06-26T05:24:12.6781303Z Entering 'android/libs/fbjni' 2024-06-26T05:24:12.6816717Z Entering 'third_party/FP16' 2024-06-26T05:24:12.6847299Z Entering 'third_party/FXdiv' 2024-06-26T05:24:12.6878438Z Entering 'third_party/NNPACK' 2024-06-26T05:24:12.6909522Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T05:24:12.6939575Z Entering 'third_party/XNNPACK' 2024-06-26T05:24:12.6981580Z Entering 'third_party/benchmark' 2024-06-26T05:24:12.7012250Z Entering 'third_party/cpp-httplib' 2024-06-26T05:24:12.7043103Z Entering 'third_party/cpuinfo' 2024-06-26T05:24:12.7074364Z Entering 'third_party/cudnn_frontend' 2024-06-26T05:24:12.7105824Z Entering 'third_party/cutlass' 2024-06-26T05:24:12.7141466Z Entering 'third_party/eigen' 2024-06-26T05:24:12.7178099Z Entering 'third_party/fbgemm' 2024-06-26T05:24:12.7209411Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T05:24:12.7238338Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T05:24:12.7268020Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T05:24:12.7301398Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T05:24:12.7330445Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T05:24:12.7360873Z Entering 'third_party/flatbuffers' 2024-06-26T05:24:12.7391169Z Entering 'third_party/fmt' 2024-06-26T05:24:12.7421618Z Entering 'third_party/foxi' 2024-06-26T05:24:12.7456932Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T05:24:12.7487163Z Entering 'third_party/gloo' 2024-06-26T05:24:12.7518178Z Entering 'third_party/googletest' 2024-06-26T05:24:12.7549629Z Entering 'third_party/ideep' 2024-06-26T05:24:12.7579722Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T05:24:12.7615542Z Entering 'third_party/ittapi' 2024-06-26T05:24:12.7646401Z Entering 'third_party/kineto' 2024-06-26T05:24:12.7677205Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T05:24:12.7706863Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T05:24:12.7736910Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T05:24:12.7766802Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T05:24:12.7797298Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T05:24:12.7826270Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T05:24:12.7859751Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T05:24:12.7889443Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T05:24:12.7919575Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T05:24:12.7949370Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T05:24:12.7979781Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T05:24:12.8009980Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T05:24:12.8039580Z Entering 'third_party/mimalloc' 2024-06-26T05:24:12.8071011Z Entering 'third_party/nccl/nccl' 2024-06-26T05:24:12.8101111Z Entering 'third_party/nlohmann' 2024-06-26T05:24:12.8132965Z Entering 'third_party/onnx' 2024-06-26T05:24:12.8175746Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T05:24:12.8206777Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T05:24:12.8238896Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T05:24:12.8272339Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T05:24:12.8301873Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T05:24:12.8331257Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T05:24:12.8361224Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T05:24:12.8392375Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T05:24:12.8420632Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T05:24:12.8450614Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T05:24:12.8480257Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T05:24:12.8512591Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T05:24:12.8543820Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T05:24:12.8593757Z Entering 'third_party/pocketfft' 2024-06-26T05:24:12.8625930Z Entering 'third_party/protobuf' 2024-06-26T05:24:12.8658371Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T05:24:12.8689161Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T05:24:12.8720789Z Entering 'third_party/psimd' 2024-06-26T05:24:12.8752408Z Entering 'third_party/pthreadpool' 2024-06-26T05:24:12.8783204Z Entering 'third_party/pybind11' 2024-06-26T05:24:12.8814164Z Entering 'third_party/python-peachpy' 2024-06-26T05:24:12.8845122Z Entering 'third_party/sleef' 2024-06-26T05:24:12.8876294Z Entering 'third_party/tensorpipe' 2024-06-26T05:24:12.8906686Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T05:24:12.8939276Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T05:24:12.8969655Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T05:24:12.9000143Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T05:24:12.9030113Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T05:24:12.9076202Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-06-26T05:24:12.9313947Z Entering 'android/libs/fbjni' 2024-06-26T05:24:12.9346042Z Entering 'third_party/FP16' 2024-06-26T05:24:12.9379305Z Entering 'third_party/FXdiv' 2024-06-26T05:24:12.9410213Z Entering 'third_party/NNPACK' 2024-06-26T05:24:12.9441911Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T05:24:12.9473733Z Entering 'third_party/XNNPACK' 2024-06-26T05:24:12.9516061Z Entering 'third_party/benchmark' 2024-06-26T05:24:12.9546687Z Entering 'third_party/cpp-httplib' 2024-06-26T05:24:12.9577483Z Entering 'third_party/cpuinfo' 2024-06-26T05:24:12.9608268Z Entering 'third_party/cudnn_frontend' 2024-06-26T05:24:12.9639893Z Entering 'third_party/cutlass' 2024-06-26T05:24:12.9681597Z Entering 'third_party/eigen' 2024-06-26T05:24:12.9714196Z Entering 'third_party/fbgemm' 2024-06-26T05:24:12.9745233Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T05:24:12.9774031Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T05:24:12.9803657Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T05:24:12.9841834Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T05:24:12.9873652Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T05:24:12.9905038Z Entering 'third_party/flatbuffers' 2024-06-26T05:24:12.9937761Z Entering 'third_party/fmt' 2024-06-26T05:24:12.9968970Z Entering 'third_party/foxi' 2024-06-26T05:24:12.9999959Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T05:24:13.0038784Z Entering 'third_party/gloo' 2024-06-26T05:24:13.0070406Z Entering 'third_party/googletest' 2024-06-26T05:24:13.0101330Z Entering 'third_party/ideep' 2024-06-26T05:24:13.0131456Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T05:24:13.0167636Z Entering 'third_party/ittapi' 2024-06-26T05:24:13.0199848Z Entering 'third_party/kineto' 2024-06-26T05:24:13.0230584Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T05:24:13.0260487Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T05:24:13.0296705Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T05:24:13.0328016Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T05:24:13.0357626Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T05:24:13.0385451Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T05:24:13.0416304Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T05:24:13.0445762Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T05:24:13.0475566Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T05:24:13.0506583Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T05:24:13.0537616Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T05:24:13.0567354Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T05:24:13.0598280Z Entering 'third_party/mimalloc' 2024-06-26T05:24:13.0629605Z Entering 'third_party/nccl/nccl' 2024-06-26T05:24:13.0660248Z Entering 'third_party/nlohmann' 2024-06-26T05:24:13.0697515Z Entering 'third_party/onnx' 2024-06-26T05:24:13.0740769Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T05:24:13.0771526Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T05:24:13.0804747Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T05:24:13.0837197Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T05:24:13.0866854Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T05:24:13.0896677Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T05:24:13.0930261Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T05:24:13.0961134Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T05:24:13.0994603Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T05:24:13.1023663Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T05:24:13.1052367Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T05:24:13.1083179Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T05:24:13.1118458Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T05:24:13.1164676Z Entering 'third_party/pocketfft' 2024-06-26T05:24:13.1198284Z Entering 'third_party/protobuf' 2024-06-26T05:24:13.1231429Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T05:24:13.1262040Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T05:24:13.1294951Z Entering 'third_party/psimd' 2024-06-26T05:24:13.1324384Z Entering 'third_party/pthreadpool' 2024-06-26T05:24:13.1354936Z Entering 'third_party/pybind11' 2024-06-26T05:24:13.1385755Z Entering 'third_party/python-peachpy' 2024-06-26T05:24:13.1416215Z Entering 'third_party/sleef' 2024-06-26T05:24:13.1447531Z Entering 'third_party/tensorpipe' 2024-06-26T05:24:13.1476841Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T05:24:13.1507666Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T05:24:13.1536611Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T05:24:13.1564036Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T05:24:13.1593459Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T05:24:13.1637368Z ##[endgroup] 2024-06-26T05:24:13.1671668Z [command]/usr/bin/git log -1 --format='%H' 2024-06-26T05:24:13.1694147Z 'b8c4c54d347aa776934c60784e35936878ef18dc' 2024-06-26T05:24:13.1872189Z Prepare all required actions 2024-06-26T05:24:13.1872696Z Getting action download info 2024-06-26T05:24:13.3362162Z ##[group]Run ./.github/actions/setup-linux 2024-06-26T05:24:13.3362611Z env: 2024-06-26T05:24:13.3362902Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:24:13.3363259Z ##[endgroup] 2024-06-26T05:24:13.3423923Z ##[group]Run set -euo pipefail 2024-06-26T05:24:13.3424549Z set -euo pipefail 2024-06-26T05:24:13.3424960Z function get_ec2_metadata() { 2024-06-26T05:24:13.3425522Z  # Pulled from instance metadata endpoint for EC2 2024-06-26T05:24:13.3426434Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-06-26T05:24:13.3427224Z  category=$1 2024-06-26T05:24:13.3427772Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-06-26T05:24:13.3428448Z  runner_name_str=i-0a8c115e2cb171c32 2024-06-26T05:24:13.3428974Z  if [[ -f /.inarc ]]; then 2024-06-26T05:24:13.3429487Z  echo "ARC Runner, no info on ec2 metadata" 2024-06-26T05:24:13.3430076Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-06-26T05:24:13.3430786Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-06-26T05:24:13.3431412Z  else 2024-06-26T05:24:13.3431921Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-06-26T05:24:13.3432516Z  fi 2024-06-26T05:24:13.3432819Z } 2024-06-26T05:24:13.3433201Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-06-26T05:24:13.3433803Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-06-26T05:24:13.3434476Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-06-26T05:24:13.3435077Z echo "system info $(uname -a)" 2024-06-26T05:24:13.3442991Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:24:13.3443502Z env: 2024-06-26T05:24:13.3443804Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:24:13.3444168Z ##[endgroup] 2024-06-26T05:24:13.3513031Z ami-id: ami-0ce0c36d7a00b20e2 2024-06-26T05:24:13.3559516Z instance-id: i-0a8c115e2cb171c32 2024-06-26T05:24:13.3606165Z instance-type: g5.12xlarge 2024-06-26T05:24:13.3613450Z system info Linux ip-10-0-37-235.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-06-26T05:24:13.3634657Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T05:24:13.3635632Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T05:24:13.3643848Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:24:13.3644375Z env: 2024-06-26T05:24:13.3644687Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:24:13.3645132Z ##[endgroup] 2024-06-26T05:24:13.3708373Z ##[group]Run if systemctl is-active --quiet docker; then 2024-06-26T05:24:13.3709016Z if systemctl is-active --quiet docker; then 2024-06-26T05:24:13.3709585Z  echo "Docker daemon is running..."; 2024-06-26T05:24:13.3710059Z else 2024-06-26T05:24:13.3710577Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-06-26T05:24:13.3711188Z fi 2024-06-26T05:24:13.3719017Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:24:13.3719537Z env: 2024-06-26T05:24:13.3719846Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:24:13.3720216Z ##[endgroup] 2024-06-26T05:24:13.3757165Z Docker daemon is running... 2024-06-26T05:24:13.3801324Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T05:24:13.3801918Z with: 2024-06-26T05:24:13.3802204Z shell: bash 2024-06-26T05:24:13.3802514Z timeout_minutes: 5 2024-06-26T05:24:13.3802865Z max_attempts: 3 2024-06-26T05:24:13.3803215Z retry_wait_seconds: 30 2024-06-26T05:24:13.3804779Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-06-26T05:24:13.3806327Z polling_interval_seconds: 1 2024-06-26T05:24:13.3806731Z warning_on_retry: true 2024-06-26T05:24:13.3807106Z continue_on_error: false 2024-06-26T05:24:13.3807470Z env: 2024-06-26T05:24:13.3807755Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:24:13.3808131Z AWS_RETRY_MODE: standard 2024-06-26T05:24:13.3808508Z AWS_MAX_ATTEMPTS: 5 2024-06-26T05:24:13.3808870Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T05:24:13.3809262Z ##[endgroup] 2024-06-26T05:24:14.2515881Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T05:24:14.2516980Z Configure a credential helper to remove this warning. See 2024-06-26T05:24:14.2517857Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T05:24:14.2518399Z 2024-06-26T05:24:14.2518546Z Login Succeeded 2024-06-26T05:24:14.4314460Z Command completed after 1 attempt(s). 2024-06-26T05:24:14.4361204Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T05:24:14.4362129Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T05:24:14.4362809Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T05:24:14.4370825Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:24:14.4371358Z env: 2024-06-26T05:24:14.4371649Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:24:14.4372019Z ##[endgroup] 2024-06-26T05:24:14.4433847Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T05:24:14.4434649Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T05:24:14.4435258Z # shellcheck disable=SC2046 2024-06-26T05:24:14.4435755Z docker stop $(docker ps -q) || true 2024-06-26T05:24:14.4436264Z # Prune all of the docker images 2024-06-26T05:24:14.4436733Z docker system prune -af 2024-06-26T05:24:14.4444603Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:24:14.4445255Z env: 2024-06-26T05:24:14.4445555Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:24:14.4445922Z ##[endgroup] 2024-06-26T05:24:14.4733994Z "docker stop" requires at least 1 argument. 2024-06-26T05:24:14.4734654Z See 'docker stop --help'. 2024-06-26T05:24:14.4734900Z 2024-06-26T05:24:14.4735134Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-06-26T05:24:14.4739037Z 2024-06-26T05:24:14.4739726Z Stop one or more running containers 2024-06-26T05:24:14.4899542Z Total reclaimed space: 0B 2024-06-26T05:24:14.4930497Z ##[group]Run set +e 2024-06-26T05:24:14.4930877Z set +e 2024-06-26T05:24:14.4931197Z set -x 2024-06-26T05:24:14.4931510Z  2024-06-26T05:24:14.4931847Z PT_DOMAIN=download.pytorch.org 2024-06-26T05:24:14.4932790Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-06-26T05:24:14.4933890Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-06-26T05:24:14.4934671Z # one is returned at random 2024-06-26T05:24:14.4935235Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-06-26T05:24:14.4935763Z  2024-06-26T05:24:14.4936100Z if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T05:24:14.4936738Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-06-26T05:24:14.4937521Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-06-26T05:24:14.4938081Z  2024-06-26T05:24:14.4938424Z  if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T05:24:14.4938993Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-06-26T05:24:14.4939525Z  exit 1 2024-06-26T05:24:14.4939844Z  fi 2024-06-26T05:24:14.4940143Z fi 2024-06-26T05:24:14.4940539Z  2024-06-26T05:24:14.4940916Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-06-26T05:24:14.4941449Z  # Clean up any old records first 2024-06-26T05:24:14.4941978Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-06-26T05:24:14.4942594Z fi 2024-06-26T05:24:14.4942915Z  2024-06-26T05:24:14.4943422Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-06-26T05:24:14.4944046Z cat /etc/hosts 2024-06-26T05:24:14.4952220Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:24:14.4952802Z env: 2024-06-26T05:24:14.4953159Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:24:14.4953568Z ##[endgroup] 2024-06-26T05:24:14.4998774Z + PT_DOMAIN=download.pytorch.org 2024-06-26T05:24:14.4999417Z ++ dig -4 +short download.pytorch.org 2024-06-26T05:24:14.4999938Z ++ tail -n1 2024-06-26T05:24:14.5097787Z + RESOLVED_IP=18.160.10.22 2024-06-26T05:24:14.5098291Z + '[' -z 18.160.10.22 ']' 2024-06-26T05:24:14.5098855Z + grep -r download.pytorch.org /etc/hosts 2024-06-26T05:24:14.5105206Z 18.160.10.76 download.pytorch.org 2024-06-26T05:24:14.5106241Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-06-26T05:24:14.5206938Z + echo '18.160.10.22 download.pytorch.org' 2024-06-26T05:24:14.5207819Z + sudo tee -a /etc/hosts 2024-06-26T05:24:14.5459030Z 18.160.10.22 download.pytorch.org 2024-06-26T05:24:14.5472218Z + cat /etc/hosts 2024-06-26T05:24:14.5476861Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-06-26T05:24:14.5488884Z ::1 localhost6 localhost6.localdomain6 2024-06-26T05:24:14.5489547Z 18.160.10.22 download.pytorch.org 2024-06-26T05:24:14.5675421Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-06-26T05:24:14.5676044Z with: 2024-06-26T05:24:14.5677157Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T05:24:14.5678417Z docker-build-dir: .ci/docker 2024-06-26T05:24:14.5678814Z working-directory: . 2024-06-26T05:24:14.5679306Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T05:24:14.5679852Z force-push: false 2024-06-26T05:24:14.5680183Z env: 2024-06-26T05:24:14.5680486Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:24:14.5680864Z ##[endgroup] 2024-06-26T05:24:14.5698981Z ##[group]Run set -ex 2024-06-26T05:24:14.5699395Z set -ex 2024-06-26T05:24:14.5699718Z  2024-06-26T05:24:14.5700318Z # If the docker build directory or the build script doesn't exist, the action will 2024-06-26T05:24:14.5701357Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-06-26T05:24:14.5702189Z # job could then download the pre-built image as usual 2024-06-26T05:24:14.5703101Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-06-26T05:24:14.5703825Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-06-26T05:24:14.5704623Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T05:24:14.5705212Z  2024-06-26T05:24:14.5705755Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-06-26T05:24:14.5706428Z  exit 0 2024-06-26T05:24:14.5706755Z else 2024-06-26T05:24:14.5707158Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-06-26T05:24:14.5707637Z fi 2024-06-26T05:24:14.5707949Z  2024-06-26T05:24:14.5708457Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-06-26T05:24:14.5709353Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-06-26T05:24:14.5710154Z  # use it as it is, but first let's extract the tag 2024-06-26T05:24:14.5710908Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-06-26T05:24:14.5711675Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T05:24:14.5712426Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T05:24:14.5713010Z else 2024-06-26T05:24:14.5713479Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-06-26T05:24:14.5714189Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T05:24:14.5715116Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T05:24:14.5715882Z fi 2024-06-26T05:24:14.5724170Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:24:14.5724706Z env: 2024-06-26T05:24:14.5725088Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:24:14.5725481Z REPO_NAME: pytorch 2024-06-26T05:24:14.5726641Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T05:24:14.5727877Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T05:24:14.5728409Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T05:24:14.5728976Z ##[endgroup] 2024-06-26T05:24:14.5749984Z + [[ ! -d .ci/docker ]] 2024-06-26T05:24:14.5750440Z + [[ ! -f .ci/docker/build.sh ]] 2024-06-26T05:24:14.5750839Z + echo skip=false 2024-06-26T05:24:14.5752838Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-06-26T05:24:14.5756395Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T05:24:14.5757679Z ++ awk -F '[:,]' '{print $2}' 2024-06-26T05:24:14.5763422Z + DOCKER_TAG=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T05:24:14.5764250Z + echo docker-tag=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T05:24:14.5765813Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T05:24:14.5794342Z ##[group]Run set +e 2024-06-26T05:24:14.5794746Z set +e 2024-06-26T05:24:14.5795083Z set -x 2024-06-26T05:24:14.5795393Z  2024-06-26T05:24:14.5795700Z login() { 2024-06-26T05:24:14.5796433Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T05:24:14.5797200Z } 2024-06-26T05:24:14.5797507Z  2024-06-26T05:24:14.5797820Z retry () { 2024-06-26T05:24:14.5798249Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T05:24:14.5798865Z } 2024-06-26T05:24:14.5799160Z  2024-06-26T05:24:14.5799519Z retry login "${DOCKER_REGISTRY}" 2024-06-26T05:24:14.5799982Z  2024-06-26T05:24:14.5800480Z # Check if image already exists, if it does then skip building it 2024-06-26T05:24:14.5801226Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-06-26T05:24:14.5801779Z  exit 0 2024-06-26T05:24:14.5802127Z fi 2024-06-26T05:24:14.5802458Z  2024-06-26T05:24:14.5803028Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-06-26T05:24:14.5803902Z # be empty. The default action would be to continue rebuild the image 2024-06-26T05:24:14.5804698Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-06-26T05:24:14.5805530Z  # if we're on the base branch then use the parent commit 2024-06-26T05:24:14.5806154Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-06-26T05:24:14.5806622Z else 2024-06-26T05:24:14.5807125Z  # otherwise we're on a PR, so use the most recent base commit 2024-06-26T05:24:14.5807835Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-06-26T05:24:14.5808378Z fi 2024-06-26T05:24:14.5808696Z  2024-06-26T05:24:14.5809051Z if [[ -z "${MERGE_BASE}" ]]; then 2024-06-26T05:24:14.5809598Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T05:24:14.5810098Z  2024-06-26T05:24:14.5810804Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-06-26T05:24:14.5811609Z  exit 0 2024-06-26T05:24:14.5811936Z fi 2024-06-26T05:24:14.5812248Z  2024-06-26T05:24:14.5812739Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-06-26T05:24:14.5813768Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-06-26T05:24:14.5814603Z  exit 1 2024-06-26T05:24:14.5814943Z fi 2024-06-26T05:24:14.5815258Z  2024-06-26T05:24:14.5815791Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-06-26T05:24:14.5816763Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-06-26T05:24:14.5817676Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-06-26T05:24:14.5818687Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-06-26T05:24:14.5819855Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-06-26T05:24:14.5820638Z fi 2024-06-26T05:24:14.5820934Z  2024-06-26T05:24:14.5821324Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T05:24:14.5829844Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:24:14.5830384Z env: 2024-06-26T05:24:14.5830696Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:24:14.5831099Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T05:24:14.5831578Z BASE_REVISION: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T05:24:14.5832924Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T05:24:14.5834198Z DOCKER_TAG: 91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T05:24:14.5834825Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T05:24:14.5835367Z ##[endgroup] 2024-06-26T05:24:14.5855963Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T05:24:14.5856660Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T05:24:14.5857466Z + aws ecr get-login-password --region us-east-1 2024-06-26T05:24:14.5858249Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T05:24:15.0093911Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T05:24:15.0094947Z Configure a credential helper to remove this warning. See 2024-06-26T05:24:15.0095922Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T05:24:15.0096453Z 2024-06-26T05:24:15.0096592Z Login Succeeded 2024-06-26T05:24:15.0107050Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T05:24:15.2355456Z { 2024-06-26T05:24:15.2355784Z "schemaVersion": 2, 2024-06-26T05:24:15.2356378Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-06-26T05:24:15.2356992Z "config": { 2024-06-26T05:24:15.2357464Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-06-26T05:24:15.2358022Z "size": 48681, 2024-06-26T05:24:15.2358584Z "digest": "sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd" 2024-06-26T05:24:15.2359226Z }, 2024-06-26T05:24:15.2359497Z "layers": [ 2024-06-26T05:24:15.2359797Z { 2024-06-26T05:24:15.2360373Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2361090Z "size": 28580681, 2024-06-26T05:24:15.2361713Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-06-26T05:24:15.2362346Z }, 2024-06-26T05:24:15.2362629Z { 2024-06-26T05:24:15.2363105Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2363675Z "size": 7943451, 2024-06-26T05:24:15.2364358Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-06-26T05:24:15.2365311Z }, 2024-06-26T05:24:15.2365672Z { 2024-06-26T05:24:15.2366181Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2366754Z "size": 55728572, 2024-06-26T05:24:15.2367417Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-06-26T05:24:15.2368235Z }, 2024-06-26T05:24:15.2368576Z { 2024-06-26T05:24:15.2369145Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2369913Z "size": 186, 2024-06-26T05:24:15.2370647Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-06-26T05:24:15.2371297Z }, 2024-06-26T05:24:15.2371562Z { 2024-06-26T05:24:15.2372001Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2372571Z "size": 6886, 2024-06-26T05:24:15.2373126Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-06-26T05:24:15.2373809Z }, 2024-06-26T05:24:15.2374367Z { 2024-06-26T05:24:15.2374945Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2375516Z "size": 1329001756, 2024-06-26T05:24:15.2376101Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-06-26T05:24:15.2376753Z }, 2024-06-26T05:24:15.2377015Z { 2024-06-26T05:24:15.2377466Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2378025Z "size": 62501, 2024-06-26T05:24:15.2378580Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-06-26T05:24:15.2379365Z + exit 0 2024-06-26T05:24:15.2379638Z }, 2024-06-26T05:24:15.2379888Z { 2024-06-26T05:24:15.2380331Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2380892Z "size": 1684, 2024-06-26T05:24:15.2381435Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-06-26T05:24:15.2382057Z }, 2024-06-26T05:24:15.2382323Z { 2024-06-26T05:24:15.2382774Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2383351Z "size": 1523, 2024-06-26T05:24:15.2383921Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-06-26T05:24:15.2384885Z }, 2024-06-26T05:24:15.2385148Z { 2024-06-26T05:24:15.2385592Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2386166Z "size": 2528295403, 2024-06-26T05:24:15.2386753Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-06-26T05:24:15.2387386Z }, 2024-06-26T05:24:15.2387644Z { 2024-06-26T05:24:15.2388100Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2388668Z "size": 86016, 2024-06-26T05:24:15.2389227Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-06-26T05:24:15.2389861Z }, 2024-06-26T05:24:15.2390121Z { 2024-06-26T05:24:15.2390573Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2391130Z "size": 1823, 2024-06-26T05:24:15.2391682Z "digest": "sha256:309b872da805fcdf74137a887464876fa0615dc8b24199307119eb65fc2bfb49" 2024-06-26T05:24:15.2392304Z }, 2024-06-26T05:24:15.2392570Z { 2024-06-26T05:24:15.2393007Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2393584Z "size": 246737542, 2024-06-26T05:24:15.2394154Z "digest": "sha256:e82932cb002c06a782077b8e5965f8733eb8563406a23cab260d9d8c4a994f7d" 2024-06-26T05:24:15.2394780Z }, 2024-06-26T05:24:15.2395032Z { 2024-06-26T05:24:15.2395484Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2396051Z "size": 545, 2024-06-26T05:24:15.2396609Z "digest": "sha256:ca9dd73e3e9378225ebdcfd12199ef44649e34fe55003cfbc8afe2e8a4527c5b" 2024-06-26T05:24:15.2397258Z }, 2024-06-26T05:24:15.2397527Z { 2024-06-26T05:24:15.2397986Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2398548Z "size": 1283, 2024-06-26T05:24:15.2399095Z "digest": "sha256:16d76a79d5d6616e228537785ec83a869f68a402520bbc6e826201621ca6cb98" 2024-06-26T05:24:15.2399724Z }, 2024-06-26T05:24:15.2399999Z { 2024-06-26T05:24:15.2400441Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2401022Z "size": 485, 2024-06-26T05:24:15.2401592Z "digest": "sha256:4bdf41c9ecbe1651e0986ccbea4663a0556af27b7145a679cfa45c35dc96b891" 2024-06-26T05:24:15.2402220Z }, 2024-06-26T05:24:15.2402491Z { 2024-06-26T05:24:15.2402939Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2403517Z "size": 91710937, 2024-06-26T05:24:15.2404101Z "digest": "sha256:f4863801e30cdbcd1d4e6cb1cbbcde7dbcaf6ae317ca80429295741ce60ac31d" 2024-06-26T05:24:15.2404746Z }, 2024-06-26T05:24:15.2405068Z { 2024-06-26T05:24:15.2405525Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2406169Z "size": 3233, 2024-06-26T05:24:15.2406750Z "digest": "sha256:d37dd9ed8948a0d02e1198bbedadbca36477a0980dd553196385aab0c025581d" 2024-06-26T05:24:15.2407393Z }, 2024-06-26T05:24:15.2407659Z { 2024-06-26T05:24:15.2408115Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2408680Z "size": 1905, 2024-06-26T05:24:15.2409250Z "digest": "sha256:cf2f34f225dbed17c1a9a8b9ea4ad030aba797774a716dc0a82fe212324aa751" 2024-06-26T05:24:15.2409886Z }, 2024-06-26T05:24:15.2410153Z { 2024-06-26T05:24:15.2410609Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2411182Z "size": 702, 2024-06-26T05:24:15.2411736Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T05:24:15.2412374Z }, 2024-06-26T05:24:15.2412643Z { 2024-06-26T05:24:15.2413100Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2413671Z "size": 2784789002, 2024-06-26T05:24:15.2414254Z "digest": "sha256:3d914950f8addfc4d657a2365c76e1143eb80b710cd3f6e9ae05640217fc6347" 2024-06-26T05:24:15.2414889Z }, 2024-06-26T05:24:15.2415142Z { 2024-06-26T05:24:15.2415600Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2416243Z "size": 381, 2024-06-26T05:24:15.2416798Z "digest": "sha256:96d0e2032184c57e62a652aaffcf0241f0d0c7a23e894ca0ad192c4a64a36583" 2024-06-26T05:24:15.2417431Z }, 2024-06-26T05:24:15.2417696Z { 2024-06-26T05:24:15.2418146Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2418712Z "size": 11879, 2024-06-26T05:24:15.2419264Z "digest": "sha256:06e12ce59790cb8b091e5d619c2a62a28329008ee184a8b9a562e3db12471f30" 2024-06-26T05:24:15.2419892Z }, 2024-06-26T05:24:15.2420171Z { 2024-06-26T05:24:15.2420623Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2421201Z "size": 803, 2024-06-26T05:24:15.2421784Z "digest": "sha256:ed712f63cde7070bdec2a6179c4c35e752c6991ddbc69fe9f2724a9903250ceb" 2024-06-26T05:24:15.2422428Z }, 2024-06-26T05:24:15.2422696Z { 2024-06-26T05:24:15.2423145Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2423743Z "size": 106, 2024-06-26T05:24:15.2424407Z "digest": "sha256:a97a71ef8a4e66234dfc2098834f8395ac5630240e60164d1a251fdb0115ba85" 2024-06-26T05:24:15.2425025Z }, 2024-06-26T05:24:15.2425301Z { 2024-06-26T05:24:15.2425753Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2426342Z "size": 503, 2024-06-26T05:24:15.2426889Z "digest": "sha256:25cd0c482ad612f936b35b4f220c580e16be982226aac8801a6a849a6bfdc221" 2024-06-26T05:24:15.2427536Z }, 2024-06-26T05:24:15.2427805Z { 2024-06-26T05:24:15.2428253Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2428840Z "size": 121477177, 2024-06-26T05:24:15.2429438Z "digest": "sha256:9bf94bc456584365fda21d0ed83b7cc690b08a71727b71c4c1e667e15028a386" 2024-06-26T05:24:15.2430079Z }, 2024-06-26T05:24:15.2430348Z { 2024-06-26T05:24:15.2430806Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2431391Z "size": 109, 2024-06-26T05:24:15.2431967Z "digest": "sha256:de45a14c9b227e4b0c9d45d42223b50a71c03522ddd9e2ca0d9517835206c823" 2024-06-26T05:24:15.2432618Z }, 2024-06-26T05:24:15.2432912Z { 2024-06-26T05:24:15.2433379Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2433947Z "size": 491, 2024-06-26T05:24:15.2434514Z "digest": "sha256:e99241e77209b0b35a0fb59a61c0821fc161190178af7e4af45e6eeaa043375e" 2024-06-26T05:24:15.2435154Z }, 2024-06-26T05:24:15.2435435Z { 2024-06-26T05:24:15.2435885Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2436468Z "size": 291, 2024-06-26T05:24:15.2437049Z "digest": "sha256:35e8f4e2a42f9df0c969dedd2199cf71719cf5ecfa676dfa014bbdeb4fdb295f" 2024-06-26T05:24:15.2437787Z }, 2024-06-26T05:24:15.2438043Z { 2024-06-26T05:24:15.2438497Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2439084Z "size": 101, 2024-06-26T05:24:15.2439658Z "digest": "sha256:a823ffa4c541394b52953391dc0a6867e26c37fe4a71fa022a2cc3f07f0ba70d" 2024-06-26T05:24:15.2440283Z }, 2024-06-26T05:24:15.2440551Z { 2024-06-26T05:24:15.2441015Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2441582Z "size": 1473, 2024-06-26T05:24:15.2442144Z "digest": "sha256:44ba272eae2064603605a6238b3f35cc6578859d660068e702a047a9cd8ab2fd" 2024-06-26T05:24:15.2442781Z }, 2024-06-26T05:24:15.2443055Z { 2024-06-26T05:24:15.2443503Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2444090Z "size": 446710781, 2024-06-26T05:24:15.2444680Z "digest": "sha256:da02d278e5240c285bc855ce4e42cd6121d5b903787f5c14f03e575ad1268650" 2024-06-26T05:24:15.2445375Z }, 2024-06-26T05:24:15.2445639Z { 2024-06-26T05:24:15.2446086Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2446656Z "size": 161, 2024-06-26T05:24:15.2447199Z "digest": "sha256:73a0ce681c295135456e190578733273276b38b4dd5babbb39298f384d43b1c6" 2024-06-26T05:24:15.2447915Z }, 2024-06-26T05:24:15.2448186Z { 2024-06-26T05:24:15.2448641Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2449213Z "size": 564, 2024-06-26T05:24:15.2449785Z "digest": "sha256:49737bd5ae709dbf7956e21fc8b05ba009e9e6c9b130b0126a73bd3d3f5efc78" 2024-06-26T05:24:15.2450429Z }, 2024-06-26T05:24:15.2450706Z { 2024-06-26T05:24:15.2451157Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2451743Z "size": 35874600, 2024-06-26T05:24:15.2452342Z "digest": "sha256:9cd4c745f8e4f9d5c9bb8050923575afad21f5b7852975a4a1981bc9eff282c5" 2024-06-26T05:24:15.2452995Z }, 2024-06-26T05:24:15.2453259Z { 2024-06-26T05:24:15.2453724Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2454320Z "size": 104, 2024-06-26T05:24:15.2454853Z "digest": "sha256:b2950a0365345d4c17a48627c905fd8930b3b3154c727f0526eb9ca547102bb7" 2024-06-26T05:24:15.2455488Z }, 2024-06-26T05:24:15.2455763Z { 2024-06-26T05:24:15.2456222Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2456800Z "size": 425, 2024-06-26T05:24:15.2457360Z "digest": "sha256:25e19d81d9fc081962ff4cbc09e098f4035419822712c19a410f5282300dfb9e" 2024-06-26T05:24:15.2457998Z }, 2024-06-26T05:24:15.2458267Z { 2024-06-26T05:24:15.2458718Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2459307Z "size": 20262084, 2024-06-26T05:24:15.2459897Z "digest": "sha256:60156b40fcbec2c25f0170b0a82fa8cd9303e0daae8bb1811fd0fc6562e88e8e" 2024-06-26T05:24:15.2460550Z }, 2024-06-26T05:24:15.2460812Z { 2024-06-26T05:24:15.2461289Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2461867Z "size": 439, 2024-06-26T05:24:15.2462413Z "digest": "sha256:7a868239227fd19b30a19927bbc8f058d247403fe5b343c55f4b4b8987918911" 2024-06-26T05:24:15.2463049Z }, 2024-06-26T05:24:15.2463350Z { 2024-06-26T05:24:15.2463833Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2464487Z "size": 702, 2024-06-26T05:24:15.2465042Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T05:24:15.2465693Z }, 2024-06-26T05:24:15.2465969Z { 2024-06-26T05:24:15.2466416Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2466998Z "size": 143, 2024-06-26T05:24:15.2467566Z "digest": "sha256:83911d7c7ecf11b5932bb7449975c52e7f737a5b867691935e1ccb85ceb25b52" 2024-06-26T05:24:15.2468193Z }, 2024-06-26T05:24:15.2468462Z { 2024-06-26T05:24:15.2468925Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2469596Z "size": 135, 2024-06-26T05:24:15.2470164Z "digest": "sha256:b3c5d66c679f58c71febdd502889bd8cd3d36e5001754cdf1e30eabe21135cce" 2024-06-26T05:24:15.2470815Z }, 2024-06-26T05:24:15.2471084Z { 2024-06-26T05:24:15.2471545Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2472117Z "size": 4911495247, 2024-06-26T05:24:15.2472701Z "digest": "sha256:84b2c7a899820f6acdb22148c5e8346d8b26c33269fd01842c9573515251e1e1" 2024-06-26T05:24:15.2473333Z }, 2024-06-26T05:24:15.2473654Z { 2024-06-26T05:24:15.2474117Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2474702Z "size": 190, 2024-06-26T05:24:15.2475292Z "digest": "sha256:c1068d03a811d8dead5da9e82fdc8869205fc4ee9cb92d4f7a4acfbb5ba83950" 2024-06-26T05:24:15.2475940Z }, 2024-06-26T05:24:15.2476204Z { 2024-06-26T05:24:15.2476660Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2477245Z "size": 565, 2024-06-26T05:24:15.2477803Z "digest": "sha256:671b4a301d81074fca17d36851f81801552c4ecb2f90ff3412aa9ab6a1a91371" 2024-06-26T05:24:15.2478447Z }, 2024-06-26T05:24:15.2478713Z { 2024-06-26T05:24:15.2479179Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2479818Z "size": 43163095, 2024-06-26T05:24:15.2480413Z "digest": "sha256:67760cf5ec46ab58d3ca98637fd9b26cb47a67446bfd0db176fe5e023606aa0d" 2024-06-26T05:24:15.2481059Z }, 2024-06-26T05:24:15.2481320Z { 2024-06-26T05:24:15.2481781Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2482370Z "size": 106, 2024-06-26T05:24:15.2482943Z "digest": "sha256:ddc9740bcea2b301e99b50525059aea8d9c7121a1c8eab566561a8968438a8c2" 2024-06-26T05:24:15.2483569Z }, 2024-06-26T05:24:15.2483844Z { 2024-06-26T05:24:15.2484294Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2484952Z "size": 1213, 2024-06-26T05:24:15.2485513Z "digest": "sha256:fb2e977685609fcc01de170ba7f50a573e73e6520e47253717e92a28952c5ee8" 2024-06-26T05:24:15.2486136Z }, 2024-06-26T05:24:15.2486406Z { 2024-06-26T05:24:15.2486859Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2487425Z "size": 702, 2024-06-26T05:24:15.2487992Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T05:24:15.2488626Z }, 2024-06-26T05:24:15.2488889Z { 2024-06-26T05:24:15.2489342Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2489909Z "size": 138, 2024-06-26T05:24:15.2490475Z "digest": "sha256:b599c5a8c7cb461cbd7b3c98b99960b8eb04506e05e79db1cc0cfadb14a47e91" 2024-06-26T05:24:15.2491103Z }, 2024-06-26T05:24:15.2491367Z { 2024-06-26T05:24:15.2491814Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2492381Z "size": 120, 2024-06-26T05:24:15.2492939Z "digest": "sha256:e08e88c1366827455ee7ddbf92b6ae615ceb3cbd3fb11cd03ddd31bd3fa94a9c" 2024-06-26T05:24:15.2493591Z }, 2024-06-26T05:24:15.2493854Z { 2024-06-26T05:24:15.2494292Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2494875Z "size": 1904157495, 2024-06-26T05:24:15.2495464Z "digest": "sha256:12ba7431ff38de8cf6af3052a2a52e79f4e712b8c99f5bafde3c20e63b1f570f" 2024-06-26T05:24:15.2496105Z }, 2024-06-26T05:24:15.2496360Z { 2024-06-26T05:24:15.2496812Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2497389Z "size": 172, 2024-06-26T05:24:15.2497953Z "digest": "sha256:cf7c2cb896cba6a594028cc0fd246e127a2f4481d48905a3aa12c5f5872332d4" 2024-06-26T05:24:15.2498581Z }, 2024-06-26T05:24:15.2498852Z { 2024-06-26T05:24:15.2499303Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2499871Z "size": 908, 2024-06-26T05:24:15.2500429Z "digest": "sha256:3a778ddfac2af5b3a11f49ddf2f3f1ad1731f74ba691f434ab69966d1e2a189f" 2024-06-26T05:24:15.2501195Z }, 2024-06-26T05:24:15.2501466Z { 2024-06-26T05:24:15.2501910Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2502495Z "size": 702, 2024-06-26T05:24:15.2503075Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T05:24:15.2503766Z }, 2024-06-26T05:24:15.2504026Z { 2024-06-26T05:24:15.2504575Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2505157Z "size": 134, 2024-06-26T05:24:15.2505712Z "digest": "sha256:236c8f0245f4b40616a4322eaf1e1b3e178d5990db1fcaba39622ea4e79c06a1" 2024-06-26T05:24:15.2506338Z }, 2024-06-26T05:24:15.2506612Z { 2024-06-26T05:24:15.2507073Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2507647Z "size": 32, 2024-06-26T05:24:15.2508219Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T05:24:15.2508868Z }, 2024-06-26T05:24:15.2509152Z { 2024-06-26T05:24:15.2509602Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2510192Z "size": 156, 2024-06-26T05:24:15.2510773Z "digest": "sha256:dda1ce302b9a9df0c8f1145d2e3eaa03c96909bb45a792451f1fcb555b06738d" 2024-06-26T05:24:15.2511486Z }, 2024-06-26T05:24:15.2511743Z { 2024-06-26T05:24:15.2512190Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2512758Z "size": 1841, 2024-06-26T05:24:15.2513318Z "digest": "sha256:e797fe86caef19319e03f8dca92cf96838ab3bb3644f4447c6dfc04230118afa" 2024-06-26T05:24:15.2513997Z }, 2024-06-26T05:24:15.2514260Z { 2024-06-26T05:24:15.2514712Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2515274Z "size": 7529784, 2024-06-26T05:24:15.2515839Z "digest": "sha256:967190076fc0ab2bb2fc0d74e58edd3d662a537d702ff3f8b6e876230154a16f" 2024-06-26T05:24:15.2516469Z }, 2024-06-26T05:24:15.2516733Z { 2024-06-26T05:24:15.2517188Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2517763Z "size": 165, 2024-06-26T05:24:15.2518316Z "digest": "sha256:4fb0a83f06855534ef82b80dcf7c685303edb2bcbd446196066dc2ad2dd71281" 2024-06-26T05:24:15.2518954Z }, 2024-06-26T05:24:15.2519211Z { 2024-06-26T05:24:15.2519657Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2520228Z "size": 7944, 2024-06-26T05:24:15.2520779Z "digest": "sha256:68af2ec4e30a33f26947ece9f3a31b7d3b04c2bca5865d4b48618457832a1da2" 2024-06-26T05:24:15.2521409Z }, 2024-06-26T05:24:15.2521671Z { 2024-06-26T05:24:15.2522126Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2522688Z "size": 8070, 2024-06-26T05:24:15.2523300Z "digest": "sha256:cc91933e985a044c6df53caa15488d2285896f5e991f38ec068c253ceec53f14" 2024-06-26T05:24:15.2523937Z }, 2024-06-26T05:24:15.2524207Z { 2024-06-26T05:24:15.2524661Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2525287Z "size": 303, 2024-06-26T05:24:15.2525864Z "digest": "sha256:9f3187916e7dcad33cc572f8f0e949d2f507b887f99cfd6acefd14826b44960d" 2024-06-26T05:24:15.2526503Z }, 2024-06-26T05:24:15.2526765Z { 2024-06-26T05:24:15.2527222Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2527808Z "size": 7629071, 2024-06-26T05:24:15.2528357Z "digest": "sha256:52608399485738d10d779254946c9e1d536010dd45593c69bf1eb9f62f85dc24" 2024-06-26T05:24:15.2528972Z }, 2024-06-26T05:24:15.2529244Z { 2024-06-26T05:24:15.2529707Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2530282Z "size": 108, 2024-06-26T05:24:15.2530847Z "digest": "sha256:05a6f6dcc1c00bac6660923a22adc7d7c34416718f8c89daf9b3351d9365f694" 2024-06-26T05:24:15.2531492Z }, 2024-06-26T05:24:15.2531766Z { 2024-06-26T05:24:15.2532220Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2532861Z "size": 54145775, 2024-06-26T05:24:15.2533447Z "digest": "sha256:6bc9dc8315b09b27c976bfba9700767ba412d72fd0528b18bbe8f8e62005ec22" 2024-06-26T05:24:15.2534086Z }, 2024-06-26T05:24:15.2534348Z { 2024-06-26T05:24:15.2534817Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2535395Z "size": 474, 2024-06-26T05:24:15.2535951Z "digest": "sha256:1b0e27c35f07b3fbf0aeb548d1949bf1d60a420136da21789d94d196c891d981" 2024-06-26T05:24:15.2536590Z }, 2024-06-26T05:24:15.2536859Z { 2024-06-26T05:24:15.2537323Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2537896Z "size": 1374858747, 2024-06-26T05:24:15.2538475Z "digest": "sha256:95237e7311a1c5ce6937db7196c866632f44e8788514a5c0eb4a3f04da4017e6" 2024-06-26T05:24:15.2539107Z }, 2024-06-26T05:24:15.2539380Z { 2024-06-26T05:24:15.2539830Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2540407Z "size": 106, 2024-06-26T05:24:15.2540999Z "digest": "sha256:789eff6c51aada70bf75eb1acafed63c0d839b768638001500616ff6e53b854a" 2024-06-26T05:24:15.2541634Z }, 2024-06-26T05:24:15.2541903Z { 2024-06-26T05:24:15.2542372Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2543008Z "size": 558, 2024-06-26T05:24:15.2543574Z "digest": "sha256:de0f8a8c45c6077b3c5ba3b84d7f07de98353fd618dad9a1e2f368ffe00493c7" 2024-06-26T05:24:15.2544334Z }, 2024-06-26T05:24:15.2544599Z { 2024-06-26T05:24:15.2545054Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2545630Z "size": 46248513, 2024-06-26T05:24:15.2546202Z "digest": "sha256:2060028f74888d218bebdd10d723f66c33ee1c708935c4ffd4654d34912ac4dc" 2024-06-26T05:24:15.2546822Z }, 2024-06-26T05:24:15.2547097Z { 2024-06-26T05:24:15.2547538Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2548113Z "size": 111, 2024-06-26T05:24:15.2548683Z "digest": "sha256:e412c7f3b2ded440992955bd92512345cbece8a8894cd3592f714ebc758ad1a0" 2024-06-26T05:24:15.2549305Z }, 2024-06-26T05:24:15.2549572Z { 2024-06-26T05:24:15.2550034Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2550625Z "size": 32, 2024-06-26T05:24:15.2551176Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T05:24:15.2551815Z }, 2024-06-26T05:24:15.2552085Z { 2024-06-26T05:24:15.2552539Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2553157Z "size": 32, 2024-06-26T05:24:15.2553720Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T05:24:15.2554371Z }, 2024-06-26T05:24:15.2554628Z { 2024-06-26T05:24:15.2555081Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2555664Z "size": 32, 2024-06-26T05:24:15.2556247Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T05:24:15.2556877Z }, 2024-06-26T05:24:15.2557143Z { 2024-06-26T05:24:15.2557606Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T05:24:15.2558192Z "size": 32, 2024-06-26T05:24:15.2558744Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T05:24:15.2559396Z } 2024-06-26T05:24:15.2559665Z ] 2024-06-26T05:24:15.2559920Z } 2024-06-26T05:24:15.2663998Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T05:24:15.2664655Z tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T05:24:15.2665221Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-06-26T05:24:15.2673205Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:24:15.2673729Z env: 2024-06-26T05:24:15.2674029Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:24:15.2675217Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T05:24:15.2676413Z ##[endgroup] 2024-06-26T05:24:15.2701635Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T05:24:15.2752422Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-06-26T05:24:15.2753027Z with: 2024-06-26T05:24:15.2754129Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T05:24:15.2755464Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T05:24:15.2756009Z env: 2024-06-26T05:24:15.2756307Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:24:15.2756671Z ##[endgroup] 2024-06-26T05:24:15.2772832Z ##[group]Run set -x 2024-06-26T05:24:15.2773197Z set -x 2024-06-26T05:24:15.2773512Z set +e 2024-06-26T05:24:15.2773835Z  2024-06-26T05:24:15.2774120Z login() { 2024-06-26T05:24:15.2774812Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T05:24:15.2775556Z } 2024-06-26T05:24:15.2775967Z  2024-06-26T05:24:15.2776293Z retry () { 2024-06-26T05:24:15.2776712Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T05:24:15.2777171Z } 2024-06-26T05:24:15.2777463Z  2024-06-26T05:24:15.2777800Z retry login "${DOCKER_REGISTRY}" 2024-06-26T05:24:15.2778234Z  2024-06-26T05:24:15.2778531Z set -e 2024-06-26T05:24:15.2779046Z # ignore output since only exit code is used for conditional 2024-06-26T05:24:15.2779791Z # only pull docker image if it's not available locally 2024-06-26T05:24:15.2780591Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-06-26T05:24:15.2781322Z  retry docker pull "${DOCKER_IMAGE}" 2024-06-26T05:24:15.2781794Z fi 2024-06-26T05:24:15.2790025Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:24:15.2790556Z env: 2024-06-26T05:24:15.2790858Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:24:15.2792046Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T05:24:15.2793390Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T05:24:15.2793928Z ##[endgroup] 2024-06-26T05:24:15.2814543Z + set +e 2024-06-26T05:24:15.2815180Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T05:24:15.2815836Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T05:24:15.2816760Z + aws ecr get-login-password --region us-east-1 2024-06-26T05:24:15.2817837Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T05:24:15.7107492Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T05:24:15.7109515Z Configure a credential helper to remove this warning. See 2024-06-26T05:24:15.7111202Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T05:24:15.7112301Z 2024-06-26T05:24:15.7112549Z Login Succeeded 2024-06-26T05:24:15.7120596Z + set -e 2024-06-26T05:24:15.7122027Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T05:24:15.7271182Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T05:24:15.7273343Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T05:24:15.9315200Z 91382da70d5719cd7007b6b80b71d2f48398f6b7: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-06-26T05:24:15.9316271Z 7a2c55901189: Pulling fs layer 2024-06-26T05:24:15.9316680Z 224fe954d725: Pulling fs layer 2024-06-26T05:24:15.9317358Z 75722010b82e: Pulling fs layer 2024-06-26T05:24:15.9317828Z d527cbbb87e3: Pulling fs layer 2024-06-26T05:24:15.9318220Z b57676e46aee: Pulling fs layer 2024-06-26T05:24:15.9318622Z a8c1e85b5e14: Pulling fs layer 2024-06-26T05:24:15.9319018Z a41a8d1c11c8: Pulling fs layer 2024-06-26T05:24:15.9319403Z 0c1227890755: Pulling fs layer 2024-06-26T05:24:15.9319805Z d8d1234baab3: Pulling fs layer 2024-06-26T05:24:15.9320209Z 7ed32bc8e469: Pulling fs layer 2024-06-26T05:24:15.9320607Z ec1e7978c1fe: Pulling fs layer 2024-06-26T05:24:15.9320989Z 309b872da805: Pulling fs layer 2024-06-26T05:24:15.9321377Z e82932cb002c: Pulling fs layer 2024-06-26T05:24:15.9321773Z ca9dd73e3e93: Pulling fs layer 2024-06-26T05:24:15.9322161Z 16d76a79d5d6: Pulling fs layer 2024-06-26T05:24:15.9322602Z 4bdf41c9ecbe: Pulling fs layer 2024-06-26T05:24:15.9322997Z f4863801e30c: Pulling fs layer 2024-06-26T05:24:15.9323389Z d37dd9ed8948: Pulling fs layer 2024-06-26T05:24:15.9323774Z a41a8d1c11c8: Waiting 2024-06-26T05:24:15.9324267Z cf2f34f225db: Pulling fs layer 2024-06-26T05:24:15.9324642Z 0c1227890755: Waiting 2024-06-26T05:24:15.9325103Z df546d7b3e34: Pulling fs layer 2024-06-26T05:24:15.9325478Z 309b872da805: Waiting 2024-06-26T05:24:15.9325827Z 3d914950f8ad: Pulling fs layer 2024-06-26T05:24:15.9326201Z b57676e46aee: Waiting 2024-06-26T05:24:15.9326534Z d8d1234baab3: Waiting 2024-06-26T05:24:15.9326875Z d527cbbb87e3: Waiting 2024-06-26T05:24:15.9327206Z a8c1e85b5e14: Waiting 2024-06-26T05:24:15.9327538Z e82932cb002c: Waiting 2024-06-26T05:24:15.9327872Z ca9dd73e3e93: Waiting 2024-06-26T05:24:15.9328194Z ec1e7978c1fe: Waiting 2024-06-26T05:24:15.9328540Z 96d0e2032184: Pulling fs layer 2024-06-26T05:24:15.9328946Z 06e12ce59790: Pulling fs layer 2024-06-26T05:24:15.9329330Z ed712f63cde7: Pulling fs layer 2024-06-26T05:24:15.9329728Z a97a71ef8a4e: Pulling fs layer 2024-06-26T05:24:15.9330123Z 25cd0c482ad6: Pulling fs layer 2024-06-26T05:24:15.9330515Z 9bf94bc45658: Pulling fs layer 2024-06-26T05:24:15.9330878Z 7ed32bc8e469: Waiting 2024-06-26T05:24:15.9331257Z de45a14c9b22: Pulling fs layer 2024-06-26T05:24:15.9331631Z cf2f34f225db: Waiting 2024-06-26T05:24:15.9331982Z e99241e77209: Pulling fs layer 2024-06-26T05:24:15.9332355Z 4bdf41c9ecbe: Waiting 2024-06-26T05:24:15.9332684Z 3d914950f8ad: Waiting 2024-06-26T05:24:15.9333009Z 96d0e2032184: Waiting 2024-06-26T05:24:15.9333348Z 35e8f4e2a42f: Pulling fs layer 2024-06-26T05:24:15.9333738Z a823ffa4c541: Pulling fs layer 2024-06-26T05:24:15.9334112Z d37dd9ed8948: Waiting 2024-06-26T05:24:15.9334467Z 44ba272eae20: Pulling fs layer 2024-06-26T05:24:15.9334836Z ed712f63cde7: Waiting 2024-06-26T05:24:15.9335189Z da02d278e524: Pulling fs layer 2024-06-26T05:24:15.9335568Z 9bf94bc45658: Waiting 2024-06-26T05:24:15.9335912Z 73a0ce681c29: Pulling fs layer 2024-06-26T05:24:15.9336286Z a97a71ef8a4e: Waiting 2024-06-26T05:24:15.9336632Z 49737bd5ae70: Pulling fs layer 2024-06-26T05:24:15.9337004Z de45a14c9b22: Waiting 2024-06-26T05:24:15.9337347Z 9cd4c745f8e4: Pulling fs layer 2024-06-26T05:24:15.9337740Z e99241e77209: Waiting 2024-06-26T05:24:15.9338068Z a823ffa4c541: Waiting 2024-06-26T05:24:15.9338412Z b2950a036534: Pulling fs layer 2024-06-26T05:24:15.9338773Z 44ba272eae20: Waiting 2024-06-26T05:24:15.9339119Z 25e19d81d9fc: Pulling fs layer 2024-06-26T05:24:15.9339515Z 60156b40fcbe: Pulling fs layer 2024-06-26T05:24:15.9339896Z 7a868239227f: Pulling fs layer 2024-06-26T05:24:15.9340266Z da02d278e524: Waiting 2024-06-26T05:24:15.9340613Z 83911d7c7ecf: Pulling fs layer 2024-06-26T05:24:15.9341006Z b3c5d66c679f: Pulling fs layer 2024-06-26T05:24:15.9341381Z f4863801e30c: Waiting 2024-06-26T05:24:15.9341728Z 84b2c7a89982: Pulling fs layer 2024-06-26T05:24:15.9342121Z c1068d03a811: Pulling fs layer 2024-06-26T05:24:15.9342497Z 7a868239227f: Waiting 2024-06-26T05:24:15.9342897Z 671b4a301d81: Pulling fs layer 2024-06-26T05:24:15.9343303Z 73a0ce681c29: Waiting 2024-06-26T05:24:15.9343654Z 67760cf5ec46: Pulling fs layer 2024-06-26T05:24:15.9344038Z 83911d7c7ecf: Waiting 2024-06-26T05:24:15.9344730Z b2950a036534: Waiting 2024-06-26T05:24:15.9345084Z 9cd4c745f8e4: Waiting 2024-06-26T05:24:15.9345437Z ddc9740bcea2: Pulling fs layer 2024-06-26T05:24:15.9345830Z fb2e97768560: Pulling fs layer 2024-06-26T05:24:15.9346215Z b599c5a8c7cb: Pulling fs layer 2024-06-26T05:24:15.9346588Z 84b2c7a89982: Waiting 2024-06-26T05:24:15.9346912Z 67760cf5ec46: Waiting 2024-06-26T05:24:15.9347232Z 671b4a301d81: Waiting 2024-06-26T05:24:15.9347559Z ddc9740bcea2: Waiting 2024-06-26T05:24:15.9347886Z fb2e97768560: Waiting 2024-06-26T05:24:15.9348225Z b599c5a8c7cb: Waiting 2024-06-26T05:24:15.9348545Z c1068d03a811: Waiting 2024-06-26T05:24:15.9348876Z b3c5d66c679f: Waiting 2024-06-26T05:24:15.9349218Z e08e88c13668: Pulling fs layer 2024-06-26T05:24:15.9349585Z e08e88c13668: Waiting 2024-06-26T05:24:15.9349932Z 12ba7431ff38: Pulling fs layer 2024-06-26T05:24:15.9350337Z cf7c2cb896cb: Pulling fs layer 2024-06-26T05:24:15.9350742Z 3a778ddfac2a: Pulling fs layer 2024-06-26T05:24:15.9351112Z 60156b40fcbe: Waiting 2024-06-26T05:24:15.9351539Z 3a778ddfac2a: Waiting 2024-06-26T05:24:15.9351896Z 236c8f0245f4: Pulling fs layer 2024-06-26T05:24:15.9352296Z 4f4fb700ef54: Pulling fs layer 2024-06-26T05:24:15.9352699Z dda1ce302b9a: Pulling fs layer 2024-06-26T05:24:15.9353138Z e797fe86caef: Pulling fs layer 2024-06-26T05:24:15.9353544Z 967190076fc0: Pulling fs layer 2024-06-26T05:24:15.9353928Z 4fb0a83f0685: Pulling fs layer 2024-06-26T05:24:15.9354336Z 68af2ec4e30a: Pulling fs layer 2024-06-26T05:24:15.9354724Z 236c8f0245f4: Waiting 2024-06-26T05:24:15.9355060Z 4f4fb700ef54: Waiting 2024-06-26T05:24:15.9355391Z dda1ce302b9a: Waiting 2024-06-26T05:24:15.9355729Z e797fe86caef: Waiting 2024-06-26T05:24:15.9356058Z cf7c2cb896cb: Waiting 2024-06-26T05:24:15.9356416Z cc91933e985a: Pulling fs layer 2024-06-26T05:24:15.9356810Z 9f3187916e7d: Pulling fs layer 2024-06-26T05:24:15.9357185Z 967190076fc0: Waiting 2024-06-26T05:24:15.9357533Z 526083994857: Pulling fs layer 2024-06-26T05:24:15.9357899Z 4fb0a83f0685: Waiting 2024-06-26T05:24:15.9358257Z 05a6f6dcc1c0: Pulling fs layer 2024-06-26T05:24:15.9358672Z 6bc9dc8315b0: Pulling fs layer 2024-06-26T05:24:15.9359053Z cc91933e985a: Waiting 2024-06-26T05:24:15.9359372Z 526083994857: Waiting 2024-06-26T05:24:15.9359718Z 1b0e27c35f07: Pulling fs layer 2024-06-26T05:24:15.9360106Z 35e8f4e2a42f: Waiting 2024-06-26T05:24:15.9360436Z 9f3187916e7d: Waiting 2024-06-26T05:24:15.9360793Z 95237e7311a1: Pulling fs layer 2024-06-26T05:24:15.9361176Z 05a6f6dcc1c0: Waiting 2024-06-26T05:24:15.9361508Z 1b0e27c35f07: Waiting 2024-06-26T05:24:15.9361861Z 789eff6c51aa: Pulling fs layer 2024-06-26T05:24:15.9362265Z de0f8a8c45c6: Pulling fs layer 2024-06-26T05:24:15.9362664Z 2060028f7488: Pulling fs layer 2024-06-26T05:24:15.9363050Z 6bc9dc8315b0: Waiting 2024-06-26T05:24:15.9363402Z e412c7f3b2de: Pulling fs layer 2024-06-26T05:24:15.9363792Z de0f8a8c45c6: Waiting 2024-06-26T05:24:15.9364124Z 2060028f7488: Waiting 2024-06-26T05:24:15.9364456Z 789eff6c51aa: Waiting 2024-06-26T05:24:15.9364857Z e412c7f3b2de: Waiting 2024-06-26T05:24:16.1105163Z 224fe954d725: Verifying Checksum 2024-06-26T05:24:16.1106026Z 224fe954d725: Download complete 2024-06-26T05:24:16.1862687Z d527cbbb87e3: Verifying Checksum 2024-06-26T05:24:16.1863380Z d527cbbb87e3: Download complete 2024-06-26T05:24:16.2616501Z b57676e46aee: Verifying Checksum 2024-06-26T05:24:16.2616958Z b57676e46aee: Download complete 2024-06-26T05:24:16.2667131Z 7a2c55901189: Verifying Checksum 2024-06-26T05:24:16.2667553Z 7a2c55901189: Download complete 2024-06-26T05:24:16.3276521Z a41a8d1c11c8: Verifying Checksum 2024-06-26T05:24:16.3276959Z a41a8d1c11c8: Download complete 2024-06-26T05:24:16.4088323Z 0c1227890755: Download complete 2024-06-26T05:24:16.4875607Z d8d1234baab3: Download complete 2024-06-26T05:24:16.5430556Z 75722010b82e: Verifying Checksum 2024-06-26T05:24:16.5431033Z 75722010b82e: Download complete 2024-06-26T05:24:16.6291291Z ec1e7978c1fe: Verifying Checksum 2024-06-26T05:24:16.6291810Z ec1e7978c1fe: Download complete 2024-06-26T05:24:16.7191512Z 309b872da805: Download complete 2024-06-26T05:24:17.2983011Z 7a2c55901189: Pull complete 2024-06-26T05:24:17.5944104Z 224fe954d725: Pull complete 2024-06-26T05:24:18.4935097Z 75722010b82e: Pull complete 2024-06-26T05:24:18.5342286Z d527cbbb87e3: Pull complete 2024-06-26T05:24:18.5828749Z b57676e46aee: Pull complete 2024-06-26T05:24:20.7743788Z e82932cb002c: Verifying Checksum 2024-06-26T05:24:20.7744428Z e82932cb002c: Download complete 2024-06-26T05:24:20.8558948Z ca9dd73e3e93: Verifying Checksum 2024-06-26T05:24:20.8559801Z ca9dd73e3e93: Download complete 2024-06-26T05:24:20.9401548Z 16d76a79d5d6: Verifying Checksum 2024-06-26T05:24:20.9401998Z 16d76a79d5d6: Download complete 2024-06-26T05:24:21.0197826Z 4bdf41c9ecbe: Verifying Checksum 2024-06-26T05:24:21.0198283Z 4bdf41c9ecbe: Download complete 2024-06-26T05:24:22.6150183Z f4863801e30c: Verifying Checksum 2024-06-26T05:24:22.6150611Z f4863801e30c: Download complete 2024-06-26T05:24:22.7096582Z d37dd9ed8948: Download complete 2024-06-26T05:24:22.8188420Z cf2f34f225db: Download complete 2024-06-26T05:24:22.9315663Z df546d7b3e34: Verifying Checksum 2024-06-26T05:24:22.9316195Z df546d7b3e34: Download complete 2024-06-26T05:24:29.6056304Z a8c1e85b5e14: Verifying Checksum 2024-06-26T05:24:29.6057946Z a8c1e85b5e14: Download complete 2024-06-26T05:24:29.6733113Z 96d0e2032184: Verifying Checksum 2024-06-26T05:24:29.6733698Z 96d0e2032184: Download complete 2024-06-26T05:24:29.7553075Z 06e12ce59790: Verifying Checksum 2024-06-26T05:24:29.7553613Z 06e12ce59790: Download complete 2024-06-26T05:24:29.8406088Z ed712f63cde7: Verifying Checksum 2024-06-26T05:24:29.8406553Z ed712f63cde7: Download complete 2024-06-26T05:24:29.9280115Z a97a71ef8a4e: Verifying Checksum 2024-06-26T05:24:29.9280690Z a97a71ef8a4e: Download complete 2024-06-26T05:24:30.0064673Z 25cd0c482ad6: Verifying Checksum 2024-06-26T05:24:30.0065179Z 25cd0c482ad6: Download complete 2024-06-26T05:24:33.1615623Z 9bf94bc45658: Verifying Checksum 2024-06-26T05:24:33.1616542Z 9bf94bc45658: Download complete 2024-06-26T05:24:33.2223195Z de45a14c9b22: Verifying Checksum 2024-06-26T05:24:33.2223822Z de45a14c9b22: Download complete 2024-06-26T05:24:33.2963450Z e99241e77209: Verifying Checksum 2024-06-26T05:24:33.2964053Z e99241e77209: Download complete 2024-06-26T05:24:33.3758816Z 35e8f4e2a42f: Verifying Checksum 2024-06-26T05:24:33.3759898Z 35e8f4e2a42f: Download complete 2024-06-26T05:24:33.4617720Z a823ffa4c541: Verifying Checksum 2024-06-26T05:24:33.4618160Z a823ffa4c541: Download complete 2024-06-26T05:24:33.5447206Z 44ba272eae20: Verifying Checksum 2024-06-26T05:24:33.5447709Z 44ba272eae20: Download complete 2024-06-26T05:24:41.6749706Z da02d278e524: Download complete 2024-06-26T05:24:41.7495666Z 73a0ce681c29: Download complete 2024-06-26T05:24:41.8353869Z 7ed32bc8e469: Verifying Checksum 2024-06-26T05:24:41.8354306Z 7ed32bc8e469: Download complete 2024-06-26T05:24:41.9626025Z b2950a036534: Verifying Checksum 2024-06-26T05:24:41.9626576Z b2950a036534: Download complete 2024-06-26T05:24:42.0021184Z a8c1e85b5e14: Pull complete 2024-06-26T05:24:42.0335603Z 25e19d81d9fc: Verifying Checksum 2024-06-26T05:24:42.0336565Z 25e19d81d9fc: Download complete 2024-06-26T05:24:42.0503445Z a41a8d1c11c8: Pull complete 2024-06-26T05:24:42.0909967Z 0c1227890755: Pull complete 2024-06-26T05:24:42.1351762Z d8d1234baab3: Pull complete 2024-06-26T05:24:42.5396504Z 60156b40fcbe: Verifying Checksum 2024-06-26T05:24:42.5396998Z 60156b40fcbe: Download complete 2024-06-26T05:24:42.6391342Z 7a868239227f: Verifying Checksum 2024-06-26T05:24:42.6392027Z 7a868239227f: Download complete 2024-06-26T05:24:42.6417991Z 9cd4c745f8e4: Download complete 2024-06-26T05:24:42.7131777Z b3c5d66c679f: Verifying Checksum 2024-06-26T05:24:42.7132450Z b3c5d66c679f: Download complete 2024-06-26T05:24:42.7223936Z 83911d7c7ecf: Verifying Checksum 2024-06-26T05:24:42.7224513Z 83911d7c7ecf: Download complete 2024-06-26T05:24:42.7946614Z c1068d03a811: Verifying Checksum 2024-06-26T05:24:42.7947481Z c1068d03a811: Download complete 2024-06-26T05:24:42.8722980Z 671b4a301d81: Verifying Checksum 2024-06-26T05:24:42.8723709Z 671b4a301d81: Download complete 2024-06-26T05:24:44.1043186Z 67760cf5ec46: Verifying Checksum 2024-06-26T05:24:44.1043682Z 67760cf5ec46: Download complete 2024-06-26T05:24:44.1931025Z ddc9740bcea2: Verifying Checksum 2024-06-26T05:24:44.1931644Z ddc9740bcea2: Download complete 2024-06-26T05:24:44.2699810Z fb2e97768560: Download complete 2024-06-26T05:24:44.3415489Z b599c5a8c7cb: Verifying Checksum 2024-06-26T05:24:44.3415955Z b599c5a8c7cb: Download complete 2024-06-26T05:24:44.4152542Z e08e88c13668: Download complete 2024-06-26T05:25:11.8041269Z 7ed32bc8e469: Pull complete 2024-06-26T05:25:12.1924162Z ec1e7978c1fe: Pull complete 2024-06-26T05:25:12.5644941Z 309b872da805: Pull complete 2024-06-26T05:25:12.6225045Z 3d914950f8ad: Verifying Checksum 2024-06-26T05:25:12.6226563Z 3d914950f8ad: Download complete 2024-06-26T05:25:12.6998063Z cf7c2cb896cb: Download complete 2024-06-26T05:25:12.8387964Z 3a778ddfac2a: Verifying Checksum 2024-06-26T05:25:12.8388843Z 3a778ddfac2a: Download complete 2024-06-26T05:25:12.9176205Z 236c8f0245f4: Verifying Checksum 2024-06-26T05:25:12.9176640Z 236c8f0245f4: Download complete 2024-06-26T05:25:12.9247407Z 4f4fb700ef54: Download complete 2024-06-26T05:25:13.0093706Z dda1ce302b9a: Verifying Checksum 2024-06-26T05:25:13.0094230Z dda1ce302b9a: Download complete 2024-06-26T05:25:13.0830586Z e797fe86caef: Download complete 2024-06-26T05:25:13.4829059Z 967190076fc0: Verifying Checksum 2024-06-26T05:25:13.4829694Z 967190076fc0: Download complete 2024-06-26T05:25:13.5638220Z 4fb0a83f0685: Verifying Checksum 2024-06-26T05:25:13.5638804Z 4fb0a83f0685: Download complete 2024-06-26T05:25:13.6326184Z 68af2ec4e30a: Verifying Checksum 2024-06-26T05:25:13.6326702Z 68af2ec4e30a: Download complete 2024-06-26T05:25:13.6958545Z cc91933e985a: Download complete 2024-06-26T05:25:13.7858562Z 9f3187916e7d: Verifying Checksum 2024-06-26T05:25:13.7859143Z 9f3187916e7d: Download complete 2024-06-26T05:25:14.2283650Z 526083994857: Download complete 2024-06-26T05:25:14.3019834Z 05a6f6dcc1c0: Verifying Checksum 2024-06-26T05:25:14.3020367Z 05a6f6dcc1c0: Download complete 2024-06-26T05:25:15.2857176Z 6bc9dc8315b0: Verifying Checksum 2024-06-26T05:25:15.2857624Z 6bc9dc8315b0: Download complete 2024-06-26T05:25:15.3613342Z 1b0e27c35f07: Verifying Checksum 2024-06-26T05:25:15.3613777Z 1b0e27c35f07: Download complete 2024-06-26T05:25:19.9035209Z 12ba7431ff38: Verifying Checksum 2024-06-26T05:25:19.9035847Z 12ba7431ff38: Download complete 2024-06-26T05:25:19.9673903Z 789eff6c51aa: Verifying Checksum 2024-06-26T05:25:19.9674357Z 789eff6c51aa: Download complete 2024-06-26T05:25:20.0837445Z de0f8a8c45c6: Download complete 2024-06-26T05:25:20.2891131Z e82932cb002c: Pull complete 2024-06-26T05:25:20.6513882Z ca9dd73e3e93: Pull complete 2024-06-26T05:25:20.9024015Z 2060028f7488: Verifying Checksum 2024-06-26T05:25:20.9024625Z 2060028f7488: Download complete 2024-06-26T05:25:20.9739213Z e412c7f3b2de: Verifying Checksum 2024-06-26T05:25:20.9739712Z e412c7f3b2de: Download complete 2024-06-26T05:25:21.1199866Z 16d76a79d5d6: Pull complete 2024-06-26T05:25:21.4971420Z 4bdf41c9ecbe: Pull complete 2024-06-26T05:25:24.1409167Z f4863801e30c: Pull complete 2024-06-26T05:25:24.5199447Z d37dd9ed8948: Pull complete 2024-06-26T05:25:24.9068736Z cf2f34f225db: Pull complete 2024-06-26T05:25:25.2951864Z df546d7b3e34: Pull complete 2024-06-26T05:25:41.0708697Z 95237e7311a1: Verifying Checksum 2024-06-26T05:25:41.0709571Z 95237e7311a1: Download complete 2024-06-26T05:26:13.3676598Z 84b2c7a89982: Download complete 2024-06-26T05:26:15.3185014Z 3d914950f8ad: Pull complete 2024-06-26T05:26:15.6948225Z 96d0e2032184: Pull complete 2024-06-26T05:26:16.1153610Z 06e12ce59790: Pull complete 2024-06-26T05:26:16.4876938Z ed712f63cde7: Pull complete 2024-06-26T05:26:16.8482287Z a97a71ef8a4e: Pull complete 2024-06-26T05:26:17.2126210Z 25cd0c482ad6: Pull complete 2024-06-26T05:26:20.9503943Z 9bf94bc45658: Pull complete 2024-06-26T05:26:21.3354404Z de45a14c9b22: Pull complete 2024-06-26T05:26:21.7165255Z e99241e77209: Pull complete 2024-06-26T05:26:22.1405911Z 35e8f4e2a42f: Pull complete 2024-06-26T05:26:22.5309661Z a823ffa4c541: Pull complete 2024-06-26T05:26:22.9024876Z 44ba272eae20: Pull complete 2024-06-26T05:26:31.5815703Z da02d278e524: Pull complete 2024-06-26T05:26:31.9481640Z 73a0ce681c29: Pull complete 2024-06-26T05:26:32.3385297Z 49737bd5ae70: Pull complete 2024-06-26T05:26:33.5033037Z 9cd4c745f8e4: Pull complete 2024-06-26T05:26:33.8658597Z b2950a036534: Pull complete 2024-06-26T05:26:34.2417640Z 25e19d81d9fc: Pull complete 2024-06-26T05:26:34.9102012Z 60156b40fcbe: Pull complete 2024-06-26T05:26:35.3103901Z 7a868239227f: Pull complete 2024-06-26T05:26:36.0608412Z 83911d7c7ecf: Pull complete 2024-06-26T05:26:36.4235834Z b3c5d66c679f: Pull complete 2024-06-26T05:27:15.6140945Z 84b2c7a89982: Pull complete 2024-06-26T05:27:15.9663651Z c1068d03a811: Pull complete 2024-06-26T05:27:16.3085796Z 671b4a301d81: Pull complete 2024-06-26T05:27:18.6642719Z 67760cf5ec46: Pull complete 2024-06-26T05:27:19.0488274Z ddc9740bcea2: Pull complete 2024-06-26T05:27:19.4227873Z fb2e97768560: Pull complete 2024-06-26T05:27:20.0974629Z b599c5a8c7cb: Pull complete 2024-06-26T05:27:20.4216516Z e08e88c13668: Pull complete 2024-06-26T05:27:55.3705111Z 12ba7431ff38: Pull complete 2024-06-26T05:27:55.7487193Z cf7c2cb896cb: Pull complete 2024-06-26T05:27:56.1147785Z 3a778ddfac2a: Pull complete 2024-06-26T05:27:56.9355823Z 236c8f0245f4: Pull complete 2024-06-26T05:27:57.3312299Z 4f4fb700ef54: Pull complete 2024-06-26T05:27:57.7196640Z dda1ce302b9a: Pull complete 2024-06-26T05:27:58.1159483Z e797fe86caef: Pull complete 2024-06-26T05:27:58.6758184Z 967190076fc0: Pull complete 2024-06-26T05:27:59.0916022Z 4fb0a83f0685: Pull complete 2024-06-26T05:27:59.4898837Z 68af2ec4e30a: Pull complete 2024-06-26T05:27:59.9040438Z cc91933e985a: Pull complete 2024-06-26T05:28:00.3204182Z 9f3187916e7d: Pull complete 2024-06-26T05:28:01.6313483Z 526083994857: Pull complete 2024-06-26T05:28:01.9984758Z 05a6f6dcc1c0: Pull complete 2024-06-26T05:28:04.2643555Z 6bc9dc8315b0: Pull complete 2024-06-26T05:28:04.6102266Z 1b0e27c35f07: Pull complete 2024-06-26T05:28:18.4146317Z 95237e7311a1: Pull complete 2024-06-26T05:28:18.8184035Z 789eff6c51aa: Pull complete 2024-06-26T05:28:19.2268424Z de0f8a8c45c6: Pull complete 2024-06-26T05:28:20.3122380Z 2060028f7488: Pull complete 2024-06-26T05:28:20.7324776Z e412c7f3b2de: Pull complete 2024-06-26T05:28:22.5462101Z Digest: sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T05:28:22.6261646Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T05:28:22.6723098Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T05:28:22.6768300Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T05:28:22.6769266Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T05:28:22.6777417Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:28:22.6777931Z env: 2024-06-26T05:28:22.6778233Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:28:22.6778601Z ##[endgroup] 2024-06-26T05:28:22.7048462Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-06-26T05:28:22.7049039Z with: 2024-06-26T05:28:22.7049343Z driver-version: 550.54.15 2024-06-26T05:28:22.7049705Z env: 2024-06-26T05:28:22.7050006Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:28:22.7050381Z ##[endgroup] 2024-06-26T05:28:22.7292289Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T05:28:22.7292927Z with: 2024-06-26T05:28:22.7293236Z timeout_minutes: 10 2024-06-26T05:28:22.7293588Z max_attempts: 3 2024-06-26T05:28:22.7326629Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-06-26T05:28:22.7358675Z retry_wait_seconds: 10 2024-06-26T05:28:22.7359066Z polling_interval_seconds: 1 2024-06-26T05:28:22.7359469Z warning_on_retry: true 2024-06-26T05:28:22.7359858Z continue_on_error: false 2024-06-26T05:28:22.7360220Z env: 2024-06-26T05:28:22.7360525Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:28:22.7360915Z DRIVER_VERSION: 550.54.15 2024-06-26T05:28:22.7361300Z ##[endgroup] 2024-06-26T05:28:22.7899086Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-06-26T05:28:22.7899864Z + pre_install_nvidia_driver_amzn2 2024-06-26T05:28:22.7900397Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-06-26T05:28:23.0938121Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T05:28:23.1327567Z No Match for argument: nvidia-driver-latest-dkms 2024-06-26T05:28:23.1570074Z No Packages marked for removal 2024-06-26T05:28:23.1705213Z + install_nvidia_driver_common 2024-06-26T05:28:23.1708619Z + echo 'Before installing NVIDIA driver' 2024-06-26T05:28:23.1709071Z + lspci 2024-06-26T05:28:23.1709406Z Before installing NVIDIA driver 2024-06-26T05:28:23.1793004Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T05:28:23.1793784Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T05:28:23.1794740Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T05:28:23.1795565Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T05:28:23.1796341Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T05:28:23.1797126Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T05:28:23.1798076Z 00:1b.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T05:28:23.1798877Z 00:1c.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T05:28:23.1799546Z 00:1d.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T05:28:23.1800204Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T05:28:23.1801030Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T05:28:23.1801646Z + lsmod 2024-06-26T05:28:23.1807376Z Module Size Used by 2024-06-26T05:28:23.1807806Z ib_core 266240 0 2024-06-26T05:28:23.1808455Z nvidia_modeset 1339392 0 2024-06-26T05:28:23.1809004Z veth 16384 0 2024-06-26T05:28:23.1809443Z nvidia_uvm 4599808 0 2024-06-26T05:28:23.1809923Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T05:28:23.1810439Z drm 421888 1 nvidia 2024-06-26T05:28:23.1810882Z i2c_core 77824 2 nvidia,drm 2024-06-26T05:28:23.1811430Z backlight 16384 1 nvidia_modeset 2024-06-26T05:28:23.1811923Z xt_conntrack 16384 1 2024-06-26T05:28:23.1812323Z ipt_MASQUERADE 16384 1 2024-06-26T05:28:23.1812777Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T05:28:23.1813261Z nf_conntrack_netlink 49152 0 2024-06-26T05:28:23.1813737Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T05:28:23.1814223Z xfrm_user 45056 1 2024-06-26T05:28:23.1814672Z xfrm_algo 16384 1 xfrm_user 2024-06-26T05:28:23.1815227Z iptable_nat 16384 1 2024-06-26T05:28:23.1815630Z nf_conntrack_ipv4 16384 3 2024-06-26T05:28:23.1816088Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T05:28:23.1816686Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T05:28:23.1817233Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-06-26T05:28:23.1818200Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-06-26T05:28:23.1819066Z xt_addrtype 16384 2 2024-06-26T05:28:23.1819466Z iptable_filter 16384 1 2024-06-26T05:28:23.1819863Z br_netfilter 24576 0 2024-06-26T05:28:23.1820279Z bridge 172032 1 br_netfilter 2024-06-26T05:28:23.1820887Z stp 16384 1 bridge 2024-06-26T05:28:23.1821330Z llc 16384 2 bridge,stp 2024-06-26T05:28:23.1821767Z overlay 86016 0 2024-06-26T05:28:23.1822151Z sunrpc 393216 1 2024-06-26T05:28:23.1822542Z dm_mirror 28672 0 2024-06-26T05:28:23.1822961Z dm_region_hash 20480 1 dm_mirror 2024-06-26T05:28:23.1823464Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T05:28:23.1823994Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T05:28:23.1824674Z dax 69632 1 dm_mod 2024-06-26T05:28:23.1825098Z crc32_pclmul 16384 0 2024-06-26T05:28:23.1825489Z ghash_clmulni_intel 16384 0 2024-06-26T05:28:23.1825886Z pcbc 16384 0 2024-06-26T05:28:23.1826272Z mousedev 24576 0 2024-06-26T05:28:23.1826660Z aesni_intel 188416 0 2024-06-26T05:28:23.1827067Z aes_x86_64 20480 1 aesni_intel 2024-06-26T05:28:23.1827548Z crypto_simd 16384 1 aesni_intel 2024-06-26T05:28:23.1828024Z glue_helper 16384 1 aesni_intel 2024-06-26T05:28:23.1828465Z psmouse 32768 0 2024-06-26T05:28:23.1828854Z evdev 20480 3 2024-06-26T05:28:23.1829386Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T05:28:23.1829949Z button 16384 0 2024-06-26T05:28:23.1830327Z ena 139264 0 2024-06-26T05:28:23.1830717Z ptp 20480 1 ena 2024-06-26T05:28:23.1831132Z pps_core 20480 1 ptp 2024-06-26T05:28:23.1831545Z crc32c_intel 24576 0 2024-06-26T05:28:23.1832007Z autofs4 49152 2 2024-06-26T05:28:23.1832386Z + modinfo nvidia 2024-06-26T05:28:23.1833100Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T05:28:23.1833871Z alias: char-major-195-* 2024-06-26T05:28:23.1834266Z version: 550.54.15 2024-06-26T05:28:23.1834637Z supported: external 2024-06-26T05:28:23.1834996Z license: NVIDIA 2024-06-26T05:28:23.1835395Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T05:28:23.1835949Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T05:28:23.1836524Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T05:28:23.1837015Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T05:28:23.1837535Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T05:28:23.1838034Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T05:28:23.1838546Z depends: i2c-core,drm 2024-06-26T05:28:23.1838925Z retpoline: Y 2024-06-26T05:28:23.1839244Z name: nvidia 2024-06-26T05:28:23.1839819Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T05:28:23.1840501Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T05:28:23.1841174Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T05:28:23.1841789Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T05:28:23.1842254Z parm: NVreg_RmLogonRC:int 2024-06-26T05:28:23.1842707Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T05:28:23.1843186Z parm: NVreg_DeviceFileUID:int 2024-06-26T05:28:23.1843641Z parm: NVreg_DeviceFileGID:int 2024-06-26T05:28:23.1844106Z parm: NVreg_DeviceFileMode:int 2024-06-26T05:28:23.1844748Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T05:28:23.1845338Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T05:28:23.1845831Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T05:28:23.1846284Z parm: NVreg_EnableMSI:int 2024-06-26T05:28:23.1846723Z parm: NVreg_TCEBypassMode:int 2024-06-26T05:28:23.1847202Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T05:28:23.1847748Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T05:28:23.1848351Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T05:28:23.1849024Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T05:28:23.1849657Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T05:28:23.1850269Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T05:28:23.1850905Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T05:28:23.1851516Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T05:28:23.1852027Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T05:28:23.1852570Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T05:28:23.1853140Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T05:28:23.1853655Z parm: NVreg_MemoryPoolSize:int 2024-06-26T05:28:23.1854132Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T05:28:23.1854637Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T05:28:23.1855127Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T05:28:23.1855594Z parm: NVreg_NvLinkDisable:int 2024-06-26T05:28:23.1856117Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T05:28:23.1856662Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T05:28:23.1857162Z parm: NVreg_EnableResizableBar:int 2024-06-26T05:28:23.1857667Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T05:28:23.1858183Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T05:28:23.1858692Z parm: NVreg_RegistryDwords:charp 2024-06-26T05:28:23.1859216Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T05:28:23.1859714Z parm: NVreg_RmMsg:charp 2024-06-26T05:28:23.1860138Z parm: NVreg_GpuBlacklist:charp 2024-06-26T05:28:23.1860625Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T05:28:23.1861113Z parm: NVreg_ExcludedGpus:charp 2024-06-26T05:28:23.1861598Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T05:28:23.1862089Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T05:28:23.1862588Z parm: NVreg_ImexChannelCount:int 2024-06-26T05:28:23.1863069Z parm: rm_firmware_active:charp 2024-06-26T05:28:23.1863498Z + HAS_NVIDIA_DRIVER=0 2024-06-26T05:28:23.1863898Z ++ command -v nvidia-smi 2024-06-26T05:28:23.1864435Z + '[' -x /usr/bin/nvidia-smi ']' 2024-06-26T05:28:23.1864825Z + set +e 2024-06-26T05:28:23.1865414Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-06-26T05:28:31.1551538Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-06-26T05:28:31.1552177Z + NVIDIA_SMI_STATUS=0 2024-06-26T05:28:31.1552702Z + '[' 0 -ne 0 ']' 2024-06-26T05:28:31.1553125Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-06-26T05:28:31.1553630Z + HAS_NVIDIA_DRIVER=1 2024-06-26T05:28:31.1554372Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-06-26T05:28:31.1555095Z + set -e 2024-06-26T05:28:31.1555406Z + '[' 1 -eq 0 ']' 2024-06-26T05:28:31.1555754Z + post_install_nvidia_driver_common 2024-06-26T05:28:31.1556436Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-06-26T05:28:31.1557148Z + sudo modprobe nvidia 2024-06-26T05:28:31.1651829Z + echo 'After installing NVIDIA driver' 2024-06-26T05:28:31.1652418Z + lspci 2024-06-26T05:28:31.1652848Z After installing NVIDIA driver 2024-06-26T05:28:31.1736632Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T05:28:31.1737675Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T05:28:31.1738633Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T05:28:31.1739455Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T05:28:31.1740226Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T05:28:31.1740998Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T05:28:31.1741734Z 00:1b.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T05:28:31.1742401Z 00:1c.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T05:28:31.1743308Z 00:1d.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T05:28:31.1743972Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T05:28:31.1745035Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T05:28:31.1745648Z + lsmod 2024-06-26T05:28:31.1751374Z Module Size Used by 2024-06-26T05:28:31.1751834Z ib_core 266240 0 2024-06-26T05:28:31.1752357Z nvidia_modeset 1339392 0 2024-06-26T05:28:31.1752880Z veth 16384 0 2024-06-26T05:28:31.1753366Z nvidia_uvm 4599808 0 2024-06-26T05:28:31.1753852Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T05:28:31.1754343Z drm 421888 1 nvidia 2024-06-26T05:28:31.1754780Z i2c_core 77824 2 nvidia,drm 2024-06-26T05:28:31.1755260Z backlight 16384 1 nvidia_modeset 2024-06-26T05:28:31.1755717Z xt_conntrack 16384 1 2024-06-26T05:28:31.1756117Z ipt_MASQUERADE 16384 1 2024-06-26T05:28:31.1756570Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T05:28:31.1757061Z nf_conntrack_netlink 49152 0 2024-06-26T05:28:31.1757523Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T05:28:31.1757998Z xfrm_user 45056 1 2024-06-26T05:28:31.1758404Z xfrm_algo 16384 1 xfrm_user 2024-06-26T05:28:31.1758831Z iptable_nat 16384 1 2024-06-26T05:28:31.1759220Z nf_conntrack_ipv4 16384 3 2024-06-26T05:28:31.1759657Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T05:28:31.1760155Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T05:28:31.1760722Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-06-26T05:28:31.1761884Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-06-26T05:28:31.1762746Z xt_addrtype 16384 2 2024-06-26T05:28:31.1763148Z iptable_filter 16384 1 2024-06-26T05:28:31.1763603Z br_netfilter 24576 0 2024-06-26T05:28:31.1764126Z bridge 172032 1 br_netfilter 2024-06-26T05:28:31.1764752Z stp 16384 1 bridge 2024-06-26T05:28:31.1765347Z llc 16384 2 bridge,stp 2024-06-26T05:28:31.1765868Z overlay 86016 0 2024-06-26T05:28:31.1766317Z sunrpc 393216 1 2024-06-26T05:28:31.1766752Z dm_mirror 28672 0 2024-06-26T05:28:31.1767213Z dm_region_hash 20480 1 dm_mirror 2024-06-26T05:28:31.1767768Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T05:28:31.1768407Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T05:28:31.1769067Z dax 69632 1 dm_mod 2024-06-26T05:28:31.1769534Z crc32_pclmul 16384 0 2024-06-26T05:28:31.1769963Z ghash_clmulni_intel 16384 0 2024-06-26T05:28:31.1770419Z pcbc 16384 0 2024-06-26T05:28:31.1770849Z mousedev 24576 0 2024-06-26T05:28:31.1771290Z aesni_intel 188416 0 2024-06-26T05:28:31.1771767Z aes_x86_64 20480 1 aesni_intel 2024-06-26T05:28:31.1772356Z crypto_simd 16384 1 aesni_intel 2024-06-26T05:28:31.1772898Z glue_helper 16384 1 aesni_intel 2024-06-26T05:28:31.1773392Z psmouse 32768 0 2024-06-26T05:28:31.1773845Z evdev 20480 3 2024-06-26T05:28:31.1774429Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T05:28:31.1775095Z button 16384 0 2024-06-26T05:28:31.1775533Z ena 139264 0 2024-06-26T05:28:31.1775999Z ptp 20480 1 ena 2024-06-26T05:28:31.1776489Z pps_core 20480 1 ptp 2024-06-26T05:28:31.1776958Z crc32c_intel 24576 0 2024-06-26T05:28:31.1777377Z autofs4 49152 2 2024-06-26T05:28:31.1777860Z + modinfo nvidia 2024-06-26T05:28:31.1778599Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T05:28:31.1779514Z alias: char-major-195-* 2024-06-26T05:28:31.1779951Z version: 550.54.15 2024-06-26T05:28:31.1780380Z supported: external 2024-06-26T05:28:31.1780838Z license: NVIDIA 2024-06-26T05:28:31.1781270Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T05:28:31.1781838Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T05:28:31.1782391Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T05:28:31.1782945Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T05:28:31.1783498Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T05:28:31.1784113Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T05:28:31.1808872Z depends: i2c-core,drm 2024-06-26T05:28:31.1809336Z retpoline: Y 2024-06-26T05:28:31.1809671Z name: nvidia 2024-06-26T05:28:31.1810298Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T05:28:31.1810986Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T05:28:31.1811680Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T05:28:31.1812295Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T05:28:31.1812762Z parm: NVreg_RmLogonRC:int 2024-06-26T05:28:31.1813225Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T05:28:31.1813709Z parm: NVreg_DeviceFileUID:int 2024-06-26T05:28:31.1814173Z parm: NVreg_DeviceFileGID:int 2024-06-26T05:28:31.1814634Z parm: NVreg_DeviceFileMode:int 2024-06-26T05:28:31.1815186Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T05:28:31.1815813Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T05:28:31.1816310Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T05:28:31.1816769Z parm: NVreg_EnableMSI:int 2024-06-26T05:28:31.1817207Z parm: NVreg_TCEBypassMode:int 2024-06-26T05:28:31.1817714Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T05:28:31.1818281Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T05:28:31.1818883Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T05:28:31.1819472Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T05:28:31.1820290Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T05:28:31.1820927Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T05:28:31.1821567Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T05:28:31.1822181Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T05:28:31.1822691Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T05:28:31.1823250Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T05:28:31.1823825Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T05:28:31.1824616Z parm: NVreg_MemoryPoolSize:int 2024-06-26T05:28:31.1825934Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T05:28:31.1826517Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T05:28:31.1827046Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T05:28:31.1827519Z parm: NVreg_NvLinkDisable:int 2024-06-26T05:28:31.1828047Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T05:28:31.1828627Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T05:28:31.1829137Z parm: NVreg_EnableResizableBar:int 2024-06-26T05:28:31.1829657Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T05:28:31.1830174Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T05:28:31.1830691Z parm: NVreg_RegistryDwords:charp 2024-06-26T05:28:31.1831280Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T05:28:31.1831979Z parm: NVreg_RmMsg:charp 2024-06-26T05:28:31.1832544Z parm: NVreg_GpuBlacklist:charp 2024-06-26T05:28:31.1839239Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T05:28:31.1839750Z parm: NVreg_ExcludedGpus:charp 2024-06-26T05:28:31.1840217Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T05:28:31.1840990Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T05:28:31.1841519Z parm: NVreg_ImexChannelCount:int 2024-06-26T05:28:31.1862719Z parm: rm_firmware_active:charp 2024-06-26T05:28:31.1863225Z + set +e 2024-06-26T05:28:31.1863677Z + nvidia-smi 2024-06-26T05:28:37.3844510Z Wed Jun 26 05:28:37 2024 2024-06-26T05:28:37.3845378Z +-----------------------------------------------------------------------------------------+ 2024-06-26T05:28:37.3846244Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-06-26T05:28:37.3847065Z |-----------------------------------------+------------------------+----------------------+ 2024-06-26T05:28:37.3847881Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-06-26T05:28:37.3848794Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-06-26T05:28:37.3849518Z | | | MIG M. | 2024-06-26T05:28:37.3850113Z |=========================================+========================+======================| 2024-06-26T05:28:37.3932569Z | 0 NVIDIA A10G Off | 00000000:00:1B.0 Off | 0 | 2024-06-26T05:28:37.3933361Z | 0% 30C P0 58W / 300W | 0MiB / 23028MiB | 0% Default | 2024-06-26T05:28:37.3934022Z | | | N/A | 2024-06-26T05:28:37.3934716Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T05:28:37.4024535Z | 1 NVIDIA A10G Off | 00000000:00:1C.0 Off | 0 | 2024-06-26T05:28:37.4025381Z | 0% 30C P0 59W / 300W | 0MiB / 23028MiB | 0% Default | 2024-06-26T05:28:37.4026171Z | | | N/A | 2024-06-26T05:28:37.4026876Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T05:28:37.4115009Z | 2 NVIDIA A10G Off | 00000000:00:1D.0 Off | 0 | 2024-06-26T05:28:37.4116040Z | 0% 30C P0 58W / 300W | 0MiB / 23028MiB | 0% Default | 2024-06-26T05:28:37.4116761Z | | | N/A | 2024-06-26T05:28:37.4117464Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T05:28:37.4216849Z | 3 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-06-26T05:28:37.4217591Z | 0% 30C P0 55W / 300W | 0MiB / 23028MiB | 5% Default | 2024-06-26T05:28:37.4218294Z | | | N/A | 2024-06-26T05:28:37.4219001Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T05:28:37.4289920Z 2024-06-26T05:28:37.4290631Z +-----------------------------------------------------------------------------------------+ 2024-06-26T05:28:37.4291284Z | Processes: | 2024-06-26T05:28:37.4292030Z | GPU GI CI PID Type Process name GPU Memory | 2024-06-26T05:28:37.4292752Z | ID ID Usage | 2024-06-26T05:28:37.4293405Z |=========================================================================================| 2024-06-26T05:28:37.4309805Z | No running processes found | 2024-06-26T05:28:37.4310601Z +-----------------------------------------------------------------------------------------+ 2024-06-26T05:28:39.8838988Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-06-26T05:28:46.0636300Z NVIDIA A10G 2024-06-26T05:28:47.8702581Z + NVIDIA_SMI_STATUS=0 2024-06-26T05:28:47.8703331Z + '[' 0 -eq 0 ']' 2024-06-26T05:28:47.8703841Z + echo 'INFO: Ignoring allowed status 0' 2024-06-26T05:28:47.8704614Z + set -e 2024-06-26T05:28:47.8704942Z INFO: Ignoring allowed status 0 2024-06-26T05:28:47.8706375Z == Installing nvidia container toolkit for amzn2 == 2024-06-26T05:28:47.8709120Z + sudo yum install -y yum-utils 2024-06-26T05:28:48.1773198Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T05:28:49.6689823Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-06-26T05:28:49.6690558Z Nothing to do 2024-06-26T05:28:49.8219933Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]] 2024-06-26T05:28:49.8220836Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T05:28:49.8221940Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T05:28:50.1490736Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T05:28:50.1761176Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T05:28:50.1762411Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T05:28:50.1763397Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T05:28:50.1891397Z + sudo yum install -y nvidia-docker2 2024-06-26T05:28:50.4886500Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T05:28:51.9521809Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-06-26T05:28:51.9522497Z Nothing to do 2024-06-26T05:28:52.1038112Z + sudo systemctl restart docker 2024-06-26T05:29:31.8588447Z Command completed after 1 attempt(s). 2024-06-26T05:29:31.8648005Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T05:29:31.8648807Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T05:29:31.8649681Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-06-26T05:29:31.8650360Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-06-26T05:29:31.8658232Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:29:31.8658757Z env: 2024-06-26T05:29:31.8659060Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:29:31.8659537Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:29:31.8660059Z ##[endgroup] 2024-06-26T05:29:32.0874696Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T05:29:32.1039603Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-06-26T05:29:32.1138648Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-06-26T05:29:32.2058079Z Prepare all required actions 2024-06-26T05:29:32.2058599Z Getting action download info 2024-06-26T05:29:32.3118021Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-06-26T05:29:32.5292588Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-06-26T05:29:32.6477710Z ##[group]Run ./.github/actions/download-build-artifacts 2024-06-26T05:29:32.6478221Z with: 2024-06-26T05:29:32.6478572Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T05:29:32.6479056Z s3-bucket: gha-artifacts 2024-06-26T05:29:32.6479426Z env: 2024-06-26T05:29:32.6479725Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:29:32.6480186Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:29:32.6480696Z ##[endgroup] 2024-06-26T05:29:32.6513247Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T05:29:32.6513708Z with: 2024-06-26T05:29:32.6514285Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T05:29:32.6514918Z s3-bucket: gha-artifacts 2024-06-26T05:29:32.6515295Z region: us-east-1 2024-06-26T05:29:32.6515617Z env: 2024-06-26T05:29:32.6515910Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:29:32.6516384Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:29:32.6516896Z ##[endgroup] 2024-06-26T05:29:33.0644167Z (node:3522) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T05:29:33.0646106Z 2024-06-26T05:29:33.0646682Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T05:29:33.0648184Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T05:29:33.0649934Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T05:29:33.1455427Z Found 1 objects with prefix pytorch/pytorch/9673646808/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-06-26T05:29:33.1456539Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T05:29:54.1080918Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T05:29:54.1088860Z Artifact download has finished successfully 2024-06-26T05:29:54.1241823Z ##[group]Run unzip -o artifacts.zip 2024-06-26T05:29:54.1242278Z unzip -o artifacts.zip 2024-06-26T05:29:54.1250320Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:29:54.1250842Z env: 2024-06-26T05:29:54.1251144Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:29:54.1251620Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:29:54.1252135Z ##[endgroup] 2024-06-26T05:29:54.1281643Z Archive: artifacts.zip 2024-06-26T05:29:54.1283329Z creating: dist/ 2024-06-26T05:29:56.1275321Z inflating: dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T05:29:56.1276032Z creating: build/custom_test_artifacts/ 2024-06-26T05:29:56.1279801Z creating: build/custom_test_artifacts/custom-op-build/ 2024-06-26T05:29:56.1280840Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-06-26T05:29:56.1281825Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-06-26T05:29:56.1283154Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T05:29:56.1284209Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-06-26T05:29:56.1285279Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T05:29:56.1286455Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T05:29:56.1287653Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T05:29:56.1288845Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T05:29:56.1290384Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T05:29:56.1291574Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T05:29:56.1292702Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T05:29:56.1294030Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T05:29:56.1295544Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T05:29:56.1296838Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T05:29:56.1298005Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T05:29:56.1299203Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T05:29:56.1300403Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T05:29:56.1301631Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T05:29:56.1302671Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T05:29:56.1335487Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T05:29:56.1373481Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T05:29:56.1374961Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T05:29:56.1417964Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T05:29:56.1420032Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T05:29:56.1423882Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T05:29:56.1427629Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T05:29:56.1430157Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T05:29:56.1431566Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T05:29:56.1433013Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T05:29:56.1434439Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T05:29:56.1435830Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T05:29:56.1437159Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T05:29:56.1438421Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T05:29:56.1439671Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T05:29:56.1440993Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T05:29:56.1442221Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T05:29:56.1443457Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T05:29:56.1484326Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T05:29:56.1542432Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T05:29:56.1543667Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T05:29:56.1544839Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-06-26T05:29:56.1545716Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-06-26T05:29:56.1546641Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-06-26T05:29:56.1547594Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-06-26T05:29:56.1548655Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-06-26T05:29:56.1549851Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-06-26T05:29:56.1551091Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-06-26T05:29:56.1552152Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-06-26T05:29:56.1553265Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-06-26T05:29:56.1554383Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-06-26T05:29:56.1555487Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-06-26T05:29:56.1556583Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-06-26T05:29:56.1557680Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-06-26T05:29:56.1567623Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-06-26T05:29:56.1695808Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-06-26T05:29:56.1696825Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-06-26T05:29:56.1697935Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-06-26T05:29:56.1699164Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-06-26T05:29:56.1700356Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-06-26T05:29:56.1701468Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-06-26T05:29:56.1702622Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-06-26T05:29:56.1703791Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-06-26T05:29:56.1705156Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-06-26T05:29:56.1706316Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-06-26T05:29:56.1707460Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-06-26T05:29:56.1720468Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-06-26T05:29:56.1800119Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-06-26T05:29:56.1802807Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T05:29:56.1805299Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-06-26T05:29:56.1807463Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-06-26T05:29:56.1809312Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-06-26T05:29:56.1810510Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-06-26T05:29:56.1811451Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-06-26T05:29:56.1812322Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-06-26T05:29:56.1813121Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-06-26T05:29:56.1813966Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-06-26T05:29:56.1914515Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-06-26T05:29:56.1975196Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-06-26T05:29:56.1976073Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-06-26T05:29:56.1976974Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-06-26T05:29:56.1977903Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-06-26T05:29:56.1982918Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T05:29:56.1983933Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-06-26T05:29:56.1985086Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T05:29:56.1986303Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T05:29:56.1987543Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T05:29:56.1988681Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T05:29:56.1989977Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T05:29:56.1991121Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T05:29:56.1992245Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T05:29:56.1993551Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T05:29:56.1995047Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T05:29:56.1996315Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T05:29:56.1997469Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T05:29:56.1998651Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T05:29:56.1999843Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T05:29:56.2000958Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T05:29:56.2001978Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T05:29:56.2034844Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T05:29:56.2072897Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T05:29:56.2074379Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T05:29:56.2117026Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T05:29:56.2118798Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T05:29:56.2120769Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T05:29:56.2122627Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T05:29:56.2124176Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T05:29:56.2125703Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T05:29:56.2127128Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T05:29:56.2128535Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T05:29:56.2129921Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T05:29:56.2131393Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T05:29:56.2132656Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T05:29:56.2133898Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T05:29:56.2135148Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T05:29:56.2136366Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T05:29:56.2137602Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T05:29:56.2182930Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T05:29:56.2241199Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T05:29:56.2242806Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T05:29:56.2244071Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-06-26T05:29:56.2245236Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-06-26T05:29:56.2246445Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-06-26T05:29:56.2247432Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-06-26T05:29:56.2248522Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-06-26T05:29:56.2249753Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-06-26T05:29:56.2250984Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-06-26T05:29:56.2252083Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-06-26T05:29:56.2253203Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-06-26T05:29:56.2254333Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-06-26T05:29:56.2255470Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-06-26T05:29:56.2256600Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-06-26T05:29:56.2257720Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-06-26T05:29:56.2267054Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-06-26T05:29:56.2328749Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-06-26T05:29:56.2330243Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T05:29:56.2331393Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-06-26T05:29:56.2332428Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-06-26T05:29:56.2333563Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-06-26T05:29:56.2334585Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-06-26T05:29:56.2335516Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-06-26T05:29:56.2336385Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-06-26T05:29:56.2337174Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-06-26T05:29:56.2338122Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-06-26T05:29:56.2382630Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-06-26T05:29:56.2383452Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-06-26T05:29:56.2384467Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-06-26T05:29:56.2385373Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-06-26T05:29:56.2390382Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T05:29:56.2391463Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-06-26T05:29:56.2392593Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T05:29:56.2393793Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T05:29:56.2395070Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T05:29:56.2396290Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T05:29:56.2397664Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T05:29:56.2398850Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T05:29:56.2400212Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T05:29:56.2401621Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T05:29:56.2403151Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T05:29:56.2404385Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T05:29:56.2405731Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T05:29:56.2406978Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T05:29:56.2408241Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T05:29:56.2409373Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T05:29:56.2410526Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T05:29:56.2442228Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T05:29:56.2480056Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T05:29:56.2481773Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T05:29:56.2526537Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T05:29:56.2530146Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T05:29:56.2532176Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T05:29:56.2533762Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T05:29:56.2535255Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T05:29:56.2536744Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T05:29:56.2538354Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T05:29:56.2539851Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T05:29:56.2541302Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T05:29:56.2542670Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T05:29:56.2544007Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T05:29:56.2545686Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T05:29:56.2547183Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T05:29:56.2548477Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T05:29:56.2549804Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T05:29:56.2591264Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T05:29:56.2649490Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T05:29:56.2651254Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T05:29:56.2652609Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-06-26T05:29:56.2653732Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-06-26T05:29:56.2655016Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-06-26T05:29:56.2656290Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-06-26T05:29:56.2657472Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-06-26T05:29:56.2658758Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-06-26T05:29:56.2660214Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-06-26T05:29:56.2661475Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-06-26T05:29:56.2662684Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-06-26T05:29:56.2664026Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-06-26T05:29:56.2665498Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-06-26T05:29:56.2666712Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-06-26T05:29:56.2667911Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-06-26T05:29:56.2669196Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-06-26T05:29:56.2781884Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-06-26T05:29:56.2783410Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-06-26T05:29:56.2785014Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-06-26T05:29:56.2786870Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-06-26T05:29:56.2788336Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-06-26T05:29:56.2789567Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-06-26T05:29:56.2790868Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-06-26T05:29:56.2792133Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-06-26T05:29:56.2793395Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-06-26T05:29:56.2794652Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-06-26T05:29:56.2795916Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-06-26T05:29:56.2808397Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-06-26T05:29:56.2862211Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-06-26T05:29:56.2863676Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T05:29:56.2865250Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-06-26T05:29:56.2866317Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-06-26T05:29:56.2867427Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-06-26T05:29:56.2868573Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-06-26T05:29:56.2869718Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-06-26T05:29:56.2870647Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-06-26T05:29:56.2871497Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-06-26T05:29:56.2872367Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-06-26T05:29:56.2968973Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-06-26T05:29:56.3011114Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-06-26T05:29:56.3011786Z creating: build/lib/ 2024-06-26T05:29:56.3096503Z inflating: build/lib/libprotobuf-lite.a 2024-06-26T05:29:56.3537280Z inflating: build/lib/libprotobuf.a 2024-06-26T05:29:56.3546952Z inflating: build/lib/libpthreadpool.a 2024-06-26T05:29:56.3554927Z inflating: build/lib/libcpuinfo.a 2024-06-26T05:29:56.3562836Z inflating: build/lib/libcpuinfo_internals.a 2024-06-26T05:29:56.3563504Z inflating: build/lib/libclog.a 2024-06-26T05:29:56.3581929Z inflating: build/lib/libnnpack.a 2024-06-26T05:29:56.3583644Z inflating: build/lib/libnnpack_reference_layers.a 2024-06-26T05:29:56.3647970Z inflating: build/lib/libgtest.a 2024-06-26T05:29:56.3720234Z inflating: build/lib/libbenchmark.a 2024-06-26T05:29:56.3782020Z inflating: build/lib/libasmjit.a 2024-06-26T05:29:56.3789907Z inflating: build/lib/libittnotify.a 2024-06-26T05:29:56.3815989Z inflating: build/lib/libtensorpipe_uv.a 2024-06-26T05:29:56.3938450Z inflating: build/lib/libgloo.a 2024-06-26T05:29:56.3959546Z inflating: build/lib/libfmt.a 2024-06-26T05:29:56.4048943Z inflating: build/lib/libc10.so 2024-06-26T05:29:56.4050012Z inflating: build/lib/libcaffe2_nvrtc.so 2024-06-26T05:29:56.4050735Z inflating: build/lib/libfoxi_loader.a 2024-06-26T05:29:56.4051894Z inflating: build/lib/libtorch_global_deps.so 2024-06-26T05:29:56.4070783Z inflating: build/lib/libpytorch_qnnpack.a 2024-06-26T05:29:56.4088084Z inflating: build/lib/libgmock.a 2024-06-26T05:29:56.4088553Z inflating: build/lib/libgtest_main.a 2024-06-26T05:29:56.4089321Z inflating: build/lib/libbenchmark_main.a 2024-06-26T05:29:56.4574829Z inflating: build/lib/libprotoc.a 2024-06-26T05:29:56.4944154Z inflating: build/lib/libgloo_cuda.a 2024-06-26T05:29:57.4675123Z inflating: build/lib/libdnnl.a 2024-06-26T05:29:57.5224024Z inflating: build/lib/libtensorpipe.a 2024-06-26T05:29:57.5276759Z inflating: build/lib/libc10_cuda.so 2024-06-26T05:29:57.5277751Z inflating: build/lib/libgmock_main.a 2024-06-26T05:29:57.6491782Z inflating: build/lib/libfbgemm.a 2024-06-26T05:29:57.6739752Z inflating: build/lib/libtensorpipe_cuda.a 2024-06-26T05:29:57.7244282Z inflating: build/lib/libkineto.a 2024-06-26T05:29:57.7421731Z inflating: build/lib/libXNNPACK.a 2024-06-26T05:29:57.7463693Z inflating: build/lib/libonnx_proto.a 2024-06-26T05:29:57.8129499Z inflating: build/lib/libonnx.a 2024-06-26T05:30:00.1460097Z inflating: build/lib/libtorch_cpu.so 2024-06-26T05:30:00.1464598Z inflating: build/lib/libunbox_lib.a 2024-06-26T05:30:00.1468638Z inflating: build/lib/libshm.so 2024-06-26T05:30:02.1386039Z inflating: build/lib/libtorch_cuda.so 2024-06-26T05:30:02.1387020Z inflating: build/lib/libtorch.so 2024-06-26T05:30:02.1390220Z inflating: build/lib/libc10d_cuda_test.so 2024-06-26T05:30:02.9359068Z inflating: build/lib/libtorch_cuda_linalg.so 2024-06-26T05:30:03.1259185Z inflating: build/lib/libtorch_python.so 2024-06-26T05:30:03.1326691Z inflating: build/lib/libtorchbind_test.so 2024-06-26T05:30:03.1346303Z inflating: build/lib/libjitbackend_test.so 2024-06-26T05:30:03.1370957Z inflating: build/lib/libbackend_with_compiler.so 2024-06-26T05:30:03.1404531Z inflating: build/lib/libnnapi_backend.so 2024-06-26T05:30:03.1405120Z creating: build/bin/ 2024-06-26T05:30:03.1454329Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-06-26T05:30:03.1505243Z inflating: build/bin/c10_DeviceGuard_test 2024-06-26T05:30:03.1553296Z inflating: build/bin/c10_Device_test 2024-06-26T05:30:03.1610671Z inflating: build/bin/c10_DispatchKeySet_test 2024-06-26T05:30:03.1662822Z inflating: build/bin/c10_Scalar_test 2024-06-26T05:30:03.1710577Z inflating: build/bin/c10_StreamGuard_test 2024-06-26T05:30:03.1760188Z inflating: build/bin/c10_SymInt_test 2024-06-26T05:30:03.1813120Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-06-26T05:30:03.1867944Z inflating: build/bin/c10_InlineStreamGuard_test 2024-06-26T05:30:03.1922111Z inflating: build/bin/c10_SizesAndStrides_test 2024-06-26T05:30:03.1991146Z inflating: build/bin/c10_cow_test 2024-06-26T05:30:03.2042902Z inflating: build/bin/c10_Bitset_test 2024-06-26T05:30:03.2091109Z inflating: build/bin/c10_ConstexprCrc_test 2024-06-26T05:30:03.2139351Z inflating: build/bin/c10_DeadlockDetection_test 2024-06-26T05:30:03.2189337Z inflating: build/bin/c10_Half_test 2024-06-26T05:30:03.2244238Z inflating: build/bin/c10_LeftRight_test 2024-06-26T05:30:03.2298631Z inflating: build/bin/c10_Metaprogramming_test 2024-06-26T05:30:03.2348236Z inflating: build/bin/c10_Synchronized_test 2024-06-26T05:30:03.2402492Z inflating: build/bin/c10_ThreadLocal_test 2024-06-26T05:30:03.2453460Z inflating: build/bin/c10_TypeIndex_test 2024-06-26T05:30:03.2503292Z inflating: build/bin/c10_TypeList_test 2024-06-26T05:30:03.2551364Z inflating: build/bin/c10_TypeTraits_test 2024-06-26T05:30:03.2601946Z inflating: build/bin/c10_accumulate_test 2024-06-26T05:30:03.2656255Z inflating: build/bin/c10_bfloat16_test 2024-06-26T05:30:03.2705846Z inflating: build/bin/c10_bit_cast_test 2024-06-26T05:30:03.2760880Z inflating: build/bin/c10_complex_math_test 2024-06-26T05:30:03.2815456Z inflating: build/bin/c10_complex_test 2024-06-26T05:30:03.2867296Z inflating: build/bin/c10_exception_test 2024-06-26T05:30:03.2916481Z inflating: build/bin/c10_flags_test 2024-06-26T05:30:03.2965524Z inflating: build/bin/c10_generic_math_test 2024-06-26T05:30:03.3124733Z inflating: build/bin/c10_intrusive_ptr_test 2024-06-26T05:30:03.3174545Z inflating: build/bin/c10_irange_test 2024-06-26T05:30:03.3227169Z inflating: build/bin/c10_lazy_test 2024-06-26T05:30:03.3282831Z inflating: build/bin/c10_logging_test 2024-06-26T05:30:03.3355912Z inflating: build/bin/c10_optional_test 2024-06-26T05:30:03.3416672Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-06-26T05:30:03.3469586Z inflating: build/bin/c10_registry_test 2024-06-26T05:30:03.3615948Z inflating: build/bin/c10_small_vector_test 2024-06-26T05:30:03.3666410Z inflating: build/bin/c10_ssize_test 2024-06-26T05:30:03.3717417Z inflating: build/bin/c10_string_util_test 2024-06-26T05:30:03.3774609Z inflating: build/bin/c10_string_view_test 2024-06-26T05:30:03.3824001Z inflating: build/bin/c10_tempfile_test 2024-06-26T05:30:03.3871933Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-06-26T05:30:03.3927003Z inflating: build/bin/c10_typeid_test 2024-06-26T05:30:03.4361056Z inflating: build/bin/protoc-3.13.0.0 2024-06-26T05:30:03.4795469Z inflating: build/bin/protoc 2024-06-26T05:30:03.4847351Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-06-26T05:30:03.4899099Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-06-26T05:30:03.4950966Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-06-26T05:30:03.5001852Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-06-26T05:30:03.5053652Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-06-26T05:30:03.5105426Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-06-26T05:30:03.5154164Z inflating: build/bin/c10_cuda_CUDATest 2024-06-26T05:30:03.5206029Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-06-26T05:30:03.5527127Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-06-26T05:30:03.5859793Z inflating: build/bin/vec_test_all_types_AVX512 2024-06-26T05:30:03.6207030Z inflating: build/bin/vec_test_all_types_AVX2 2024-06-26T05:30:03.6258750Z inflating: build/bin/FileStoreTest 2024-06-26T05:30:03.6313372Z inflating: build/bin/TCPStoreTest 2024-06-26T05:30:03.6365783Z inflating: build/bin/HashStoreTest 2024-06-26T05:30:03.6379208Z inflating: build/bin/ProcessGroupMPITest 2024-06-26T05:30:03.6432121Z inflating: build/bin/test_edge_op_registration 2024-06-26T05:30:03.6436501Z inflating: build/bin/torch_shm_manager 2024-06-26T05:30:03.6439595Z inflating: build/bin/example_allreduce 2024-06-26T05:30:03.6493461Z inflating: build/bin/test_dist_autograd 2024-06-26T05:30:03.6561578Z inflating: build/bin/test_cpp_rpc 2024-06-26T05:30:03.6564135Z inflating: build/bin/parallel_benchmark 2024-06-26T05:30:03.6629782Z inflating: build/bin/test_mobile_nnc 2024-06-26T05:30:03.6638896Z inflating: build/bin/aot_model_compiler_test 2024-06-26T05:30:03.6973049Z inflating: build/bin/test_lazy 2024-06-26T05:30:03.8119532Z inflating: build/bin/test_api 2024-06-26T05:30:03.8191292Z inflating: build/bin/Dict_test 2024-06-26T05:30:03.8242160Z inflating: build/bin/Dimname_test 2024-06-26T05:30:03.8305256Z inflating: build/bin/MaybeOwned_test 2024-06-26T05:30:03.8361315Z inflating: build/bin/NamedTensor_test 2024-06-26T05:30:03.8420378Z inflating: build/bin/apply_utils_test 2024-06-26T05:30:03.8478934Z inflating: build/bin/atest 2024-06-26T05:30:03.8541566Z inflating: build/bin/basic 2024-06-26T05:30:03.8595243Z inflating: build/bin/broadcast_test 2024-06-26T05:30:03.8645062Z inflating: build/bin/cpu_allocator_test 2024-06-26T05:30:03.8702045Z inflating: build/bin/cpu_generator_test 2024-06-26T05:30:03.8754046Z inflating: build/bin/cpu_profiling_allocator_test 2024-06-26T05:30:03.8845199Z inflating: build/bin/cpu_rng_test 2024-06-26T05:30:03.8894158Z inflating: build/bin/dispatch_key_set_test 2024-06-26T05:30:03.8943416Z inflating: build/bin/dlconvertor_test 2024-06-26T05:30:03.9000640Z inflating: build/bin/extension_backend_test 2024-06-26T05:30:03.9054649Z inflating: build/bin/half_test 2024-06-26T05:30:03.9148480Z inflating: build/bin/ivalue_test 2024-06-26T05:30:03.9197215Z inflating: build/bin/lazy_tensor_test 2024-06-26T05:30:03.9250625Z inflating: build/bin/math_kernel_test 2024-06-26T05:30:03.9303647Z inflating: build/bin/memory_format_test 2024-06-26T05:30:03.9355563Z inflating: build/bin/memory_overlapping_test 2024-06-26T05:30:03.9407597Z inflating: build/bin/mobile_memory_cleanup 2024-06-26T05:30:03.9462330Z inflating: build/bin/native_test 2024-06-26T05:30:03.9512021Z inflating: build/bin/operator_name_test 2024-06-26T05:30:03.9562438Z inflating: build/bin/operators_test 2024-06-26T05:30:03.9613436Z inflating: build/bin/packedtensoraccessor_test 2024-06-26T05:30:03.9680003Z inflating: build/bin/pow_test 2024-06-26T05:30:03.9735932Z inflating: build/bin/quantized_test 2024-06-26T05:30:03.9785085Z inflating: build/bin/reduce_ops_test 2024-06-26T05:30:03.9835214Z inflating: build/bin/reportMemoryUsage_test 2024-06-26T05:30:03.9890653Z inflating: build/bin/scalar_tensor_test 2024-06-26T05:30:03.9947914Z inflating: build/bin/scalar_test 2024-06-26T05:30:03.9998535Z inflating: build/bin/StorageUtils_test 2024-06-26T05:30:04.0050163Z inflating: build/bin/stride_properties_test 2024-06-26T05:30:04.0127003Z inflating: build/bin/tensor_iterator_test 2024-06-26T05:30:04.0180107Z inflating: build/bin/test_parallel 2024-06-26T05:30:04.0183292Z inflating: build/bin/thread_init_test 2024-06-26T05:30:04.0238092Z inflating: build/bin/type_ptr_test 2024-06-26T05:30:04.0296661Z inflating: build/bin/type_test 2024-06-26T05:30:04.0347896Z inflating: build/bin/undefined_tensor_test 2024-06-26T05:30:04.0349245Z inflating: build/bin/verify_api_visibility 2024-06-26T05:30:04.0417397Z inflating: build/bin/legacy_vmap_test 2024-06-26T05:30:04.0468314Z inflating: build/bin/weakref_test 2024-06-26T05:30:04.0518804Z inflating: build/bin/wrapdim_test 2024-06-26T05:30:04.0569516Z inflating: build/bin/xla_tensor_test 2024-06-26T05:30:04.0628808Z inflating: build/bin/IListRef_test 2024-06-26T05:30:04.0732193Z inflating: build/bin/List_test 2024-06-26T05:30:04.0797852Z inflating: build/bin/KernelFunction_test 2024-06-26T05:30:04.0916591Z inflating: build/bin/kernel_function_legacy_test 2024-06-26T05:30:04.1010714Z inflating: build/bin/kernel_function_test 2024-06-26T05:30:04.1135706Z inflating: build/bin/kernel_lambda_legacy_test 2024-06-26T05:30:04.1236504Z inflating: build/bin/kernel_lambda_test 2024-06-26T05:30:04.1296379Z inflating: build/bin/kernel_stackbased_test 2024-06-26T05:30:04.1390222Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-06-26T05:30:04.1440132Z inflating: build/bin/CppSignature_test 2024-06-26T05:30:04.1494939Z inflating: build/bin/backend_fallback_test 2024-06-26T05:30:04.1543035Z inflating: build/bin/op_allowlist_test 2024-06-26T05:30:04.1605575Z inflating: build/bin/inline_container_test 2024-06-26T05:30:04.1898242Z inflating: build/bin/op_registration_test 2024-06-26T05:30:04.1950300Z inflating: build/bin/cuda_apply_test 2024-06-26T05:30:04.1999979Z inflating: build/bin/cuda_allocator_test 2024-06-26T05:30:04.2058372Z inflating: build/bin/cuda_atomic_ops_test 2024-06-26T05:30:04.2112996Z inflating: build/bin/cuda_caching_host_allocator_test 2024-06-26T05:30:04.2181787Z inflating: build/bin/cuda_complex_math_test 2024-06-26T05:30:04.2239575Z inflating: build/bin/cuda_complex_test 2024-06-26T05:30:04.2288551Z inflating: build/bin/cuda_device_test 2024-06-26T05:30:04.2344910Z inflating: build/bin/cuda_cub_test 2024-06-26T05:30:04.2395142Z inflating: build/bin/cuda_dlconvertor_test 2024-06-26T05:30:04.2458449Z inflating: build/bin/cuda_distributions_test 2024-06-26T05:30:04.2514218Z inflating: build/bin/cuda_generator_test 2024-06-26T05:30:04.2562899Z inflating: build/bin/cuda_half_test 2024-06-26T05:30:04.2613207Z inflating: build/bin/cuda_integer_divider_test 2024-06-26T05:30:04.2661353Z inflating: build/bin/cuda_optional_test 2024-06-26T05:30:04.2712216Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-06-26T05:30:04.2763750Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-06-26T05:30:04.2812887Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-06-26T05:30:04.2872192Z inflating: build/bin/cuda_stream_test 2024-06-26T05:30:04.2920509Z inflating: build/bin/cuda_cudnn_test 2024-06-26T05:30:04.2971624Z inflating: build/bin/cuda_vectorized_test 2024-06-26T05:30:04.2986553Z inflating: build/bin/tutorial_tensorexpr 2024-06-26T05:30:04.3051061Z inflating: build/bin/ProcessGroupGlooTest 2024-06-26T05:30:04.3107271Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-06-26T05:30:04.3169220Z inflating: build/bin/ProcessGroupNCCLTest 2024-06-26T05:30:04.3229992Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-06-26T05:30:04.4037418Z inflating: build/bin/test_tensorexpr 2024-06-26T05:30:04.4597927Z inflating: build/bin/test_jit 2024-06-26T05:30:04.4598501Z creating: .additional_ci_files/ 2024-06-26T05:30:04.4652249Z inflating: .additional_ci_files/test-times.json 2024-06-26T05:30:04.4863976Z inflating: .additional_ci_files/test-class-times.json 2024-06-26T05:30:04.4893690Z ##[group]Run rm artifacts.zip 2024-06-26T05:30:04.4894086Z rm artifacts.zip 2024-06-26T05:30:04.4901833Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:30:04.4902349Z env: 2024-06-26T05:30:04.4902669Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:30:04.4903133Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:30:04.4903633Z ##[endgroup] 2024-06-26T05:30:04.9763452Z ##[group]Run df -H 2024-06-26T05:30:04.9763783Z df -H 2024-06-26T05:30:04.9771832Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:30:04.9772350Z env: 2024-06-26T05:30:04.9772643Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:30:04.9773121Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:30:04.9773626Z ##[endgroup] 2024-06-26T05:30:04.9800320Z Filesystem Size Used Avail Use% Mounted on 2024-06-26T05:30:04.9800852Z devtmpfs 101G 0 101G 0% /dev 2024-06-26T05:30:04.9801572Z tmpfs 101G 13M 101G 1% /dev/shm 2024-06-26T05:30:04.9802113Z tmpfs 101G 599k 101G 1% /run 2024-06-26T05:30:04.9802615Z tmpfs 101G 0 101G 0% /sys/fs/cgroup 2024-06-26T05:30:04.9803114Z /dev/nvme0n1p1 162G 45G 117G 28% / 2024-06-26T05:30:04.9840716Z Prepare all required actions 2024-06-26T05:30:04.9841175Z Getting action download info 2024-06-26T05:30:05.0947911Z ##[group]Run ./.github/actions/download-td-artifacts 2024-06-26T05:30:05.0948404Z with: 2024-06-26T05:30:05.0948673Z env: 2024-06-26T05:30:05.0948980Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:30:05.0949461Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:30:05.0949981Z ##[endgroup] 2024-06-26T05:30:05.1185708Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T05:30:05.1186171Z with: 2024-06-26T05:30:05.1186449Z name: td_results 2024-06-26T05:30:05.1186792Z s3-bucket: gha-artifacts 2024-06-26T05:30:05.1187165Z region: us-east-1 2024-06-26T05:30:05.1187472Z env: 2024-06-26T05:30:05.1187760Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:30:05.1188237Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:30:05.1188737Z ##[endgroup] 2024-06-26T05:30:05.5391289Z (node:3586) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T05:30:05.5392378Z 2024-06-26T05:30:05.5392663Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T05:30:05.5393418Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T05:30:05.5394289Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T05:30:05.5994435Z Found 0 objects with prefix pytorch/pytorch/9673646808/td_results/ 2024-06-26T05:30:05.6001138Z Artifact download has finished successfully 2024-06-26T05:30:05.6133766Z ##[group]Run mkdir -p .additional_ci_files 2024-06-26T05:30:05.6134276Z mkdir -p .additional_ci_files 2024-06-26T05:30:05.6134872Z mv td_results.json .additional_ci_files/td_results.json 2024-06-26T05:30:05.6143065Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:30:05.6143584Z env: 2024-06-26T05:30:05.6143888Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:30:05.6144580Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:30:05.6145104Z ##[endgroup] 2024-06-26T05:30:05.6180977Z mv: cannot stat 'td_results.json': No such file or directory 2024-06-26T05:30:05.6192312Z ##[error]Process completed with exit code 1. 2024-06-26T05:30:05.6381108Z ##[group]Run .github/scripts/parse_ref.py 2024-06-26T05:30:05.6381611Z .github/scripts/parse_ref.py 2024-06-26T05:30:05.6389576Z shell: /usr/bin/bash -e {0} 2024-06-26T05:30:05.6389945Z env: 2024-06-26T05:30:05.6390255Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:30:05.6390748Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:30:05.6391255Z ##[endgroup] 2024-06-26T05:30:05.6623124Z Prepare all required actions 2024-06-26T05:30:05.6676588Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-06-26T05:30:05.6677112Z with: 2024-06-26T05:30:05.6677737Z github-token: *** 2024-06-26T05:30:05.6678093Z env: 2024-06-26T05:30:05.6678391Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:30:05.6678882Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:30:05.6679397Z ##[endgroup] 2024-06-26T05:30:05.6918156Z ##[group]Run set -eux 2024-06-26T05:30:05.6918501Z set -eux 2024-06-26T05:30:05.6919106Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-06-26T05:30:05.6927219Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:30:05.6927745Z env: 2024-06-26T05:30:05.6928036Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:30:05.6928511Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:30:05.6929179Z GITHUB_TOKEN: *** 2024-06-26T05:30:05.6929514Z ##[endgroup] 2024-06-26T05:30:05.6953931Z + python3 .github/scripts/get_workflow_job_id.py 9673646808 i-0a8c115e2cb171c32 2024-06-26T05:30:06.3787809Z setting job-id=26688910858 2024-06-26T05:30:06.3788682Z setting job-name=cuda12.1-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu) 2024-06-26T05:30:06.3999901Z Prepare all required actions 2024-06-26T05:30:06.4000431Z Getting action download info 2024-06-26T05:30:06.5107342Z ##[group]Run ./.github/actions/filter-test-configs 2024-06-26T05:30:06.5107819Z with: 2024-06-26T05:30:06.5108289Z github-token: *** 2024-06-26T05:30:06.5116585Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T05:30:06.5125759Z job-name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu) 2024-06-26T05:30:06.5126504Z env: 2024-06-26T05:30:06.5126802Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:30:06.5127284Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:30:06.5127783Z ##[endgroup] 2024-06-26T05:30:06.5387660Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T05:30:06.5388240Z with: 2024-06-26T05:30:06.5388515Z shell: bash 2024-06-26T05:30:06.5388831Z timeout_minutes: 10 2024-06-26T05:30:06.5389172Z max_attempts: 5 2024-06-26T05:30:06.5389499Z retry_wait_seconds: 30 2024-06-26T05:30:06.5390651Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T05:30:06.5391857Z polling_interval_seconds: 1 2024-06-26T05:30:06.5392252Z warning_on_retry: true 2024-06-26T05:30:06.5392625Z continue_on_error: false 2024-06-26T05:30:06.5392973Z env: 2024-06-26T05:30:06.5393266Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:30:06.5393738Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:30:06.5394409Z GITHUB_TOKEN: *** 2024-06-26T05:30:06.5394734Z ##[endgroup] 2024-06-26T05:30:06.5905184Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T05:30:06.8051684Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T05:30:06.8213667Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1) 2024-06-26T05:30:06.8358510Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-06-26T05:30:06.8366437Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.6.2) 2024-06-26T05:30:06.8374698Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7) 2024-06-26T05:30:06.8387496Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12) 2024-06-26T05:30:06.8407752Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19) 2024-06-26T05:30:07.5903828Z Command completed after 1 attempt(s). 2024-06-26T05:30:07.6068224Z ##[group]Run set -x 2024-06-26T05:30:07.6068577Z set -x 2024-06-26T05:30:07.6068879Z  2024-06-26T05:30:07.6069451Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T05:30:07.6070267Z # in runner workspace 2024-06-26T05:30:07.6070806Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-06-26T05:30:07.6078991Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:30:07.6079515Z env: 2024-06-26T05:30:07.6079817Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:30:07.6080281Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:30:07.6080779Z ##[endgroup] 2024-06-26T05:30:07.6102548Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-06-26T05:30:07.6310685Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T05:30:07.6311268Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T05:30:07.6311749Z echo "Job name: ${JOB_NAME}" 2024-06-26T05:30:07.6312175Z  2024-06-26T05:30:07.6312776Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T05:30:07.6313494Z # in runner workspace 2024-06-26T05:30:07.6314095Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-06-26T05:30:07.6314779Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-06-26T05:30:07.6315266Z  --job-name "${JOB_NAME}" \ 2024-06-26T05:30:07.6323831Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-06-26T05:30:07.6332536Z  --selected-test-configs "" \ 2024-06-26T05:30:07.6333020Z  --pr-number "${PR_NUMBER}" \ 2024-06-26T05:30:07.6333467Z  --tag "${TAG}" \ 2024-06-26T05:30:07.6333887Z  --event-name "${EVENT_NAME}" \ 2024-06-26T05:30:07.6334352Z  --schedule "${SCHEDULE}" \ 2024-06-26T05:30:07.6334806Z  --branch "${HEAD_BRANCH}" 2024-06-26T05:30:07.6342679Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:30:07.6343206Z env: 2024-06-26T05:30:07.6343508Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:30:07.6344001Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:30:07.6344977Z GITHUB_TOKEN: *** 2024-06-26T05:30:07.6345663Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu) 2024-06-26T05:30:07.6346414Z PR_NUMBER: 2024-06-26T05:30:07.6346751Z TAG: ciflow/inductor/129470 2024-06-26T05:30:07.6347146Z EVENT_NAME: push 2024-06-26T05:30:07.6347479Z SCHEDULE: 2024-06-26T05:30:07.6347781Z HEAD_BRANCH: 2024-06-26T05:30:07.6348099Z ##[endgroup] 2024-06-26T05:30:07.6372453Z Workflow: inductor 2024-06-26T05:30:07.6373993Z Job name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu) 2024-06-26T05:30:07.8540527Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-06-26T05:30:08.0479077Z ##[group]Run echo "Filtered matrix:" 2024-06-26T05:30:08.0479561Z echo "Filtered matrix:" 2024-06-26T05:30:08.0488030Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-06-26T05:30:08.0496785Z  2024-06-26T05:30:08.0497086Z echo 2024-06-26T05:30:08.0497497Z echo "Is the current job unstable? False" 2024-06-26T05:30:08.0497970Z  2024-06-26T05:30:08.0498269Z echo 2024-06-26T05:30:08.0498652Z echo "Is keep-going label set? False" 2024-06-26T05:30:08.0499112Z  2024-06-26T05:30:08.0499405Z echo 2024-06-26T05:30:08.0499749Z echo "Renabled issues? " 2024-06-26T05:30:08.0507962Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:30:08.0508492Z env: 2024-06-26T05:30:08.0508803Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:30:08.0509289Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:30:08.0509801Z ##[endgroup] 2024-06-26T05:30:08.0531261Z Filtered matrix: 2024-06-26T05:30:08.0541144Z {include: [{config: inductor, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-06-26T05:30:08.0549838Z 2024-06-26T05:30:08.0550014Z Is the current job unstable? False 2024-06-26T05:30:08.0550305Z 2024-06-26T05:30:08.0550519Z Is keep-going label set? False 2024-06-26T05:30:08.0550801Z 2024-06-26T05:30:08.0550931Z Renabled issues? 2024-06-26T05:30:08.0591200Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T05:30:08.0591924Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T05:30:08.0599253Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:30:08.0599778Z env: 2024-06-26T05:30:08.0600077Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:30:08.0600562Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:30:08.0601075Z JOB_TIMEOUT: 240 2024-06-26T05:30:08.0601395Z ##[endgroup] 2024-06-26T05:30:08.0678562Z ##[group]Run set -x 2024-06-26T05:30:08.0678972Z set -x 2024-06-26T05:30:08.0679287Z  2024-06-26T05:30:08.0679659Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-06-26T05:30:08.0680216Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-06-26T05:30:08.0680803Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-06-26T05:30:08.0681333Z  TEST_COMMAND=.ci/onnx/test.sh 2024-06-26T05:30:08.0681764Z else 2024-06-26T05:30:08.0682119Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T05:30:08.0682563Z fi 2024-06-26T05:30:08.0682856Z  2024-06-26T05:30:08.0683341Z # detached container should get cleaned up by teardown_ec2_linux 2024-06-26T05:30:08.0684136Z # TODO: Stop building test binaries as part of the build phase 2024-06-26T05:30:08.0684837Z # Used for GPU_FLAG since that doesn't play nice 2024-06-26T05:30:08.0685515Z # shellcheck disable=SC2086,SC2090 2024-06-26T05:30:08.0685994Z container_name=$(docker run \ 2024-06-26T05:30:08.0686431Z  ${GPU_FLAG:-} \ 2024-06-26T05:30:08.0686823Z  -e BUILD_ENVIRONMENT \ 2024-06-26T05:30:08.0687249Z  -e PR_NUMBER \ 2024-06-26T05:30:08.0687623Z  -e GITHUB_ACTIONS \ 2024-06-26T05:30:08.0688041Z  -e GITHUB_REPOSITORY \ 2024-06-26T05:30:08.0688481Z  -e GITHUB_WORKFLOW \ 2024-06-26T05:30:08.0688892Z  -e GITHUB_JOB \ 2024-06-26T05:30:08.0689270Z  -e GITHUB_RUN_ID \ 2024-06-26T05:30:08.0689677Z  -e GITHUB_RUN_NUMBER \ 2024-06-26T05:30:08.0690104Z  -e GITHUB_RUN_ATTEMPT \ 2024-06-26T05:30:08.0690634Z  -e JOB_ID \ 2024-06-26T05:30:08.0690998Z  -e JOB_NAME \ 2024-06-26T05:30:08.0691360Z  -e BASE_SHA \ 2024-06-26T05:30:08.0691723Z  -e BRANCH \ 2024-06-26T05:30:08.0692070Z  -e SHA1 \ 2024-06-26T05:30:08.0692440Z  -e AWS_DEFAULT_REGION \ 2024-06-26T05:30:08.0692870Z  -e IN_WHEEL_TEST \ 2024-06-26T05:30:08.0693275Z  -e SHARD_NUMBER \ 2024-06-26T05:30:08.0693665Z  -e TEST_CONFIG \ 2024-06-26T05:30:08.0694063Z  -e NUM_TEST_SHARDS \ 2024-06-26T05:30:08.0694481Z  -e REENABLED_ISSUES \ 2024-06-26T05:30:08.0694917Z  -e CONTINUE_THROUGH_ERROR \ 2024-06-26T05:30:08.0695358Z  -e VERBOSE_TEST_LOGS \ 2024-06-26T05:30:08.0695783Z  -e NO_TEST_TIMEOUT \ 2024-06-26T05:30:08.0696182Z  -e NO_TD \ 2024-06-26T05:30:08.0696545Z  -e TD_DISTRIBUTED \ 2024-06-26T05:30:08.0696961Z  -e PR_LABELS \ 2024-06-26T05:30:08.0697405Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-06-26T05:30:08.0697912Z  -e SCCACHE_BUCKET \ 2024-06-26T05:30:08.0698330Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-06-26T05:30:08.0698779Z  -e XLA_CUDA \ 2024-06-26T05:30:08.0699226Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-06-26T05:30:08.0699776Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-06-26T05:30:08.0700322Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-06-26T05:30:08.0700873Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-06-26T05:30:08.0701389Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-06-26T05:30:08.0701849Z  -e DASHBOARD_TAG \ 2024-06-26T05:30:08.0702351Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-06-26T05:30:08.0702944Z  --security-opt seccomp=unconfined \ 2024-06-26T05:30:08.0703467Z  --cap-add=SYS_PTRACE \ 2024-06-26T05:30:08.0703925Z  --ipc=host \ 2024-06-26T05:30:08.0704422Z  --shm-size="${SHM_SIZE}" \ 2024-06-26T05:30:08.0704849Z  --tty \ 2024-06-26T05:30:08.0705189Z  --detach \ 2024-06-26T05:30:08.0705582Z  --name="${container_name}" \ 2024-06-26T05:30:08.0706022Z  --user jenkins \ 2024-06-26T05:30:08.0706618Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-06-26T05:30:08.0707220Z  -w /var/lib/jenkins/workspace \ 2024-06-26T05:30:08.0707705Z  "${DOCKER_IMAGE}" 2024-06-26T05:30:08.0708080Z ) 2024-06-26T05:30:08.0708524Z # Propagate download.pytorch.org IP to container 2024-06-26T05:30:08.0709497Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-06-26T05:30:08.0710517Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-06-26T05:30:08.0711456Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-06-26T05:30:08.0719795Z shell: /usr/bin/bash -e {0} 2024-06-26T05:30:08.0720164Z env: 2024-06-26T05:30:08.0720462Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:30:08.0720928Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:30:08.0721559Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T05:30:08.0722085Z PR_NUMBER: 2024-06-26T05:30:08.0722426Z GITHUB_REPOSITORY: pytorch/pytorch 2024-06-26T05:30:08.0722859Z GITHUB_WORKFLOW: inductor 2024-06-26T05:30:08.0723237Z GITHUB_JOB: test 2024-06-26T05:30:08.0723620Z GITHUB_RUN_ID: 9673646808 2024-06-26T05:30:08.0724000Z GITHUB_RUN_NUMBER: 76977 2024-06-26T05:30:08.0724368Z GITHUB_RUN_ATTEMPT: 1 2024-06-26T05:30:08.0724723Z JOB_ID: 26688910858 2024-06-26T05:30:08.0725464Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu) 2024-06-26T05:30:08.0726209Z BRANCH: 2024-06-26T05:30:08.0726558Z SHA1: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T05:30:08.0727097Z BASE_SHA: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T05:30:08.0727676Z TEST_CONFIG: inductor_distributed 2024-06-26T05:30:08.0728087Z SHARD_NUMBER: 1 2024-06-26T05:30:08.0728405Z NUM_TEST_SHARDS: 1 2024-06-26T05:30:08.0728741Z REENABLED_ISSUES: 2024-06-26T05:30:08.0729110Z CONTINUE_THROUGH_ERROR: False 2024-06-26T05:30:08.0729509Z VERBOSE_TEST_LOGS: False 2024-06-26T05:30:08.0729885Z NO_TEST_TIMEOUT: False 2024-06-26T05:30:08.0730247Z NO_TD: False 2024-06-26T05:30:08.0730591Z TD_DISTRIBUTED: False 2024-06-26T05:30:08.0731032Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-06-26T05:30:08.0731568Z SCCACHE_S3_KEY_PREFIX: inductor 2024-06-26T05:30:08.0731981Z SHM_SIZE: 2g 2024-06-26T05:30:08.0733111Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T05:30:08.0734354Z XLA_CUDA: 2024-06-26T05:30:08.0734878Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-06-26T05:30:08.0735541Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-06-26T05:30:08.0736017Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-06-26T05:30:08.0736454Z DASHBOARD_TAG: 2024-06-26T05:30:08.0737003Z HUGGING_FACE_HUB_TOKEN: *** 2024-06-26T05:30:08.0737404Z ##[endgroup] 2024-06-26T05:30:08.0761740Z + [[ inductor_distributed == \m\u\l\t\i\g\p\u ]] 2024-06-26T05:30:08.0763190Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-06-26T05:30:08.0763908Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T05:30:08.0770503Z +++ nproc --ignore=2 2024-06-26T05:30:08.0785918Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=46 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9673646808 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T05:30:14.9293605Z + container_name=f816bf66af59392d564c87e8387c46db59db2501aae5f8b2c2310f41077fe0f2 2024-06-26T05:30:14.9294703Z + grep download.pytorch.org /etc/hosts 2024-06-26T05:30:14.9295799Z + docker exec -i f816bf66af59392d564c87e8387c46db59db2501aae5f8b2c2310f41077fe0f2 sudo bash -c '/bin/cat >> /etc/hosts' 2024-06-26T05:30:14.9906058Z + echo DOCKER_CONTAINER_ID=f816bf66af59392d564c87e8387c46db59db2501aae5f8b2c2310f41077fe0f2 2024-06-26T05:30:14.9908909Z ++ echo dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T05:30:14.9910630Z + docker exec -t f816bf66af59392d564c87e8387c46db59db2501aae5f8b2c2310f41077fe0f2 sh -c 'pip install dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-06-26T05:30:15.3694850Z Processing ./dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+gitb8c4c54) 2024-06-26T05:30:15.6828697Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.13.1) 2024-06-26T05:30:15.6833878Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (4.12.2) 2024-06-26T05:30:15.6837702Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.12.1) 2024-06-26T05:30:15.6839870Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.8.8) 2024-06-26T05:30:15.6841887Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.1.4) 2024-06-26T05:30:15.6843834Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2024.2.0) 2024-06-26T05:30:15.6868966Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.3.0) 2024-06-26T05:30:15.6925759Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.21.2) 2024-06-26T05:30:15.7354832Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.1.5) 2024-06-26T05:30:15.7508293Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.3.0) 2024-06-26T05:30:17.0325879Z Installing collected packages: torch 2024-06-26T05:30:25.7181637Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-06-26T05:30:25.7183036Z timm 0.9.7 requires torchvision, which is not installed. 2024-06-26T05:30:25.7183721Z Successfully installed torch-2.5.0a0+gitb8c4c54 2024-06-26T05:30:25.7954679Z ++ dirname .ci/pytorch/test.sh 2024-06-26T05:30:25.7957804Z + source .ci/pytorch/common.sh 2024-06-26T05:30:25.7961093Z +++ dirname .ci/pytorch/common.sh 2024-06-26T05:30:25.7967694Z ++ source .ci/pytorch/common_utils.sh 2024-06-26T05:30:25.7971127Z +++ declare -f -t trap_add 2024-06-26T05:30:25.7975975Z ++ set -ex 2024-06-26T05:30:25.7976565Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T05:30:25.7977146Z ++ BUILD_TEST_LIBTORCH=0 2024-06-26T05:30:25.7977878Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-06-26T05:30:25.7980354Z ++ stat -c %u /var/lib/jenkins/workspace 2024-06-26T05:30:25.7989197Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-06-26T05:30:25.7989826Z + trap_add cleanup_workspace EXIT 2024-06-26T05:30:25.7990367Z + trap_add_cmd=cleanup_workspace 2024-06-26T05:30:25.7990763Z + shift 2024-06-26T05:30:25.7991064Z + for trap_add_name in "$@" 2024-06-26T05:30:25.7995937Z +++ trap -p EXIT 2024-06-26T05:30:25.7998698Z ++ eval 'extract_trap_cmd ' 2024-06-26T05:30:25.7999095Z +++ extract_trap_cmd 2024-06-26T05:30:25.7999466Z +++ printf '%s\n' '' 2024-06-26T05:30:25.7999859Z ++ printf '%s\n' cleanup_workspace 2024-06-26T05:30:25.8001667Z + trap -- ' 2024-06-26T05:30:25.8002035Z cleanup_workspace' EXIT 2024-06-26T05:30:25.8002516Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-06-26T05:30:26.2142173Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-06-26T05:30:26.2154382Z + echo 'Environment variables:' 2024-06-26T05:30:26.2154867Z Environment variables: 2024-06-26T05:30:26.2155217Z + env 2024-06-26T05:30:26.2161104Z INSTALLED_DB=yes 2024-06-26T05:30:26.2161661Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T05:30:26.2162169Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T05:30:26.2162592Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T05:30:26.2163355Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T05:30:26.2164170Z CONTINUE_THROUGH_ERROR=False 2024-06-26T05:30:26.2165104Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T05:30:26.2165930Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T05:30:26.2166671Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T05:30:26.2167356Z HOSTNAME=f816bf66af59 2024-06-26T05:30:26.2168510Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_6855dd30-6896-4770-8a6b-8b90e963dcc9 2024-06-26T05:30:26.2169462Z GITHUB_ACTION=__self 2024-06-26T05:30:26.2169952Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T05:30:26.2174947Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T05:30:26.2179125Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T05:30:26.2179786Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T05:30:26.2180167Z GITHUB_RUN_NUMBER=76977 2024-06-26T05:30:26.2180561Z TEST_CONFIG=inductor_distributed 2024-06-26T05:30:26.2180985Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T05:30:26.2181475Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T05:30:26.2181954Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T05:30:26.2182410Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T05:30:26.2182857Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T05:30:26.2183331Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T05:30:26.2183908Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T05:30:26.2184643Z GITHUB_REF_TYPE=tag 2024-06-26T05:30:26.2184980Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T05:30:26.2185375Z NCCL_VERSION=2.17.1-1 2024-06-26T05:30:26.2185780Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T05:30:26.2186229Z XLA_CUDA= 2024-06-26T05:30:26.2186918Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T05:30:26.2188957Z *** 2024-06-26T05:30:26.2189267Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T05:30:26.2189685Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T05:30:26.2190058Z GITHUB_ACTIONS=true 2024-06-26T05:30:26.2190407Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:30:26.2190937Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T05:30:26.2191483Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T05:30:26.2191971Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T05:30:26.2192490Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T05:30:26.2192988Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T05:30:26.2193789Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/129470 2024-06-26T05:30:26.2194527Z UCC_HOME=/usr 2024-06-26T05:30:26.2194880Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T05:30:26.2195298Z VERBOSE_TEST_LOGS=False 2024-06-26T05:30:26.2195659Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T05:30:26.2196136Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T05:30:26.2196653Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T05:30:26.2197136Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T05:30:26.2197523Z SHARD_NUMBER=1 2024-06-26T05:30:26.2197849Z GITHUB_REF_PROTECTED=false 2024-06-26T05:30:26.2198225Z HOME=/var/lib/jenkins 2024-06-26T05:30:26.2198620Z GITHUB_API_URL=https://api.github.com 2024-06-26T05:30:26.2199112Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T05:30:26.2199609Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T05:30:26.2200112Z SCCACHE_S3_KEY_PREFIX=inductor 2024-06-26T05:30:26.2200491Z CUDA_VERSION=12.1.1 2024-06-26T05:30:26.2201065Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T05:30:26.2201522Z NUM_TEST_SHARDS=1 2024-06-26T05:30:26.2201838Z UCX_HOME=/usr 2024-06-26T05:30:26.2202358Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T05:30:26.2203453Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_6855dd30-6896-4770-8a6b-8b90e963dcc9 2024-06-26T05:30:26.2204693Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu) 2024-06-26T05:30:26.2206024Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_6855dd30-6896-4770-8a6b-8b90e963dcc9 2024-06-26T05:30:26.2207149Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T05:30:26.2207823Z GITHUB_EVENT_NAME=push 2024-06-26T05:30:26.2208170Z DASHBOARD_TAG= 2024-06-26T05:30:26.2208494Z GITHUB_RUN_ID=9673646808 2024-06-26T05:30:26.2208960Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T05:30:26.2209516Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T05:30:26.2210524Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_6855dd30-6896-4770-8a6b-8b90e963dcc9 2024-06-26T05:30:26.2211446Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T05:30:26.2211876Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T05:30:26.2212271Z PR_NUMBER= 2024-06-26T05:30:26.2212572Z GITHUB_RUN_ATTEMPT=1 2024-06-26T05:30:26.2212922Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T05:30:26.2213382Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T05:30:26.2213856Z TERM=xterm 2024-06-26T05:30:26.2214204Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T05:30:26.2214627Z INSTALLED_VISION=yes 2024-06-26T05:30:26.2214959Z BRANCH= 2024-06-26T05:30:26.2215259Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T05:30:26.2215680Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T05:30:26.2216116Z CUDA_PATH=/usr/local/cuda 2024-06-26T05:30:26.2216885Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T05:30:26.2217666Z GITHUB_SERVER_URL=https://github.com 2024-06-26T05:30:26.2218161Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T05:30:26.2218649Z REENABLED_ISSUES= 2024-06-26T05:30:26.2218961Z SHLVL=1 2024-06-26T05:30:26.2219314Z MAX_JOBS=46 2024-06-26T05:30:26.2219637Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T05:30:26.2220006Z NVARCH=x86_64 2024-06-26T05:30:26.2220321Z GITHUB_ACTOR_ID=54816060 2024-06-26T05:30:26.2220793Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T05:30:26.2221339Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T05:30:26.2221839Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T05:30:26.2222518Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T05:30:26.2223099Z GITHUB_JOB=test 2024-06-26T05:30:26.2223511Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T05:30:26.2224092Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T05:30:26.2224906Z NO_TEST_TIMEOUT=False 2024-06-26T05:30:26.2225244Z TD_DISTRIBUTED=False 2024-06-26T05:30:26.2225645Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T05:30:26.2226100Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T05:30:26.2226535Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T05:30:26.2226915Z GITHUB_RETENTION_DAYS=90 2024-06-26T05:30:26.2227289Z OPENSSL_DIR=/opt/openssl 2024-06-26T05:30:26.2227655Z GITHUB_ACTION_REPOSITORY= 2024-06-26T05:30:26.2228728Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T05:30:26.2229818Z GITHUB_BASE_REF= 2024-06-26T05:30:26.2230160Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T05:30:26.2230557Z CI=true 2024-06-26T05:30:26.2230881Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T05:30:26.2231317Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T05:30:26.2231708Z JOB_ID=26688910858 2024-06-26T05:30:26.2232036Z INSTALLED_PROTOBUF=yes 2024-06-26T05:30:26.2232470Z GITHUB_HEAD_REF= 2024-06-26T05:30:26.2232789Z GITHUB_ACTION_REF= 2024-06-26T05:30:26.2233234Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T05:30:26.2233729Z GITHUB_WORKFLOW=inductor 2024-06-26T05:30:26.2234095Z DEBIAN_FRONTEND=noninteractive 2024-06-26T05:30:26.2235023Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_6855dd30-6896-4770-8a6b-8b90e963dcc9 2024-06-26T05:30:26.2235857Z NO_TD=False 2024-06-26T05:30:26.2236176Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T05:30:26.2236551Z _=/usr/bin/env 2024-06-26T05:30:26.2237039Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-06-26T05:30:26.2313776Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-06-26T05:30:26.2315025Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T05:30:26.2316133Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-06-26T05:30:26.2317042Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-06-26T05:30:26.2317682Z + BUILD_DIR=build 2024-06-26T05:30:26.2318024Z + BUILD_RENAMED_DIR=build_renamed 2024-06-26T05:30:26.2318425Z + BUILD_BIN_DIR=build/bin 2024-06-26T05:30:26.2318777Z + SHARD_NUMBER=1 2024-06-26T05:30:26.2319172Z + NUM_TEST_SHARDS=1 2024-06-26T05:30:26.2319576Z + export VALGRIND=ON 2024-06-26T05:30:26.2319922Z + VALGRIND=ON 2024-06-26T05:30:26.2320415Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-06-26T05:30:26.2320922Z + [[ 0 == \1 ]] 2024-06-26T05:30:26.2321223Z + [[ False == \1 ]] 2024-06-26T05:30:26.2321704Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-06-26T05:30:26.2322248Z ++ realpath build/custom_test_artifacts 2024-06-26T05:30:26.2326565Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-06-26T05:30:26.2327307Z + [[ -n '' ]] 2024-06-26T05:30:26.2327685Z + echo 'Environment variables' 2024-06-26T05:30:26.2328090Z Environment variables 2024-06-26T05:30:26.2328443Z + env 2024-06-26T05:30:26.2333413Z INSTALLED_DB=yes 2024-06-26T05:30:26.2333918Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T05:30:26.2334412Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T05:30:26.2334923Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T05:30:26.2335887Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T05:30:26.2336547Z CONTINUE_THROUGH_ERROR=False 2024-06-26T05:30:26.2337251Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T05:30:26.2337819Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T05:30:26.2338442Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T05:30:26.2339112Z HOSTNAME=f816bf66af59 2024-06-26T05:30:26.2340102Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_6855dd30-6896-4770-8a6b-8b90e963dcc9 2024-06-26T05:30:26.2341091Z GITHUB_ACTION=__self 2024-06-26T05:30:26.2341520Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T05:30:26.2346270Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T05:30:26.2350221Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T05:30:26.2350750Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T05:30:26.2351128Z GITHUB_RUN_NUMBER=76977 2024-06-26T05:30:26.2351505Z TEST_CONFIG=inductor_distributed 2024-06-26T05:30:26.2352065Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T05:30:26.2352554Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T05:30:26.2353041Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T05:30:26.2353511Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T05:30:26.2353958Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T05:30:26.2354422Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T05:30:26.2354943Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T05:30:26.2355419Z GITHUB_REF_TYPE=tag 2024-06-26T05:30:26.2355768Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T05:30:26.2356168Z NCCL_VERSION=2.17.1-1 2024-06-26T05:30:26.2356579Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T05:30:26.2357031Z XLA_CUDA= 2024-06-26T05:30:26.2357533Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T05:30:26.2357946Z *** 2024-06-26T05:30:26.2358245Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T05:30:26.2358682Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T05:30:26.2359105Z GITHUB_ACTIONS=true 2024-06-26T05:30:26.2359466Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:30:26.2359983Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T05:30:26.2360531Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T05:30:26.2361015Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T05:30:26.2361549Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T05:30:26.2362053Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T05:30:26.2362858Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/129470 2024-06-26T05:30:26.2363599Z UCC_HOME=/usr 2024-06-26T05:30:26.2363963Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T05:30:26.2364380Z VERBOSE_TEST_LOGS=False 2024-06-26T05:30:26.2364735Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T05:30:26.2365304Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T05:30:26.2365841Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T05:30:26.2366322Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T05:30:26.2366726Z SHARD_NUMBER=1 2024-06-26T05:30:26.2367067Z GITHUB_REF_PROTECTED=false 2024-06-26T05:30:26.2367436Z HOME=/var/lib/jenkins 2024-06-26T05:30:26.2367814Z GITHUB_API_URL=https://api.github.com 2024-06-26T05:30:26.2368283Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T05:30:26.2368865Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T05:30:26.2369423Z SCCACHE_S3_KEY_PREFIX=inductor 2024-06-26T05:30:26.2369803Z CUDA_VERSION=12.1.1 2024-06-26T05:30:26.2370252Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T05:30:26.2370707Z NUM_TEST_SHARDS=1 2024-06-26T05:30:26.2371019Z UCX_HOME=/usr 2024-06-26T05:30:26.2371552Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T05:30:26.2372648Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_6855dd30-6896-4770-8a6b-8b90e963dcc9 2024-06-26T05:30:26.2373890Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu) 2024-06-26T05:30:26.2375169Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_6855dd30-6896-4770-8a6b-8b90e963dcc9 2024-06-26T05:30:26.2376307Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T05:30:26.2376976Z GITHUB_EVENT_NAME=push 2024-06-26T05:30:26.2377331Z DASHBOARD_TAG= 2024-06-26T05:30:26.2377650Z GITHUB_RUN_ID=9673646808 2024-06-26T05:30:26.2378134Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T05:30:26.2378682Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T05:30:26.2379697Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_6855dd30-6896-4770-8a6b-8b90e963dcc9 2024-06-26T05:30:26.2380627Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T05:30:26.2381045Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T05:30:26.2381440Z PR_NUMBER= 2024-06-26T05:30:26.2381741Z GITHUB_RUN_ATTEMPT=1 2024-06-26T05:30:26.2382082Z VALGRIND=ON 2024-06-26T05:30:26.2382394Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T05:30:26.2382867Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T05:30:26.2383424Z TERM=xterm 2024-06-26T05:30:26.2383783Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T05:30:26.2384208Z INSTALLED_VISION=yes 2024-06-26T05:30:26.2384758Z BRANCH= 2024-06-26T05:30:26.2385070Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T05:30:26.2385485Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T05:30:26.2385945Z CUDA_PATH=/usr/local/cuda 2024-06-26T05:30:26.2386722Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T05:30:26.2387506Z GITHUB_SERVER_URL=https://github.com 2024-06-26T05:30:26.2388007Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T05:30:26.2388492Z REENABLED_ISSUES= 2024-06-26T05:30:26.2388820Z SHLVL=1 2024-06-26T05:30:26.2389101Z MAX_JOBS=46 2024-06-26T05:30:26.2389433Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T05:30:26.2389808Z NVARCH=x86_64 2024-06-26T05:30:26.2390128Z GITHUB_ACTOR_ID=54816060 2024-06-26T05:30:26.2390593Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T05:30:26.2391162Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T05:30:26.2391678Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T05:30:26.2392363Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T05:30:26.2392954Z GITHUB_JOB=test 2024-06-26T05:30:26.2393369Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T05:30:26.2393960Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T05:30:26.2394507Z NO_TEST_TIMEOUT=False 2024-06-26T05:30:26.2394846Z TD_DISTRIBUTED=False 2024-06-26T05:30:26.2395255Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T05:30:26.2395716Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T05:30:26.2396158Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T05:30:26.2396538Z GITHUB_RETENTION_DAYS=90 2024-06-26T05:30:26.2396907Z OPENSSL_DIR=/opt/openssl 2024-06-26T05:30:26.2397275Z GITHUB_ACTION_REPOSITORY= 2024-06-26T05:30:26.2398352Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T05:30:26.2399504Z GITHUB_BASE_REF= 2024-06-26T05:30:26.2399857Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T05:30:26.2400242Z CI=true 2024-06-26T05:30:26.2400665Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T05:30:26.2401114Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T05:30:26.2401517Z JOB_ID=26688910858 2024-06-26T05:30:26.2401857Z INSTALLED_PROTOBUF=yes 2024-06-26T05:30:26.2402200Z GITHUB_HEAD_REF= 2024-06-26T05:30:26.2402525Z GITHUB_ACTION_REF= 2024-06-26T05:30:26.2402979Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T05:30:26.2403476Z GITHUB_WORKFLOW=inductor 2024-06-26T05:30:26.2403872Z DEBIAN_FRONTEND=noninteractive 2024-06-26T05:30:26.2404817Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_6855dd30-6896-4770-8a6b-8b90e963dcc9 2024-06-26T05:30:26.2405758Z NO_TD=False 2024-06-26T05:30:26.2406098Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T05:30:26.2406518Z _=/usr/bin/env 2024-06-26T05:30:26.2406901Z + echo 'Testing pytorch' 2024-06-26T05:30:26.2407292Z Testing pytorch 2024-06-26T05:30:26.2407650Z + export LANG=C.UTF-8 2024-06-26T05:30:26.2408007Z + LANG=C.UTF-8 2024-06-26T05:30:26.2408316Z + PR_NUMBER= 2024-06-26T05:30:26.2408695Z + [[ inductor_distributed == \d\e\f\a\u\l\t ]] 2024-06-26T05:30:26.2409239Z + [[ inductor_distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-06-26T05:30:26.2409775Z + [[ inductor_distributed == \s\l\o\w ]] 2024-06-26T05:30:26.2410433Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-06-26T05:30:26.2411149Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-06-26T05:30:26.2411725Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T05:30:26.2412233Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T05:30:26.2412713Z + [[ inductor_distributed == *crossref* ]] 2024-06-26T05:30:26.2413302Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T05:30:26.2413959Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-06-26T05:30:26.2414729Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T05:30:26.2415319Z + pip_install --user ninja==1.10.2 2024-06-26T05:30:26.2415874Z + pip install --progress-bar off --user ninja==1.10.2 2024-06-26T05:30:26.5992147Z Collecting ninja==1.10.2 2024-06-26T05:30:26.6187788Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-06-26T05:30:26.6330830Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-06-26T05:30:27.8941491Z Installing collected packages: ninja 2024-06-26T05:30:27.9007698Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-06-26T05:30:27.9008997Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-06-26T05:30:27.9050196Z Successfully installed ninja-1.10.2 2024-06-26T05:30:27.9891484Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T05:30:27.9893882Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T05:30:27.9895356Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T05:30:27.9895883Z + install_tlparse 2024-06-26T05:30:27.9896272Z + pip_install --user tlparse==0.3.7 2024-06-26T05:30:27.9896837Z + pip install --progress-bar off --user tlparse==0.3.7 2024-06-26T05:30:28.3459796Z Collecting tlparse==0.3.7 2024-06-26T05:30:28.3620049Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-06-26T05:30:28.3749361Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-06-26T05:30:29.6727504Z Installing collected packages: tlparse 2024-06-26T05:30:29.7081365Z Successfully installed tlparse-0.3.7 2024-06-26T05:30:29.7981470Z ++ python -m site --user-base 2024-06-26T05:30:29.8143311Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T05:30:29.8145121Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-06-26T05:30:29.8145980Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-06-26T05:30:29.8146784Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T05:30:29.8147753Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-06-26T05:30:29.8148917Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-06-26T05:30:29.8149643Z + cd test 2024-06-26T05:30:29.8150186Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-06-26T05:30:31.3219805Z + [[ inductor_distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-06-26T05:30:31.3220484Z + [[ inductor_distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-06-26T05:30:31.3221428Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T05:30:31.3221939Z + pushd test 2024-06-26T05:30:31.3222308Z ~/workspace/test ~/workspace 2024-06-26T05:30:31.3222876Z ++ python -c 'import torch; print(torch.version.cuda)' 2024-06-26T05:30:32.8163045Z + CUDA_VERSION=12.1 2024-06-26T05:30:32.8163676Z + '[' 12.1 == 12.4 ']' 2024-06-26T05:30:32.8164046Z + ISCUDA124= 2024-06-26T05:30:32.8164353Z + popd 2024-06-26T05:30:32.8164645Z ~/workspace 2024-06-26T05:30:32.8167914Z + DYNAMO_BENCHMARK_FLAGS=() 2024-06-26T05:30:32.8168483Z + [[ inductor_distributed == *dynamo_eager* ]] 2024-06-26T05:30:32.8169019Z + [[ inductor_distributed == *aot_eager* ]] 2024-06-26T05:30:32.8169535Z + [[ inductor_distributed == *aot_inductor* ]] 2024-06-26T05:30:32.8170245Z + [[ inductor_distributed == *inductor* ]] 2024-06-26T05:30:32.8170725Z + [[ inductor_distributed != *perf* ]] 2024-06-26T05:30:32.8171272Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-06-26T05:30:32.8171750Z + [[ inductor_distributed == *dynamic* ]] 2024-06-26T05:30:32.8172272Z + [[ inductor_distributed == *cpu_inductor* ]] 2024-06-26T05:30:32.8172815Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-06-26T05:30:32.8194668Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T05:30:32.8195474Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-06-26T05:30:32.8196463Z + cd test 2024-06-26T05:30:32.8196944Z + python -c 'import torch; print(torch.__config__.show())' 2024-06-26T05:30:34.5360076Z PyTorch built with: 2024-06-26T05:30:34.5360600Z - GCC 9.4 2024-06-26T05:30:34.5361053Z - C++ Version: 201703 2024-06-26T05:30:34.5362120Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T05:30:34.5363244Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T05:30:34.5363927Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T05:30:34.5364449Z - LAPACK is enabled (usually provided by MKL) 2024-06-26T05:30:34.5364948Z - NNPACK is enabled 2024-06-26T05:30:34.5365428Z - CPU capability usage: AVX2 2024-06-26T05:30:34.5365846Z - CUDA Runtime 12.1 2024-06-26T05:30:34.5366377Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-06-26T05:30:34.5366993Z - CuDNN 90.1 (built against CUDA 12.4) 2024-06-26T05:30:34.5367440Z - Magma 2.6.1 2024-06-26T05:30:34.5374660Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-06-26T05:30:34.5381355Z 2024-06-26T05:30:34.9091950Z + cd test 2024-06-26T05:30:34.9092560Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-06-26T05:30:36.1506384Z ATen/Parallel: 2024-06-26T05:30:36.1506921Z at::get_num_threads() : 24 2024-06-26T05:30:36.1507450Z at::get_num_interop_threads() : 48 2024-06-26T05:30:36.1508018Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T05:30:36.1508431Z omp_get_max_threads() : 24 2024-06-26T05:30:36.1509413Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T05:30:36.1510242Z mkl_get_max_threads() : 24 2024-06-26T05:30:36.1510891Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T05:30:36.1511541Z std::thread::hardware_concurrency() : 48 2024-06-26T05:30:36.1511981Z Environment variables: 2024-06-26T05:30:36.1512341Z OMP_NUM_THREADS : [not set] 2024-06-26T05:30:36.1512726Z MKL_NUM_THREADS : [not set] 2024-06-26T05:30:36.1513117Z ATen parallel backend: OpenMP 2024-06-26T05:30:36.1513376Z 2024-06-26T05:30:36.3990563Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T05:30:36.3991959Z + [[ inductor_distributed == *backward* ]] 2024-06-26T05:30:36.3992443Z + [[ inductor_distributed == *xla* ]] 2024-06-26T05:30:36.3992922Z + [[ inductor_distributed == *executorch* ]] 2024-06-26T05:30:36.3993456Z + [[ inductor_distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-06-26T05:30:36.3994117Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T05:30:36.3994688Z + [[ inductor_distributed == distributed ]] 2024-06-26T05:30:36.3995224Z + [[ inductor_distributed == *inductor_distributed* ]] 2024-06-26T05:30:36.3995717Z + test_inductor_distributed 2024-06-26T05:30:36.3996230Z + echo 'Testing multi_gpu tests in test_torchinductor' 2024-06-26T05:30:36.3996789Z Testing multi_gpu tests in test_torchinductor 2024-06-26T05:30:36.3997572Z + python test/run_test.py -i inductor/test_torchinductor.py -k test_multi_gpu --verbose 2024-06-26T05:30:36.4847169Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T05:30:36.4849438Z import pkg_resources 2024-06-26T05:30:39.0424940Z No TD results found 2024-06-26T05:30:39.0427305Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T05:30:39.0941113Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T05:30:39.1499457Z Ignoring disabled issues: [''] 2024-06-26T05:30:39.1627758Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:30:39.1966475Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:30:39.1974117Z Running all tests 2024-06-26T05:30:39.1976699Z Running parallel tests on 3 processes 2024-06-26T05:30:39.1977350Z Name: tests to run (est. time: 0.06min) 2024-06-26T05:30:39.1977927Z Serial tests (1): 2024-06-26T05:30:39.1978485Z inductor/test_torchinductor 1/1 2024-06-26T05:30:39.1978932Z Parallel tests (0): 2024-06-26T05:30:39.1979316Z Name: excluded (est. time: 0.0min) 2024-06-26T05:30:39.1979731Z Serial tests (0): 2024-06-26T05:30:39.1980068Z Parallel tests (0): 2024-06-26T05:30:39.1980694Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T05:30:39.2029073Z Running inductor/test_torchinductor 1/1 ... [2024-06-26 05:30:39.202593] 2024-06-26T05:30:39.2033210Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_torchinductor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_multi_gpu', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 05:30:39.202899] 2024-06-26T05:30:50.9866639Z 2024-06-26T05:30:50.9868068Z inductor/test_torchinductor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_torchinductor_1.1_4a28d8ddaa1c8330_.log 2024-06-26T05:30:50.9870717Z Running 4 items in this shard: test/inductor/test_torchinductor.py::CpuTests::test_multi_gpu_device_cpu, test/inductor/test_torchinductor.py::CpuTests::test_multi_gpu_recompile_on_index_cpu, test/inductor/test_torchinductor.py::GPUTests::test_multi_gpu_device_cuda, test/inductor/test_torchinductor.py::GPUTests::test_multi_gpu_recompile_on_index_cuda 2024-06-26T05:30:50.9872493Z 2024-06-26T05:30:52.0515993Z + python test/run_test.py -i inductor/test_aot_inductor.py -k test_non_default_cuda_device --verbose 2024-06-26T05:30:52.1372278Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T05:30:52.1373640Z import pkg_resources 2024-06-26T05:30:54.6746543Z No TD results found 2024-06-26T05:30:54.6747858Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T05:30:54.6749813Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T05:30:54.6846575Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:30:54.7207283Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:30:54.7215361Z Running all tests 2024-06-26T05:30:54.7218090Z Running parallel tests on 3 processes 2024-06-26T05:30:54.7218884Z Name: tests to run (est. time: 0.82min) 2024-06-26T05:30:54.7219384Z Serial tests (1): 2024-06-26T05:30:54.7219752Z inductor/test_aot_inductor 1/1 2024-06-26T05:30:54.7220171Z Parallel tests (0): 2024-06-26T05:30:54.7220534Z Name: excluded (est. time: 0.0min) 2024-06-26T05:30:54.7220957Z Serial tests (0): 2024-06-26T05:30:54.7221297Z Parallel tests (0): 2024-06-26T05:30:54.7221924Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T05:30:54.7271410Z Running inductor/test_aot_inductor 1/1 ... [2024-06-26 05:30:54.726810] 2024-06-26T05:30:54.7275221Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_non_default_cuda_device', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 05:30:54.727103] 2024-06-26T05:31:26.3926885Z 2024-06-26T05:31:26.3928907Z inductor/test_aot_inductor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_1.1_e1e98bd85d6f7c5b_.log 2024-06-26T05:31:26.3934654Z Running 6 items in this shard: test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_non_default_cuda_device_abi_compatible_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpuWithStackAllocation::test_non_default_cuda_device_abi_compatible_cpu_with_stack_allocation, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpuWithStackAllocationAndMinimalArrayRefInterface::test_non_default_cuda_device_abi_compatible_cpu_with_stack_allocation_and_minimal_arrayref_interface, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCuda::test_non_default_cuda_device_abi_compatible_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestNonABICompatibleCpu::test_non_default_cuda_device_non_abi_compatible_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestNonABICompatibleCuda::test_non_default_cuda_device_non_abi_compatible_cuda 2024-06-26T05:31:26.3939069Z 2024-06-26T05:31:27.4320926Z + python test/run_test.py -i inductor/test_aot_inductor.py -k test_replicate_on_devices --verbose 2024-06-26T05:31:27.5179792Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T05:31:27.5180980Z import pkg_resources 2024-06-26T05:31:30.0518903Z No TD results found 2024-06-26T05:31:30.0520130Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T05:31:30.0521798Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T05:31:30.0619203Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:31:30.0980988Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:31:30.0988811Z Running all tests 2024-06-26T05:31:30.0991525Z Running parallel tests on 3 processes 2024-06-26T05:31:30.0992007Z Name: tests to run (est. time: 0.82min) 2024-06-26T05:31:30.0992446Z Serial tests (1): 2024-06-26T05:31:30.0994073Z inductor/test_aot_inductor 1/1 2024-06-26T05:31:30.0994632Z Parallel tests (0): 2024-06-26T05:31:30.0995021Z Name: excluded (est. time: 0.0min) 2024-06-26T05:31:30.0995427Z Serial tests (0): 2024-06-26T05:31:30.0995764Z Parallel tests (0): 2024-06-26T05:31:30.0996397Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T05:31:30.1044426Z Running inductor/test_aot_inductor 1/1 ... [2024-06-26 05:31:30.104076] 2024-06-26T05:31:30.1049811Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_replicate_on_devices', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 05:31:30.104554] 2024-06-26T05:32:00.1174697Z 2024-06-26T05:32:00.1176252Z inductor/test_aot_inductor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_1.1_fac06b80b0bca0da_.log 2024-06-26T05:32:00.1182666Z Running 6 items in this shard: test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_replicate_on_devices_abi_compatible_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpuWithStackAllocation::test_replicate_on_devices_abi_compatible_cpu_with_stack_allocation, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpuWithStackAllocationAndMinimalArrayRefInterface::test_replicate_on_devices_abi_compatible_cpu_with_stack_allocation_and_minimal_arrayref_interface, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCuda::test_replicate_on_devices_abi_compatible_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestNonABICompatibleCpu::test_replicate_on_devices_non_abi_compatible_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestNonABICompatibleCuda::test_replicate_on_devices_non_abi_compatible_cuda 2024-06-26T05:32:00.1187263Z 2024-06-26T05:32:01.1068052Z + python test/run_test.py -i distributed/test_c10d_functional_native.py --verbose 2024-06-26T05:32:01.1919674Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T05:32:01.1921999Z import pkg_resources 2024-06-26T05:32:03.7080482Z No TD results found 2024-06-26T05:32:03.7082000Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T05:32:03.7083713Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T05:32:03.7179280Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:32:03.7540941Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:32:03.7550492Z Running all tests 2024-06-26T05:32:03.7553530Z Running parallel tests on 3 processes 2024-06-26T05:32:03.7555304Z Name: tests to run (est. time: 2.75min) 2024-06-26T05:32:03.7556080Z Serial tests (1): 2024-06-26T05:32:03.7556584Z distributed/test_c10d_functional_native 1/1 2024-06-26T05:32:03.7557066Z Parallel tests (0): 2024-06-26T05:32:03.7557438Z Name: excluded (est. time: 0.0min) 2024-06-26T05:32:03.7558075Z Serial tests (0): 2024-06-26T05:32:03.7558462Z Parallel tests (0): 2024-06-26T05:32:03.7559082Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T05:32:03.7606317Z Running distributed/test_c10d_functional_native 1/1 ... [2024-06-26 05:32:03.760266] 2024-06-26T05:32:03.7609568Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_functional_native.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 05:32:03.760567] 2024-06-26T05:34:51.5483228Z 2024-06-26T05:34:51.5485261Z distributed/test_c10d_functional_native 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_functional_native_1.1_565881f8a41c7ffe_.log 2024-06-26T05:34:51.5496734Z Running 23 items in this shard: test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_to_all_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_broadcast, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_threading, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_unwaited, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_to_all_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_broadcast, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_inplace_op_on_view, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reuse_buffer_after_inplace_collective, test/distributed/test_c10d_functional_native.py::CompileTest::test_ranks_and_tag 2024-06-26T05:34:51.5507346Z 2024-06-26T05:34:52.6103649Z + python test/run_test.py -i distributed/_tensor/test_dtensor_compile.py --verbose 2024-06-26T05:34:52.6963479Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T05:34:52.6964672Z import pkg_resources 2024-06-26T05:34:55.2059464Z No TD results found 2024-06-26T05:34:55.2060742Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T05:34:55.2062387Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T05:34:55.2158518Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:34:55.2524184Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:34:55.2527425Z Running all tests 2024-06-26T05:34:55.2530073Z Running parallel tests on 3 processes 2024-06-26T05:34:55.2530555Z Name: tests to run (est. time: 0.54min) 2024-06-26T05:34:55.2530992Z Serial tests (1): 2024-06-26T05:34:55.2531395Z distributed/_tensor/test_dtensor_compile 1/1 2024-06-26T05:34:55.2531868Z Parallel tests (0): 2024-06-26T05:34:55.2536313Z Name: excluded (est. time: 0.0min) 2024-06-26T05:34:55.2536774Z Serial tests (0): 2024-06-26T05:34:55.2537122Z Parallel tests (0): 2024-06-26T05:34:55.2537850Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T05:34:55.2582459Z Running distributed/_tensor/test_dtensor_compile 1/1 ... [2024-06-26 05:34:55.257896] 2024-06-26T05:34:55.2586732Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 05:34:55.258195] 2024-06-26T05:35:30.4879216Z 2024-06-26T05:35:30.4881207Z distributed/_tensor/test_dtensor_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_compile_1.1_d81e077b58a4ce3d_.log 2024-06-26T05:35:30.4895794Z Running 24 items in this shard: test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_device_mesh_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_attribute_access_on_intermediate, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_dynamo_disable, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_graph_break, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_contiguous_dtensor_noncontiguous_local_as_tangent, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_different_gradient_placement, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamo_device_mesh_attrs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_noncontiguous_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_graph_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_redistribute_unbalanced_correct_strides, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs_forward_hook, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_fakify_dtensor, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_graph_input_is_async, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_placement_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_tp_compile_comm_reordering, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_dtensor_redistribute_backward, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_True 2024-06-26T05:35:30.4909583Z 2024-06-26T05:35:31.5343308Z + python test/run_test.py -i distributed/tensor/parallel/test_fsdp_2d_parallel.py --verbose 2024-06-26T05:35:31.6197046Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T05:35:31.6198190Z import pkg_resources 2024-06-26T05:35:34.1386093Z No TD results found 2024-06-26T05:35:34.1387236Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T05:35:34.1388886Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T05:35:34.1483467Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:35:34.1844394Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:35:34.1853964Z Running all tests 2024-06-26T05:35:34.1856360Z Running parallel tests on 3 processes 2024-06-26T05:35:34.1856837Z Name: tests to run (est. time: 1.1min) 2024-06-26T05:35:34.1857270Z Serial tests (1): 2024-06-26T05:35:34.1857716Z distributed/tensor/parallel/test_fsdp_2d_parallel 1/1 2024-06-26T05:35:34.1858233Z Parallel tests (0): 2024-06-26T05:35:34.1858606Z Name: excluded (est. time: 0.0min) 2024-06-26T05:35:34.1859016Z Serial tests (0): 2024-06-26T05:35:34.1861284Z Parallel tests (0): 2024-06-26T05:35:34.1862108Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T05:35:34.1909283Z Running distributed/tensor/parallel/test_fsdp_2d_parallel 1/1 ... [2024-06-26 05:35:34.190583] 2024-06-26T05:35:34.1913059Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_fsdp_2d_parallel.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 05:35:34.190879] 2024-06-26T05:36:42.7305701Z 2024-06-26T05:36:42.7307880Z distributed/tensor/parallel/test_fsdp_2d_parallel 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_fsdp_2d_parallel_1.1_c8ac91e729bc824b_.log 2024-06-26T05:36:42.7316765Z Running 12 items in this shard: test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelTraining::test_2d_e2e_training_default, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelTraining::test_2d_e2e_training_not_use_orig_params, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelTraining::test_2d_e2e_training_use_orig_params, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelTraining::test_2d_fsdp_state_enable_extension, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelTraining::test_raise_invalid_tp_composition, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelStateDict::test_2d_load_state_dict_is_even_sharded_model_False, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelStateDict::test_2d_load_state_dict_is_even_sharded_model_True, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelStateDict::test_2d_optim_state_dict_is_even_sharded_model_False, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelStateDict::test_2d_optim_state_dict_is_even_sharded_model_True, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelStateDict::test_2d_state_dict_is_even_sharded_model_False, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelStateDict::test_2d_state_dict_is_even_sharded_model_True, test/distributed/tensor/parallel/test_fsdp_2d_parallel.py::TestNew2dParallelStateDict::test_fsdp_2d_extension 2024-06-26T05:36:42.7325160Z 2024-06-26T05:36:43.7789050Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_comm.py --verbose 2024-06-26T05:36:43.8643924Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T05:36:43.8645132Z import pkg_resources 2024-06-26T05:36:46.4012973Z No TD results found 2024-06-26T05:36:46.4014202Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T05:36:46.4015851Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T05:36:46.4111337Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:36:46.4471920Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:36:46.4481037Z Running all tests 2024-06-26T05:36:46.4483900Z Running parallel tests on 3 processes 2024-06-26T05:36:46.4484481Z Name: tests to run (est. time: 2.15min) 2024-06-26T05:36:46.4485012Z Serial tests (1): 2024-06-26T05:36:46.4485464Z distributed/_composable/fsdp/test_fully_shard_comm 1/1 2024-06-26T05:36:46.4488240Z Parallel tests (0): 2024-06-26T05:36:46.4489609Z Name: excluded (est. time: 0.0min) 2024-06-26T05:36:46.4490166Z Serial tests (0): 2024-06-26T05:36:46.4490573Z Parallel tests (0): 2024-06-26T05:36:46.4491219Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T05:36:46.4535917Z Running distributed/_composable/fsdp/test_fully_shard_comm 1/1 ... [2024-06-26 05:36:46.453187] 2024-06-26T05:36:46.4540489Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_comm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 05:36:46.453569] 2024-06-26T05:38:59.3892377Z 2024-06-26T05:38:59.3894560Z distributed/_composable/fsdp/test_fully_shard_comm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_comm_1.1_0515b49296d7f582_.log 2024-06-26T05:38:59.3902704Z Running 11 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_all_gather_fp32, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_reduce_scatter_fp16, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_reduce_scatter_fp32, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_fully_shard_communication_count, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_manual_reshard_with_reshard_after_forward_false, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_fully_shard_backward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_set_modules_to_backward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_set_modules_to_forward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardUnshardMultiProcess::test_unshard_async, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardUnshardMultiThread::test_unshard_no_param_group, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardUnshardMultiThread::test_unshard_without_lazy_init 2024-06-26T05:38:59.3910075Z 2024-06-26T05:39:00.4415812Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_training.py -k test_train_parity_multi_group --verbose 2024-06-26T05:39:00.5267026Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T05:39:00.5268212Z import pkg_resources 2024-06-26T05:39:03.0255423Z No TD results found 2024-06-26T05:39:03.0256593Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T05:39:03.0258236Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T05:39:03.0354244Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:39:03.0717718Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:39:03.0723835Z Running all tests 2024-06-26T05:39:03.0726358Z Running parallel tests on 3 processes 2024-06-26T05:39:03.0726837Z Name: tests to run (est. time: 2.7min) 2024-06-26T05:39:03.0727284Z Serial tests (1): 2024-06-26T05:39:03.0727755Z distributed/_composable/fsdp/test_fully_shard_training 1/1 2024-06-26T05:39:03.0728293Z Parallel tests (0): 2024-06-26T05:39:03.0728667Z Name: excluded (est. time: 0.0min) 2024-06-26T05:39:03.0729083Z Serial tests (0): 2024-06-26T05:39:03.0729423Z Parallel tests (0): 2024-06-26T05:39:03.0732358Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T05:39:03.0775929Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2024-06-26 05:39:03.077246] 2024-06-26T05:39:03.0780283Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_train_parity_multi_group', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 05:39:03.077542] 2024-06-26T05:43:16.6281995Z 2024-06-26T05:43:16.6283640Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_14c84f6bd12e4ef1_.log 2024-06-26T05:43:16.6288409Z Running 2 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_multi_group, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_multi_group_cpu_offload_eager 2024-06-26T05:43:16.6290073Z 2024-06-26T05:43:17.6593727Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_training.py -k test_train_parity_with_activation_checkpointing --verbose 2024-06-26T05:43:17.7449273Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T05:43:17.7450439Z import pkg_resources 2024-06-26T05:43:20.2966525Z No TD results found 2024-06-26T05:43:20.2968712Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T05:43:20.2970556Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T05:43:20.3064933Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:43:20.3427655Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:43:20.3437104Z Running all tests 2024-06-26T05:43:20.3439395Z Running parallel tests on 3 processes 2024-06-26T05:43:20.3439878Z Name: tests to run (est. time: 2.7min) 2024-06-26T05:43:20.3440307Z Serial tests (1): 2024-06-26T05:43:20.3442760Z distributed/_composable/fsdp/test_fully_shard_training 1/1 2024-06-26T05:43:20.3443416Z Parallel tests (0): 2024-06-26T05:43:20.3443797Z Name: excluded (est. time: 0.0min) 2024-06-26T05:43:20.3444199Z Serial tests (0): 2024-06-26T05:43:20.3444535Z Parallel tests (0): 2024-06-26T05:43:20.3445277Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T05:43:20.3493574Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2024-06-26 05:43:20.348982] 2024-06-26T05:43:20.3496753Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_train_parity_with_activation_checkpointing', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 05:43:20.349276] 2024-06-26T05:43:59.2785644Z 2024-06-26T05:43:59.2787517Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_8cf153b0073d98eb_.log 2024-06-26T05:43:59.2791490Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCompose::test_train_parity_with_activation_checkpointing 2024-06-26T05:43:59.2792693Z 2024-06-26T05:44:00.3278564Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_training.py -k test_train_parity_2d_mlp --verbose 2024-06-26T05:44:00.4137136Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T05:44:00.4138291Z import pkg_resources 2024-06-26T05:44:02.9222613Z No TD results found 2024-06-26T05:44:02.9223826Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T05:44:02.9225679Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T05:44:02.9323919Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:44:02.9689754Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:44:02.9696633Z Running all tests 2024-06-26T05:44:02.9699242Z Running parallel tests on 3 processes 2024-06-26T05:44:02.9699881Z Name: tests to run (est. time: 2.7min) 2024-06-26T05:44:02.9702390Z Serial tests (1): 2024-06-26T05:44:02.9703354Z distributed/_composable/fsdp/test_fully_shard_training 1/1 2024-06-26T05:44:02.9704005Z Parallel tests (0): 2024-06-26T05:44:02.9704666Z Name: excluded (est. time: 0.0min) 2024-06-26T05:44:02.9705077Z Serial tests (0): 2024-06-26T05:44:02.9705418Z Parallel tests (0): 2024-06-26T05:44:02.9706039Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T05:44:02.9751953Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2024-06-26 05:44:02.974881] 2024-06-26T05:44:02.9755685Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_train_parity_2d_mlp', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 05:44:02.975175] 2024-06-26T05:44:15.1123376Z 2024-06-26T05:44:15.1125651Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_376ce079db104d5e_.log 2024-06-26T05:44:15.1127769Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard2DTraining::test_train_parity_2d_mlp 2024-06-26T05:44:15.1128680Z 2024-06-26T05:44:16.1212336Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_training.py -k test_train_parity_hsdp --verbose 2024-06-26T05:44:16.2074159Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T05:44:16.2075376Z import pkg_resources 2024-06-26T05:44:18.7421131Z No TD results found 2024-06-26T05:44:18.7422232Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T05:44:18.7423873Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T05:44:18.7521448Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:44:18.7879489Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:44:18.7888321Z Running all tests 2024-06-26T05:44:18.7891012Z Running parallel tests on 3 processes 2024-06-26T05:44:18.7891714Z Name: tests to run (est. time: 2.7min) 2024-06-26T05:44:18.7892237Z Serial tests (1): 2024-06-26T05:44:18.7892835Z distributed/_composable/fsdp/test_fully_shard_training 1/1 2024-06-26T05:44:18.7893490Z Parallel tests (0): 2024-06-26T05:44:18.7893883Z Name: excluded (est. time: 0.0min) 2024-06-26T05:44:18.7894297Z Serial tests (0): 2024-06-26T05:44:18.7894640Z Parallel tests (0): 2024-06-26T05:44:18.7895254Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T05:44:18.7943570Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2024-06-26 05:44:18.794021] 2024-06-26T05:44:18.7947470Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_train_parity_hsdp', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 05:44:18.794317] 2024-06-26T05:44:35.3874912Z 2024-06-26T05:44:35.3876872Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_2d98cb6ae7f90560_.log 2024-06-26T05:44:35.3879037Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardHSDPTraining::test_train_parity_hsdp 2024-06-26T05:44:35.3879935Z 2024-06-26T05:44:36.4585943Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_training.py -k test_train_parity_2d_transformer_checkpoint_resume --verbose 2024-06-26T05:44:36.5439476Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T05:44:36.5440897Z import pkg_resources 2024-06-26T05:44:39.0975752Z No TD results found 2024-06-26T05:44:39.0976908Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T05:44:39.0978573Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T05:44:39.1075619Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:44:39.1436894Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:44:39.1444341Z Running all tests 2024-06-26T05:44:39.1446540Z Running parallel tests on 3 processes 2024-06-26T05:44:39.1447022Z Name: tests to run (est. time: 2.7min) 2024-06-26T05:44:39.1447466Z Serial tests (1): 2024-06-26T05:44:39.1447933Z distributed/_composable/fsdp/test_fully_shard_training 1/1 2024-06-26T05:44:39.1448483Z Parallel tests (0): 2024-06-26T05:44:39.1448859Z Name: excluded (est. time: 0.0min) 2024-06-26T05:44:39.1449269Z Serial tests (0): 2024-06-26T05:44:39.1452923Z Parallel tests (0): 2024-06-26T05:44:39.1453681Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T05:44:39.1498013Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2024-06-26 05:44:39.149459] 2024-06-26T05:44:39.1501664Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_train_parity_2d_transformer_checkpoint_resume', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 05:44:39.149754] 2024-06-26T05:44:59.4993112Z 2024-06-26T05:44:59.4995079Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_93bd0bc415b1f7fb_.log 2024-06-26T05:44:59.4997173Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard2DTraining::test_train_parity_2d_transformer_checkpoint_resume 2024-06-26T05:44:59.4998125Z 2024-06-26T05:45:00.5259529Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_training.py -k test_gradient_accumulation --verbose 2024-06-26T05:45:00.6118572Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T05:45:00.6119855Z import pkg_resources 2024-06-26T05:45:03.1246263Z No TD results found 2024-06-26T05:45:03.1248402Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T05:45:03.1250160Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T05:45:03.1349213Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:45:03.1708782Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:45:03.1718446Z Running all tests 2024-06-26T05:45:03.1721885Z Running parallel tests on 3 processes 2024-06-26T05:45:03.1722549Z Name: tests to run (est. time: 2.7min) 2024-06-26T05:45:03.1723273Z Serial tests (1): 2024-06-26T05:45:03.1723735Z distributed/_composable/fsdp/test_fully_shard_training 1/1 2024-06-26T05:45:03.1726329Z Parallel tests (0): 2024-06-26T05:45:03.1726737Z Name: excluded (est. time: 0.0min) 2024-06-26T05:45:03.1727167Z Serial tests (0): 2024-06-26T05:45:03.1727496Z Parallel tests (0): 2024-06-26T05:45:03.1728146Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T05:45:03.1775938Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2024-06-26 05:45:03.177259] 2024-06-26T05:45:03.1779822Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_gradient_accumulation', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 05:45:03.177596] 2024-06-26T05:45:33.8428732Z 2024-06-26T05:45:33.8432842Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_3b6c32673f56fc9d_.log 2024-06-26T05:45:33.8434931Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardGradientAccumulation::test_gradient_accumulation 2024-06-26T05:45:33.8435847Z 2024-06-26T05:45:34.8575844Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_frozen.py --verbose 2024-06-26T05:45:34.9427642Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T05:45:34.9428780Z import pkg_resources 2024-06-26T05:45:37.4623924Z No TD results found 2024-06-26T05:45:37.4625414Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T05:45:37.4627360Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T05:45:37.4724928Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:45:37.5083156Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:45:37.5092300Z Running all tests 2024-06-26T05:45:37.5094886Z Running parallel tests on 3 processes 2024-06-26T05:45:37.5095356Z Name: tests to run (est. time: 0.43min) 2024-06-26T05:45:37.5095792Z Serial tests (1): 2024-06-26T05:45:37.5096248Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 2024-06-26T05:45:37.5096782Z Parallel tests (0): 2024-06-26T05:45:37.5099633Z Name: excluded (est. time: 0.0min) 2024-06-26T05:45:37.5100217Z Serial tests (0): 2024-06-26T05:45:37.5100571Z Parallel tests (0): 2024-06-26T05:45:37.5101191Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T05:45:37.5146815Z Running distributed/_composable/fsdp/test_fully_shard_frozen 1/1 ... [2024-06-26 05:45:37.514333] 2024-06-26T05:45:37.5150493Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_frozen.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 05:45:37.514647] 2024-06-26T05:46:05.9279807Z 2024-06-26T05:46:05.9283353Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_frozen_1.1_540370cfecc66f0b_.log 2024-06-26T05:46:05.9286568Z Running 3 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_multi_forward_mixed_requires_grad, test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_train_mixed_requires_grad_across_groups, test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_train_mixed_requires_grad_per_group 2024-06-26T05:46:05.9289029Z 2024-06-26T05:46:06.9682790Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_mixed_precision.py -k test_compute_dtype --verbose 2024-06-26T05:46:07.0536716Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T05:46:07.0537848Z import pkg_resources 2024-06-26T05:46:09.5698030Z No TD results found 2024-06-26T05:46:09.5699375Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T05:46:09.5701077Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T05:46:09.5798509Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:46:09.6158600Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:46:09.6164482Z Running all tests 2024-06-26T05:46:09.6167518Z Running parallel tests on 3 processes 2024-06-26T05:46:09.6168207Z Name: tests to run (est. time: 0.29min) 2024-06-26T05:46:09.6168792Z Serial tests (1): 2024-06-26T05:46:09.6169411Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 2024-06-26T05:46:09.6170191Z Parallel tests (0): 2024-06-26T05:46:09.6172293Z Name: excluded (est. time: 0.0min) 2024-06-26T05:46:09.6172890Z Serial tests (0): 2024-06-26T05:46:09.6173313Z Parallel tests (0): 2024-06-26T05:46:09.6173968Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T05:46:09.6218541Z Running distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 ... [2024-06-26 05:46:09.621414] 2024-06-26T05:46:09.6221419Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_mixed_precision.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_compute_dtype', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 05:46:09.621710] 2024-06-26T05:46:19.6052470Z 2024-06-26T05:46:19.6054340Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_e184dcf7651e54c7_.log 2024-06-26T05:46:19.6056511Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_compute_dtype 2024-06-26T05:46:19.6057435Z 2024-06-26T05:46:20.6480094Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_mixed_precision.py -k test_reduce_dtype --verbose 2024-06-26T05:46:20.7340071Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T05:46:20.7341249Z import pkg_resources 2024-06-26T05:46:23.2533017Z No TD results found 2024-06-26T05:46:23.2534448Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T05:46:23.2555348Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T05:46:23.2634478Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:46:23.3000557Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:46:23.3009791Z Running all tests 2024-06-26T05:46:23.3012591Z Running parallel tests on 3 processes 2024-06-26T05:46:23.3013245Z Name: tests to run (est. time: 0.29min) 2024-06-26T05:46:23.3013720Z Serial tests (1): 2024-06-26T05:46:23.3014229Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 2024-06-26T05:46:23.3014971Z Parallel tests (0): 2024-06-26T05:46:23.3015391Z Name: excluded (est. time: 0.0min) 2024-06-26T05:46:23.3015889Z Serial tests (0): 2024-06-26T05:46:23.3016368Z Parallel tests (0): 2024-06-26T05:46:23.3017128Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T05:46:23.3065183Z Running distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 ... [2024-06-26 05:46:23.306118] 2024-06-26T05:46:23.3068844Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_mixed_precision.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_reduce_dtype', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 05:46:23.306456] 2024-06-26T05:46:35.9921288Z 2024-06-26T05:46:35.9923498Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_fd7ada7fe388b680_.log 2024-06-26T05:46:35.9925773Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_reduce_dtype 2024-06-26T05:46:35.9926720Z 2024-06-26T05:46:36.9910686Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py -k test_clip_grad_norm_2d --verbose 2024-06-26T05:46:37.0766111Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T05:46:37.0767420Z import pkg_resources 2024-06-26T05:46:39.6138066Z No TD results found 2024-06-26T05:46:39.6139566Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T05:46:39.6141228Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T05:46:39.6238038Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:46:39.6598668Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:46:39.6607645Z Running all tests 2024-06-26T05:46:39.6609777Z Running parallel tests on 3 processes 2024-06-26T05:46:39.6610313Z Name: tests to run (est. time: 0.18min) 2024-06-26T05:46:39.6610862Z Serial tests (1): 2024-06-26T05:46:39.6611477Z distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 2024-06-26T05:46:39.6612116Z Parallel tests (0): 2024-06-26T05:46:39.6612502Z Name: excluded (est. time: 0.0min) 2024-06-26T05:46:39.6612914Z Serial tests (0): 2024-06-26T05:46:39.6613260Z Parallel tests (0): 2024-06-26T05:46:39.6613880Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T05:46:39.6660704Z Running distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 ... [2024-06-26 05:46:39.665589] 2024-06-26T05:46:39.6663399Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_clip_grad_norm_2d', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 05:46:39.665911] 2024-06-26T05:46:52.9531394Z 2024-06-26T05:46:52.9533332Z distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_clip_grad_norm__1.1_f4b0c8242d3c9394_.log 2024-06-26T05:46:52.9535578Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py::TestClipGradNormWorldSize4::test_clip_grad_norm_2d 2024-06-26T05:46:52.9536534Z 2024-06-26T05:46:53.9512791Z + python test/run_test.py -i distributed/fsdp/test_fsdp_tp_integration.py -k test_fsdp_tp_integration --verbose 2024-06-26T05:46:54.0371867Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T05:46:54.0373062Z import pkg_resources 2024-06-26T05:46:56.5553461Z No TD results found 2024-06-26T05:46:56.5554632Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T05:46:56.5556272Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T05:46:56.5652469Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:46:56.6015374Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:46:56.6024010Z Running all tests 2024-06-26T05:46:56.6026917Z Running parallel tests on 3 processes 2024-06-26T05:46:56.6027647Z Name: tests to run (est. time: 0.41min) 2024-06-26T05:46:56.6028104Z Serial tests (1): 2024-06-26T05:46:56.6028521Z distributed/fsdp/test_fsdp_tp_integration 1/1 2024-06-26T05:46:56.6028998Z Parallel tests (0): 2024-06-26T05:46:56.6029370Z Name: excluded (est. time: 0.0min) 2024-06-26T05:46:56.6029794Z Serial tests (0): 2024-06-26T05:46:56.6030131Z Parallel tests (0): 2024-06-26T05:46:56.6034155Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T05:46:56.6078928Z Running distributed/fsdp/test_fsdp_tp_integration 1/1 ... [2024-06-26 05:46:56.607582] 2024-06-26T05:46:56.6082862Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_tp_integration.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_fsdp_tp_integration', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 05:46:56.607881] 2024-06-26T05:47:24.0221989Z 2024-06-26T05:47:24.0223760Z distributed/fsdp/test_fsdp_tp_integration 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_tp_integration_1.1_98b2af498d872a3c_.log 2024-06-26T05:47:24.0226686Z Running 3 items in this shard: test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_extension_grad, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_integration, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_sync_module_state 2024-06-26T05:47:24.0228453Z 2024-06-26T05:47:25.0580039Z + python test/run_test.py --include distributed/test_dynamo_distributed distributed/test_inductor_collectives --verbose 2024-06-26T05:47:25.1434395Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T05:47:25.1435561Z import pkg_resources 2024-06-26T05:47:27.6786781Z No TD results found 2024-06-26T05:47:27.6788113Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T05:47:27.6789786Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T05:47:27.6886908Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:47:27.7252627Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-06-26T05:47:27.7259458Z Running all tests 2024-06-26T05:47:27.7262469Z Running parallel tests on 3 processes 2024-06-26T05:47:27.7262950Z Name: tests to run (est. time: 4.8min) 2024-06-26T05:47:27.7263404Z Serial tests (2): 2024-06-26T05:47:27.7263793Z distributed/test_dynamo_distributed 1/1 2024-06-26T05:47:27.7264444Z distributed/test_inductor_collectives 1/1 2024-06-26T05:47:27.7264912Z Parallel tests (0): 2024-06-26T05:47:27.7265283Z Name: excluded (est. time: 0.0min) 2024-06-26T05:47:27.7265697Z Serial tests (0): 2024-06-26T05:47:27.7266032Z Parallel tests (0): 2024-06-26T05:47:27.7266651Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T05:47:27.7314552Z Running distributed/test_dynamo_distributed 1/1 ... [2024-06-26 05:47:27.731147] 2024-06-26T05:47:27.7318307Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_dynamo_distributed.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 05:47:27.731442] 2024-06-26T05:50:25.2192847Z 2024-06-26T05:50:25.2195040Z distributed/test_dynamo_distributed 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_dynamo_distributed_1.1_32611f7cef1567b4_.log 2024-06-26T05:50:25.2212765Z Running 38 items in this shard: test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_call_method_forward, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_issue90375, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_symbol_splitting, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_baseline_aot_eager_multiprocess, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_setattr, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager_static_graph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor_static_graph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_aot_autograd, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_async_subclass_no_specialize, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_custom_layer, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_aot_eager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_empty_graph_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_diff_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_same_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_orig_params_assert, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_guards, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_register_attr_or_module, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_staticmethod, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_ctx_manager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_inference, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_training, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_transpose, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_higher_order_op, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ignored_parameters, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_no_split 2024-06-26T05:50:25.2231023Z 2024-06-26T05:50:25.2231651Z Running distributed/test_inductor_collectives 1/1 ... [2024-06-26 05:50:25.219235] 2024-06-26T05:50:25.2234023Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_inductor_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 05:50:25.219578] 2024-06-26T05:51:58.5418861Z 2024-06-26T05:51:58.5420781Z distributed/test_inductor_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_inductor_collectives_1.1_5b3fd2a163c15304_.log 2024-06-26T05:51:58.5447411Z Running 47 items in this shard: test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor_split_sizes_none, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_contiguous_input, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_into_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_output_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor_cudagraph_trees, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_input_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_broadcast_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_c10d_functional_tagged_pt2_compliant, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_eager_allreduce_inductor_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_inductor_allreduce_eager_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_permute_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_reduce_scatter_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_backwards, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source_GroupMember_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source__get_default_group, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source_group_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_graphbreaks_unsupported_async_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_pg_var, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_args_match, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_list, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_to_all_single, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_kwargs, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_kwargs_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_positional, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_positional_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_unspecified, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op0, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op1, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op2, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op3, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op4, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_reduce_scatter, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_support_collective_op_with_async_op_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor_pg, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allgather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allreduce, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_reduce_scatter_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_all_gather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_doesnt_mutate_shared, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_reduce_scatter_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_single_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_steal_buffer, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_meta 2024-06-26T05:51:58.5472843Z 2024-06-26T05:51:59.6128146Z + assert_git_not_dirty 2024-06-26T05:51:59.6128984Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-06-26T05:51:59.6130306Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *xla* ]] 2024-06-26T05:51:59.6133034Z ++ git status --porcelain 2024-06-26T05:51:59.6133523Z ++ grep -v '?? third_party' 2024-06-26T05:52:01.6906462Z ++ true 2024-06-26T05:52:01.6906806Z + git_status= 2024-06-26T05:52:01.6907345Z + [[ -n '' ]] 2024-06-26T05:52:01.6910038Z + cleanup_workspace 2024-06-26T05:52:01.6910894Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-06-26T05:52:01.6912016Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-06-26T05:52:01.6912958Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-06-26T05:52:01.6913616Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-06-26T05:52:01.6914439Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-06-26T05:52:01.6915340Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-06-26T05:52:01.6916064Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-06-26T05:52:02.1045302Z ##[group]Run cat test/**/*_toprint.log || true 2024-06-26T05:52:02.1045854Z cat test/**/*_toprint.log || true 2024-06-26T05:52:02.1053545Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:52:02.1054072Z env: 2024-06-26T05:52:02.1054366Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:52:02.1054848Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:52:02.1055611Z DOCKER_CONTAINER_ID: f816bf66af59392d564c87e8387c46db59db2501aae5f8b2c2310f41077fe0f2 2024-06-26T05:52:02.1056267Z ##[endgroup] 2024-06-26T05:52:02.1123306Z cat: test/**/*_toprint.log: No such file or directory 2024-06-26T05:52:02.1154166Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-06-26T05:52:02.1154675Z kill "$MONITOR_SCRIPT_PID" 2024-06-26T05:52:02.1162431Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:52:02.1162962Z env: 2024-06-26T05:52:02.1163269Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:52:02.1163750Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:52:02.1164522Z DOCKER_CONTAINER_ID: f816bf66af59392d564c87e8387c46db59db2501aae5f8b2c2310f41077fe0f2 2024-06-26T05:52:02.1165307Z MONITOR_SCRIPT_PID: 3464 2024-06-26T05:52:02.1165675Z ##[endgroup] 2024-06-26T05:52:02.1355396Z Prepare all required actions 2024-06-26T05:52:02.1355873Z Getting action download info 2024-06-26T05:52:02.3113599Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-06-26T05:52:02.4485513Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-06-26T05:52:02.4485999Z with: 2024-06-26T05:52:02.4486561Z file-suffix: test-inductor_distributed-1-1-linux.g5.12xlarge.nvidia.gpu_26688910858 2024-06-26T05:52:02.4487255Z s3-bucket: gha-artifacts 2024-06-26T05:52:02.4487606Z env: 2024-06-26T05:52:02.4487890Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:52:02.4488366Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:52:02.4489198Z DOCKER_CONTAINER_ID: f816bf66af59392d564c87e8387c46db59db2501aae5f8b2c2310f41077fe0f2 2024-06-26T05:52:02.4489849Z ##[endgroup] 2024-06-26T05:52:02.4515502Z ##[group]Run # Remove any previous test jsons if they exist 2024-06-26T05:52:02.4516190Z # Remove any previous test jsons if they exist 2024-06-26T05:52:02.4516744Z rm -f test-jsons-*.zip 2024-06-26T05:52:02.4517298Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-06-26T05:52:02.4525344Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:52:02.4525880Z env: 2024-06-26T05:52:02.4526188Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:52:02.4526674Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:52:02.4527449Z DOCKER_CONTAINER_ID: f816bf66af59392d564c87e8387c46db59db2501aae5f8b2c2310f41077fe0f2 2024-06-26T05:52:02.4528385Z FILE_SUFFIX: test-inductor_distributed-1-1-linux.g5.12xlarge.nvidia.gpu_26688910858 2024-06-26T05:52:02.4529193Z ##[endgroup] 2024-06-26T05:52:02.4710022Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-06-26T05:52:02.4739891Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-06-26T05:52:02.4741316Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-06-26T05:52:02.4746650Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-06-26T05:52:02.4748638Z adding: test/test-reports/td_exclusions-70a6c7f12e2ecde929b2.json (deflated 25%) 2024-06-26T05:52:02.4749636Z adding: test/test-reports/td_exclusions-11a629e8e83208ede0eb.json (deflated 24%) 2024-06-26T05:52:02.4750599Z adding: test/test-reports/td_exclusions-9e91abb99ed24ad4c54f.json (deflated 24%) 2024-06-26T05:52:02.4751551Z adding: test/test-reports/td_exclusions-444bfc16768f3288bfeb.json (deflated 13%) 2024-06-26T05:52:02.4752498Z adding: test/test-reports/td_exclusions-563bdf351c7a1b196f23.json (deflated 19%) 2024-06-26T05:52:02.4753447Z adding: test/test-reports/td_exclusions-c4627c221475f49c80dc.json (deflated 18%) 2024-06-26T05:52:02.4754406Z adding: test/test-reports/td_exclusions-8cb87dc38f50f7158591.json (deflated 14%) 2024-06-26T05:52:02.4755361Z adding: test/test-reports/td_exclusions-386f45bcac7c33b2c63c.json (deflated 15%) 2024-06-26T05:52:02.4756317Z adding: test/test-reports/td_exclusions-19026a077df40abde182.json (deflated 15%) 2024-06-26T05:52:02.4757268Z adding: test/test-reports/td_exclusions-aec7ccd43db9613811b5.json (deflated 15%) 2024-06-26T05:52:02.4758210Z adding: test/test-reports/td_exclusions-534b36435431574b6940.json (deflated 15%) 2024-06-26T05:52:02.4759146Z adding: test/test-reports/td_exclusions-c7345a8defe87f96bb40.json (deflated 15%) 2024-06-26T05:52:02.4760083Z adding: test/test-reports/td_exclusions-eab3c025de19ee43aba7.json (deflated 15%) 2024-06-26T05:52:02.4761038Z adding: test/test-reports/td_exclusions-0aa1a2673a42e96bf20d.json (deflated 13%) 2024-06-26T05:52:02.4762275Z adding: test/test-reports/td_exclusions-011a0353a00464907278.json (deflated 16%) 2024-06-26T05:52:02.4763225Z adding: test/test-reports/td_exclusions-a415474d39025a7456ea.json (deflated 16%) 2024-06-26T05:52:02.4764170Z adding: test/test-reports/td_exclusions-8fe9ad963deeed3239e9.json (deflated 15%) 2024-06-26T05:52:02.4765383Z adding: test/test-reports/td_exclusions-210c714cda5a844f5345.json (deflated 14%) 2024-06-26T05:52:02.4766434Z adding: test/test-reports/td_exclusions-b841fcc1a3e5cd409ccd.json (deflated 36%) 2024-06-26T05:52:02.4767292Z adding: test/.pytorch-slow-tests.json (deflated 77%) 2024-06-26T05:52:02.4768033Z adding: test/.pytorch-disabled-tests.json (deflated 84%) 2024-06-26T05:52:02.4791282Z ##[group]Run # Remove any previous test reports if they exist 2024-06-26T05:52:02.4791950Z # Remove any previous test reports if they exist 2024-06-26T05:52:02.4792508Z rm -f test-reports-*.zip 2024-06-26T05:52:02.4793130Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-06-26T05:52:02.4800956Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:52:02.4801475Z env: 2024-06-26T05:52:02.4801779Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:52:02.4802268Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:52:02.4803034Z DOCKER_CONTAINER_ID: f816bf66af59392d564c87e8387c46db59db2501aae5f8b2c2310f41077fe0f2 2024-06-26T05:52:02.4803963Z FILE_SUFFIX: test-inductor_distributed-1-1-linux.g5.12xlarge.nvidia.gpu_26688910858 2024-06-26T05:52:02.4804640Z ##[endgroup] 2024-06-26T05:52:02.4978725Z adding: test/test-reports/python-pytest/inductor.test_torchinductor/inductor.test_torchinductor-4096ccfb6fd76bd2.xml (deflated 73%) 2024-06-26T05:52:02.4980260Z adding: test/test-reports/python-pytest/inductor.test_aot_inductor/inductor.test_aot_inductor-1298124b913055e9.xml (deflated 73%) 2024-06-26T05:52:02.4981752Z adding: test/test-reports/python-pytest/inductor.test_aot_inductor/inductor.test_aot_inductor-2a8ab9ed64139b67.xml (deflated 76%) 2024-06-26T05:52:02.4983520Z adding: test/test-reports/python-pytest/distributed.test_c10d_functional_native/distributed.test_c10d_functional_native-09ed44e514b58479.xml (deflated 81%) 2024-06-26T05:52:02.4985521Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor_compile/distributed._tensor.test_dtensor_compile-db6156883ba36d4d.xml (deflated 79%) 2024-06-26T05:52:02.4987436Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_fsdp_2d_parallel/distributed.tensor.parallel.test_fsdp_2d_parallel-d65c9adabceffb22.xml (deflated 81%) 2024-06-26T05:52:02.4989444Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_comm/distributed._composable.fsdp.test_fully_shard_comm-800ee427f492a5df.xml (deflated 78%) 2024-06-26T05:52:02.4991464Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_training/distributed._composable.fsdp.test_fully_shard_training-41198ab67425c477.xml (deflated 53%) 2024-06-26T05:52:02.4993749Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_training/distributed._composable.fsdp.test_fully_shard_training-336cb30de6d1b5b0.xml (deflated 35%) 2024-06-26T05:52:02.4995831Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_training/distributed._composable.fsdp.test_fully_shard_training-b1c847464534712b.xml (deflated 36%) 2024-06-26T05:52:02.4997882Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_training/distributed._composable.fsdp.test_fully_shard_training-174556f46b9457b8.xml (deflated 35%) 2024-06-26T05:52:02.4999917Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_training/distributed._composable.fsdp.test_fully_shard_training-cc234446741fa9b1.xml (deflated 36%) 2024-06-26T05:52:02.5001968Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_training/distributed._composable.fsdp.test_fully_shard_training-2759261951d4837d.xml (deflated 36%) 2024-06-26T05:52:02.5004008Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_frozen/distributed._composable.fsdp.test_fully_shard_frozen-27a9a4d7c90c522d.xml (deflated 58%) 2024-06-26T05:52:02.5006258Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_mixed_precision/distributed._composable.fsdp.test_fully_shard_mixed_precision-aea4ad8d0a6a4692.xml (deflated 38%) 2024-06-26T05:52:02.5008466Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_mixed_precision/distributed._composable.fsdp.test_fully_shard_mixed_precision-a59fefb388ae838c.xml (deflated 38%) 2024-06-26T05:52:02.5010718Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_clip_grad_norm_/distributed._composable.fsdp.test_fully_shard_clip_grad_norm_-e00095d48d7307e8.xml (deflated 38%) 2024-06-26T05:52:02.5012769Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_tp_integration/distributed.fsdp.test_fsdp_tp_integration-833dc8d0cc7ed263.xml (deflated 58%) 2024-06-26T05:52:02.5014555Z adding: test/test-reports/python-pytest/distributed.test_dynamo_distributed/distributed.test_dynamo_distributed-e3fe32843098a727.xml (deflated 87%) 2024-06-26T05:52:02.5016335Z adding: test/test-reports/python-pytest/distributed.test_inductor_collectives/distributed.test_inductor_collectives-e86b64ec724f6205.xml (deflated 87%) 2024-06-26T05:52:02.5038284Z ##[group]Run # Remove any previous usage logs if they exist 2024-06-26T05:52:02.5038958Z # Remove any previous usage logs if they exist 2024-06-26T05:52:02.5039502Z rm -f logs-*.zip 2024-06-26T05:52:02.5040227Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-06-26T05:52:02.5041056Z # so check to see if the file exists first 2024-06-26T05:52:02.5041585Z if [ -f 'usage_log.txt' ]; then 2024-06-26T05:52:02.5042238Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-06-26T05:52:02.5042776Z fi 2024-06-26T05:52:02.5043184Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-06-26T05:52:02.5043791Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-06-26T05:52:02.5044326Z fi 2024-06-26T05:52:02.5051050Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:52:02.5051583Z env: 2024-06-26T05:52:02.5051891Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:52:02.5052397Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:52:02.5053179Z DOCKER_CONTAINER_ID: f816bf66af59392d564c87e8387c46db59db2501aae5f8b2c2310f41077fe0f2 2024-06-26T05:52:02.5054135Z FILE_SUFFIX: test-inductor_distributed-1-1-linux.g5.12xlarge.nvidia.gpu_26688910858 2024-06-26T05:52:02.5054812Z ##[endgroup] 2024-06-26T05:52:02.5098511Z adding: usage_log.txt (deflated 92%) 2024-06-26T05:52:02.5287199Z adding: test/test-reports/inductor.test_torchinductor_1.1_4a28d8ddaa1c8330_.log (deflated 62%) 2024-06-26T05:52:02.5288334Z adding: test/test-reports/inductor.test_aot_inductor_1.1_e1e98bd85d6f7c5b_.log (deflated 74%) 2024-06-26T05:52:02.5289414Z adding: test/test-reports/inductor.test_aot_inductor_1.1_fac06b80b0bca0da_.log (deflated 74%) 2024-06-26T05:52:02.5290569Z adding: test/test-reports/distributed.test_c10d_functional_native_1.1_565881f8a41c7ffe_.log (deflated 87%) 2024-06-26T05:52:02.5291776Z adding: test/test-reports/distributed._tensor.test_dtensor_compile_1.1_d81e077b58a4ce3d_.log (deflated 80%) 2024-06-26T05:52:02.5295305Z adding: test/test-reports/distributed.tensor.parallel.test_fsdp_2d_parallel_1.1_c8ac91e729bc824b_.log (deflated 96%) 2024-06-26T05:52:02.5296651Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_comm_1.1_0515b49296d7f582_.log (deflated 78%) 2024-06-26T05:52:02.5299375Z adding: test/test-reports/distributed.test_inductor_collectives_1.1_5b3fd2a163c15304_.log (deflated 89%) 2024-06-26T05:52:02.5300665Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_14c84f6bd12e4ef1_.log (deflated 72%) 2024-06-26T05:52:02.5302724Z adding: test/test-reports/distributed.test_dynamo_distributed_1.1_32611f7cef1567b4_.log (deflated 88%) 2024-06-26T05:52:02.5304167Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_8cf153b0073d98eb_.log (deflated 61%) 2024-06-26T05:52:02.5305738Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_376ce079db104d5e_.log (deflated 57%) 2024-06-26T05:52:02.5307100Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_2d98cb6ae7f90560_.log (deflated 57%) 2024-06-26T05:52:02.5308464Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_93bd0bc415b1f7fb_.log (deflated 81%) 2024-06-26T05:52:02.5309817Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_3b6c32673f56fc9d_.log (deflated 58%) 2024-06-26T05:52:02.5311168Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_frozen_1.1_540370cfecc66f0b_.log (deflated 83%) 2024-06-26T05:52:02.5312548Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_fd7ada7fe388b680_.log (deflated 58%) 2024-06-26T05:52:02.5314215Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_e184dcf7651e54c7_.log (deflated 58%) 2024-06-26T05:52:02.5315556Z adding: test/test-reports/distributed.fsdp.test_fsdp_tp_integration_1.1_98b2af498d872a3c_.log (deflated 66%) 2024-06-26T05:52:02.5316900Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_clip_grad_norm__1.1_f4b0c8242d3c9394_.log (deflated 58%) 2024-06-26T05:52:02.5339817Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-06-26T05:52:02.5340563Z # Remove any previous debugging artifacts if they exist 2024-06-26T05:52:02.5341143Z rm -f debug-*.zip 2024-06-26T05:52:02.5341653Z if [ -d 'test/debug' ]; then 2024-06-26T05:52:02.5342174Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-06-26T05:52:02.5342679Z fi 2024-06-26T05:52:02.5350307Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:52:02.5350828Z env: 2024-06-26T05:52:02.5351126Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:52:02.5351609Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:52:02.5352382Z DOCKER_CONTAINER_ID: f816bf66af59392d564c87e8387c46db59db2501aae5f8b2c2310f41077fe0f2 2024-06-26T05:52:02.5353322Z FILE_SUFFIX: test-inductor_distributed-1-1-linux.g5.12xlarge.nvidia.gpu_26688910858 2024-06-26T05:52:02.5353994Z ##[endgroup] 2024-06-26T05:52:02.5432663Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T05:52:02.5433131Z with: 2024-06-26T05:52:02.5433436Z s3-bucket: gha-artifacts 2024-06-26T05:52:02.5433887Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T05:52:02.5434381Z retention-days: 14 2024-06-26T05:52:02.5434750Z if-no-files-found: warn 2024-06-26T05:52:02.5435139Z path: test-jsons-*.zip 2024-06-26T05:52:02.5435506Z name: artifact 2024-06-26T05:52:02.5435823Z region: us-east-1 2024-06-26T05:52:02.5436159Z env: 2024-06-26T05:52:02.5436458Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:52:02.5436957Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:52:02.5437713Z DOCKER_CONTAINER_ID: f816bf66af59392d564c87e8387c46db59db2501aae5f8b2c2310f41077fe0f2 2024-06-26T05:52:02.5438377Z ##[endgroup] 2024-06-26T05:52:02.8486526Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T05:52:02.8487515Z With the provided path, there will be 1 file uploaded 2024-06-26T05:52:02.8488199Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T05:52:02.8513719Z Starting upload of test-jsons-test-inductor_distributed-1-1-linux.g5.12xlarge.nvidia.gpu_26688910858.zip 2024-06-26T05:52:02.9918527Z Finished upload of test-jsons-test-inductor_distributed-1-1-linux.g5.12xlarge.nvidia.gpu_26688910858.zip 2024-06-26T05:52:03.0053178Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T05:52:03.0053635Z with: 2024-06-26T05:52:03.0053934Z s3-bucket: gha-artifacts 2024-06-26T05:52:03.0054381Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T05:52:03.0054875Z retention-days: 14 2024-06-26T05:52:03.0055359Z if-no-files-found: error 2024-06-26T05:52:03.0055759Z path: test-reports-*.zip 2024-06-26T05:52:03.0056130Z name: artifact 2024-06-26T05:52:03.0056445Z region: us-east-1 2024-06-26T05:52:03.0056768Z env: 2024-06-26T05:52:03.0057067Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:52:03.0057544Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:52:03.0058296Z DOCKER_CONTAINER_ID: f816bf66af59392d564c87e8387c46db59db2501aae5f8b2c2310f41077fe0f2 2024-06-26T05:52:03.0058991Z ##[endgroup] 2024-06-26T05:52:03.3172440Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T05:52:03.3173064Z With the provided path, there will be 1 file uploaded 2024-06-26T05:52:03.3173730Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T05:52:03.3199889Z Starting upload of test-reports-test-inductor_distributed-1-1-linux.g5.12xlarge.nvidia.gpu_26688910858.zip 2024-06-26T05:52:03.4157418Z Finished upload of test-reports-test-inductor_distributed-1-1-linux.g5.12xlarge.nvidia.gpu_26688910858.zip 2024-06-26T05:52:03.4306711Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T05:52:03.4307170Z with: 2024-06-26T05:52:03.4307471Z s3-bucket: gha-artifacts 2024-06-26T05:52:03.4307927Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T05:52:03.4308427Z retention-days: 14 2024-06-26T05:52:03.4308782Z if-no-files-found: ignore 2024-06-26T05:52:03.4309222Z path: logs-*.zip 2024-06-26T05:52:03.4309558Z name: artifact 2024-06-26T05:52:03.4309875Z region: us-east-1 2024-06-26T05:52:03.4310194Z env: 2024-06-26T05:52:03.4310497Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:52:03.4310980Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:52:03.4311869Z DOCKER_CONTAINER_ID: f816bf66af59392d564c87e8387c46db59db2501aae5f8b2c2310f41077fe0f2 2024-06-26T05:52:03.4312534Z ##[endgroup] 2024-06-26T05:52:03.7454605Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T05:52:03.7455270Z With the provided path, there will be 1 file uploaded 2024-06-26T05:52:03.7455916Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T05:52:03.7482792Z Starting upload of logs-test-inductor_distributed-1-1-linux.g5.12xlarge.nvidia.gpu_26688910858.zip 2024-06-26T05:52:03.8469722Z Finished upload of logs-test-inductor_distributed-1-1-linux.g5.12xlarge.nvidia.gpu_26688910858.zip 2024-06-26T05:52:03.8609604Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T05:52:03.8610068Z with: 2024-06-26T05:52:03.8610363Z s3-bucket: gha-artifacts 2024-06-26T05:52:03.8610814Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T05:52:03.8611312Z retention-days: 14 2024-06-26T05:52:03.8611677Z if-no-files-found: ignore 2024-06-26T05:52:03.8612060Z path: debug-*.zip 2024-06-26T05:52:03.8612383Z name: artifact 2024-06-26T05:52:03.8612697Z region: us-east-1 2024-06-26T05:52:03.8613015Z env: 2024-06-26T05:52:03.8613308Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:52:03.8613803Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:52:03.8614559Z DOCKER_CONTAINER_ID: f816bf66af59392d564c87e8387c46db59db2501aae5f8b2c2310f41077fe0f2 2024-06-26T05:52:03.8615224Z ##[endgroup] 2024-06-26T05:52:04.1627398Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-06-26T05:52:04.1769863Z ##[group]Run # shellcheck disable=SC2156 2024-06-26T05:52:04.1770371Z # shellcheck disable=SC2156 2024-06-26T05:52:04.1771200Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-06-26T05:52:04.1779617Z shell: /usr/bin/bash -e {0} 2024-06-26T05:52:04.1779986Z env: 2024-06-26T05:52:04.1780324Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:52:04.1780810Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:52:04.1781567Z DOCKER_CONTAINER_ID: f816bf66af59392d564c87e8387c46db59db2501aae5f8b2c2310f41077fe0f2 2024-06-26T05:52:04.1782238Z ##[endgroup] 2024-06-26T05:52:04.3415428Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-06-26T05:52:04.3416021Z with: 2024-06-26T05:52:04.3416312Z env: 2024-06-26T05:52:04.3416620Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:52:04.3417097Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:52:04.3417865Z DOCKER_CONTAINER_ID: f816bf66af59392d564c87e8387c46db59db2501aae5f8b2c2310f41077fe0f2 2024-06-26T05:52:04.3418530Z ##[endgroup] 2024-06-26T05:52:04.3521580Z ##[group]Run set -eou pipefail 2024-06-26T05:52:04.3522004Z set -eou pipefail 2024-06-26T05:52:04.3522372Z  2024-06-26T05:52:04.3522901Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-06-26T05:52:04.3523564Z for _ in $(seq 1440); do 2024-06-26T05:52:04.3524058Z  # Break if no ssh session exists anymore 2024-06-26T05:52:04.3524571Z  if [ "$(who)" = "" ]; then 2024-06-26T05:52:04.3525115Z  break 2024-06-26T05:52:04.3525465Z  fi 2024-06-26T05:52:04.3525768Z  echo "." 2024-06-26T05:52:04.3526106Z  sleep 5 2024-06-26T05:52:04.3526431Z done 2024-06-26T05:52:04.3534068Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:52:04.3534579Z env: 2024-06-26T05:52:04.3534881Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:52:04.3535362Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:52:04.3536134Z DOCKER_CONTAINER_ID: f816bf66af59392d564c87e8387c46db59db2501aae5f8b2c2310f41077fe0f2 2024-06-26T05:52:04.3536781Z ##[endgroup] 2024-06-26T05:52:04.3558444Z Holding runner for 2 hours until all ssh sessions have logged out 2024-06-26T05:52:04.3598616Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T05:52:04.3599546Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T05:52:04.3600174Z # shellcheck disable=SC2046 2024-06-26T05:52:04.3600680Z docker stop $(docker ps -q) || true 2024-06-26T05:52:04.3601182Z # Prune all of the docker images 2024-06-26T05:52:04.3601665Z docker system prune -af 2024-06-26T05:52:04.3609639Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:52:04.3610145Z env: 2024-06-26T05:52:04.3610447Z GIT_DEFAULT_BRANCH: main 2024-06-26T05:52:04.3610933Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T05:52:04.3611689Z DOCKER_CONTAINER_ID: f816bf66af59392d564c87e8387c46db59db2501aae5f8b2c2310f41077fe0f2 2024-06-26T05:52:04.3612337Z ##[endgroup] 2024-06-26T05:52:05.6609664Z f816bf66af59 2024-06-26T05:52:06.1872403Z Deleted Containers: 2024-06-26T05:52:06.1873034Z f816bf66af59392d564c87e8387c46db59db2501aae5f8b2c2310f41077fe0f2 2024-06-26T05:52:06.1873547Z 2024-06-26T05:52:11.1928486Z Deleted Images: 2024-06-26T05:52:11.1931875Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T05:52:11.1934210Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T05:52:11.1935769Z deleted: sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd 2024-06-26T05:52:11.1936620Z deleted: sha256:754ead7012f0b7691885a9660f02e80be6cee8c5665486fb995c1fb06adb6e92 2024-06-26T05:52:11.1937468Z deleted: sha256:fbe4496d8830e11be86d8f222522e7956f32f85bfc06b05ea53961f34a58af74 2024-06-26T05:52:11.1938326Z deleted: sha256:1c83814aa1a1a7e0cc998e3479bdab8e8766b975560cbf00690acf39ed899a80 2024-06-26T05:52:11.1939186Z deleted: sha256:e9ea8e61942c7f8ae4aab97b8d102b01d6bb4b6cf2a605808cd34130a17d8a0f 2024-06-26T05:52:11.1940059Z deleted: sha256:acc5fc32588cd95e082a6ce35e48d0f303ae8f96e3ab65c4e375b85e1f3c4a92 2024-06-26T05:52:11.1940919Z deleted: sha256:a93e53400904d29e4185d3be942ea34f19572ea438962788ae4e9b3d9bf4c498 2024-06-26T05:52:11.1941962Z deleted: sha256:c97092b88b5e4c77c68bbc4b51afab64e38719af1ff58013ec49a696cdd40a6e 2024-06-26T05:52:11.1942861Z deleted: sha256:b132889c90d67ffa53655846603e3fcc2372c23f04dbdbc54690a39d304eb89b 2024-06-26T05:52:11.1943731Z deleted: sha256:915d54081ed607710624abe3f2490114c745858831f4fc13396bd3b4b9a121bc 2024-06-26T05:52:11.1945069Z deleted: sha256:bfa8d446c498c87f535aef102695564de1b22c755e6c63944ae2f5073a2b31ec 2024-06-26T05:52:11.1946212Z deleted: sha256:e71460fb363c29e70bbe357d680aaa7ae694f094ef2f2713609158ad90611744 2024-06-26T05:52:11.1947249Z deleted: sha256:84079bb1b4ee1d0119ae3d3e701de460a1324912297c675425c9a36d30fcdb0f 2024-06-26T05:52:11.1948176Z deleted: sha256:d86ac3ca148d4758cb03b493fdd49109d8f1a2c7c85c4b755493c3c4529553e5 2024-06-26T05:52:11.1949034Z deleted: sha256:532b3ab97c44a673a0d5f62ee46fa6a83248a84de8877754a612355be56ed0a3 2024-06-26T05:52:11.1949872Z deleted: sha256:617e2a35f25ae24500c88c0c9b8475f0687fbcbed3d200013b4c99c6d8941aff 2024-06-26T05:52:11.1950726Z deleted: sha256:b88120bcd5a26a660c09c5df13176e0ef098fb2c8cdb57d74ab0796a3d1194d6 2024-06-26T05:52:11.1951572Z deleted: sha256:c016b2e7764735243205fb76fa9a273bb3dd4bf05532288e3a613ce022131ac3 2024-06-26T05:52:11.1952462Z deleted: sha256:bcaca462c390deddd2995de2d245a7495e7d2e1e94f0edad30a573c83d0588fd 2024-06-26T05:52:11.1953302Z deleted: sha256:9e260eba003d5c25d7b410e5c4c3093845d9e46611896840901c689d7fbf8ee6 2024-06-26T05:52:11.1954131Z deleted: sha256:3e62b86f8b911a9b59c62d55f063e71683ae6a01c22418db5d27930712adaad6 2024-06-26T05:52:11.1954966Z deleted: sha256:905920ccb1d88449b8dde686c34a4b2a8b6f59e78ac44916b189ea3593b9fbbb 2024-06-26T05:52:11.1955795Z deleted: sha256:961201237e03acd5061406238dd7a3c1343ac888d7583212cd0ab5ae1b4f0540 2024-06-26T05:52:11.1956797Z deleted: sha256:70047a2aac62159fa28fb4dbd4567ec2ffbab68fe046967e09acb0eb3e5cb9cb 2024-06-26T05:52:11.1957663Z deleted: sha256:1ec680ad781436ae65fa987f5fa1ed032ecd5c82a6e35eea6707de501f62c0f4 2024-06-26T05:52:11.1958779Z deleted: sha256:ef859623a50e85abf6e6c1dbd6188aa765a66cc7b2bd618a1cbe4825b7b2bd91 2024-06-26T05:52:11.1959749Z deleted: sha256:404ade5cacff59773b93ab438bfa64cd3a171d07f6f168bdc4a5d4bfa0b43526 2024-06-26T05:52:11.1960618Z deleted: sha256:5b27de6ee3d9f2f18217bf18dec725f7b4954c0e48c85d35752634c22d62209b 2024-06-26T05:52:11.1961459Z deleted: sha256:498e39c575b3fa069acaac7e46b91b4e54ae3a05f2ada5c4347091f346f25d4c 2024-06-26T05:52:11.1962295Z deleted: sha256:1405919d2694d65669b1f4c842282021afe89208ce86570e96464903733da2d0 2024-06-26T05:52:11.1963123Z deleted: sha256:d5b41e5d70896c0472fc3e6da0516a17d89cd6635dc75010ecddc31ce9690fad 2024-06-26T05:52:11.1963972Z deleted: sha256:18e9379b62b9d0bb40293d28a652eafaee66a499f66f0776bf14e37a513101b7 2024-06-26T05:52:11.1964816Z deleted: sha256:a781f80ed00ff2926a699fd01a309bb0b4fb7d5655eb652c205bb9c7f811035f 2024-06-26T05:52:11.1965736Z deleted: sha256:c0430973b7cf759075330ee29c79bfd1312c91e6033d9dd883b4b1c29743a171 2024-06-26T05:52:11.1966596Z deleted: sha256:e733b3d66efb9342e5a283376c0e1ddffa47a42fecc26e61ecaf0a3472885d97 2024-06-26T05:52:11.1967450Z deleted: sha256:8929c0b2047f74b84c82c39eff9077a92ae1a828cde00c9ab78fb3e0ed5e0283 2024-06-26T05:52:11.1968284Z deleted: sha256:f5b56155fb58a0a60fe6391d3581cf91610ed69d4f0a321257c19944da296a52 2024-06-26T05:52:11.1969124Z deleted: sha256:b54d2cb355238d1aa91e58df93064447b80f53706d1ca62823acc834f5ac54fe 2024-06-26T05:52:11.1969970Z deleted: sha256:e7311285af9eeddb51aa5629aa98386e89f1456ff32843b6aee6443247edf66c 2024-06-26T05:52:11.1970817Z deleted: sha256:67657ceec347da8dc8085d5731633efeac115ba677fa4b7c5966c4260f7abab0 2024-06-26T05:52:11.1971658Z deleted: sha256:b496e50409dbf414930fc9b4437aa0aa4ce27bc7840e050ae459aaffcf9aca86 2024-06-26T05:52:11.1972497Z deleted: sha256:8f5276170cc4a60d36ad7b964a502306b29546254cedcbcb101ea9123497d2c0 2024-06-26T05:52:11.1973330Z deleted: sha256:53d524d1564745e96a7e515462171b73184d9f1ff5454b32fcfbbca39d361467 2024-06-26T05:52:11.1974181Z deleted: sha256:2acbadb00d69b6aca575f04baba4ea26ae2b5c2d85f09945004ad93a78a8eb75 2024-06-26T05:52:11.1975169Z deleted: sha256:02665efeff878aeb603c1edb2e21cb700c171c025d64f5aae1b2b20b12f3964a 2024-06-26T05:52:11.1976036Z deleted: sha256:19bd23e9a51dfe883a8b8d2ff6d66ff1b3476b8912856406bb79b81762aa6d4b 2024-06-26T05:52:11.1976881Z deleted: sha256:73aa8637bf9299d6298dabd858480c274a62117388f2ca1d5e9eb15c6d1e4155 2024-06-26T05:52:11.1977717Z deleted: sha256:5f8968406305f348b1caebc0e035187d6e9965ace7b9d5f2f21114d37c9fe2e1 2024-06-26T05:52:11.1978561Z deleted: sha256:d827104914db0c503cf5c81f13e2456bf4efce42d4cf1f8d5d70838207278bee 2024-06-26T05:52:11.1979411Z deleted: sha256:867b219724f2f1aaf56f42e3e9f03a5d64bdca8a6a72407efd10328aeea5d510 2024-06-26T05:52:11.1980272Z deleted: sha256:b9ff2a7cceb9237b26bcb17728edf8a558eec5a76f63626e84f80541dfc0f1d4 2024-06-26T05:52:11.1981140Z deleted: sha256:be73f509106819007ce2b0f5a1301ac56ae583cb6c1fb69bee93efa0840368f1 2024-06-26T05:52:11.1981993Z deleted: sha256:2a11c07c915c3fdbb4e73096dea73ea950c970d50fe1a5764060bd62b7b7cf2c 2024-06-26T05:52:11.1982888Z deleted: sha256:5e9989622e2abc1c72f8184343507b45d75e4ff971edb68b96cbcd75ae484ff5 2024-06-26T05:52:11.1983752Z deleted: sha256:05ce642bd9cf5a9205bff107df28b1d6cef7d5f6ab3ea944d28c032ec61270df 2024-06-26T05:52:11.1984831Z deleted: sha256:017702617f6c07b73f2088678f4710b794cfe897a213a5cf122a0e2519eae0ac 2024-06-26T05:52:11.1985661Z deleted: sha256:44b3b37bb1ca39dc5cf356de41a23a5c12f605a1992813e408087fb2b0ad6d1c 2024-06-26T05:52:11.1986494Z deleted: sha256:8c387982e9f27db357a1d149dc149cabc0bd04668ad6956387a5072682dada67 2024-06-26T05:52:11.1987337Z deleted: sha256:10d415dca4e3fbb88a747b40317f3c6333c041ba8f73855f67fb9fe981fae5ee 2024-06-26T05:52:11.1988179Z deleted: sha256:6c65469bf0f53dcd625faefe363b8f3136c08388a6cf64d9158e8576e4ee89b1 2024-06-26T05:52:11.1989145Z deleted: sha256:027db5d8b76deeefbd8c34b485ab7ff14ed2a3e6800425c1ce0d77c2e9fdd4ab 2024-06-26T05:52:11.1989984Z deleted: sha256:24941234ad28cf5c8174d7cf18326c10b4e17f2c2137672ea0322df461597dd7 2024-06-26T05:52:11.1990831Z deleted: sha256:240ea2eebfa8519d6c2d88f4c99e66ce4e038d000f93ce8d03726c01d81180ef 2024-06-26T05:52:11.1991678Z deleted: sha256:09e823cb391a69f6a195e3ca8a41624b5f8b34eee14c6eb36277de814a91966e 2024-06-26T05:52:11.1992508Z deleted: sha256:730479d695169a6593bf0ac16b377491dce65175a6d158e012422367d04577fd 2024-06-26T05:52:11.1993336Z deleted: sha256:b0a30f9f7bd5459ff662453b9346a7ba8ea8db1191528345c1953eab4aa2f54e 2024-06-26T05:52:11.1994173Z deleted: sha256:75cefa87effaa3ad361c74642839923101f0f9982f814f3c9e2ed4ad52f2a008 2024-06-26T05:52:11.1995010Z deleted: sha256:6f97db87ba9512e615e871c6982000602982e1e5a5be334a56e7292251e84e00 2024-06-26T05:52:11.1995853Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-06-26T05:52:11.1996702Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-06-26T05:52:11.1997532Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-06-26T05:52:11.1998372Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-06-26T05:52:11.1999233Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-06-26T05:52:11.2000075Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-06-26T05:52:11.2000926Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-06-26T05:52:11.2001786Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-06-26T05:52:11.2002651Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-06-26T05:52:11.2003510Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-06-26T05:52:11.2004343Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-06-26T05:52:11.2004924Z 2024-06-26T05:52:11.2005083Z Total reclaimed space: 31.39GB 2024-06-26T05:52:11.2069810Z Post job cleanup. 2024-06-26T05:52:11.2119343Z Post job cleanup. 2024-06-26T05:52:11.2953380Z [command]/usr/bin/git version 2024-06-26T05:52:11.2988201Z git version 2.40.1 2024-06-26T05:52:11.3026265Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/a7489e33-8baf-4cb8-b025-2302ff6c5ad3' before making global git config changes 2024-06-26T05:52:11.3027545Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T05:52:11.3031800Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T05:52:11.3059075Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T05:52:11.3082013Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T05:52:11.3317954Z Entering 'android/libs/fbjni' 2024-06-26T05:52:11.3355619Z Entering 'third_party/FP16' 2024-06-26T05:52:11.3394267Z Entering 'third_party/FXdiv' 2024-06-26T05:52:11.3432152Z Entering 'third_party/NNPACK' 2024-06-26T05:52:11.3470900Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T05:52:11.3508069Z Entering 'third_party/XNNPACK' 2024-06-26T05:52:11.3554684Z Entering 'third_party/benchmark' 2024-06-26T05:52:11.3592253Z Entering 'third_party/cpp-httplib' 2024-06-26T05:52:11.3628220Z Entering 'third_party/cpuinfo' 2024-06-26T05:52:11.3665111Z Entering 'third_party/cudnn_frontend' 2024-06-26T05:52:11.3703127Z Entering 'third_party/cutlass' 2024-06-26T05:52:11.3746301Z Entering 'third_party/eigen' 2024-06-26T05:52:11.3788636Z Entering 'third_party/fbgemm' 2024-06-26T05:52:11.3826331Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T05:52:11.3864693Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T05:52:11.3902342Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T05:52:11.3941648Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T05:52:11.3979870Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T05:52:11.4017630Z Entering 'third_party/flatbuffers' 2024-06-26T05:52:11.4058147Z Entering 'third_party/fmt' 2024-06-26T05:52:11.4096240Z Entering 'third_party/foxi' 2024-06-26T05:52:11.4133682Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T05:52:11.4172151Z Entering 'third_party/gloo' 2024-06-26T05:52:11.4209984Z Entering 'third_party/googletest' 2024-06-26T05:52:11.4247761Z Entering 'third_party/ideep' 2024-06-26T05:52:11.4284560Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T05:52:11.4326629Z Entering 'third_party/ittapi' 2024-06-26T05:52:11.4364082Z Entering 'third_party/kineto' 2024-06-26T05:52:11.4400480Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T05:52:11.4439222Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T05:52:11.4478108Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T05:52:11.4518598Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T05:52:11.4554251Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T05:52:11.4591510Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T05:52:11.4627770Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T05:52:11.4665746Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T05:52:11.4703481Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T05:52:11.4743806Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T05:52:11.4782602Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T05:52:11.4819466Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T05:52:11.4858945Z Entering 'third_party/mimalloc' 2024-06-26T05:52:11.4897937Z Entering 'third_party/nccl/nccl' 2024-06-26T05:52:11.4936280Z Entering 'third_party/nlohmann' 2024-06-26T05:52:11.4976006Z Entering 'third_party/onnx' 2024-06-26T05:52:11.5026891Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T05:52:11.5068013Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T05:52:11.5107810Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T05:52:11.5148564Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T05:52:11.5185295Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T05:52:11.5223291Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T05:52:11.5260213Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T05:52:11.5298625Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T05:52:11.5339415Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T05:52:11.5381271Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T05:52:11.5418646Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T05:52:11.5457158Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T05:52:11.5496425Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T05:52:11.5550570Z Entering 'third_party/pocketfft' 2024-06-26T05:52:11.5588643Z Entering 'third_party/protobuf' 2024-06-26T05:52:11.5628008Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T05:52:11.5665812Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T05:52:11.5710316Z Entering 'third_party/psimd' 2024-06-26T05:52:11.5749334Z Entering 'third_party/pthreadpool' 2024-06-26T05:52:11.5789692Z Entering 'third_party/pybind11' 2024-06-26T05:52:11.5828478Z Entering 'third_party/python-peachpy' 2024-06-26T05:52:11.5867242Z Entering 'third_party/sleef' 2024-06-26T05:52:11.5905513Z Entering 'third_party/tensorpipe' 2024-06-26T05:52:11.5942272Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T05:52:11.5981094Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T05:52:11.6018533Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T05:52:11.6055794Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T05:52:11.6091641Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T05:52:11.6148624Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T05:52:11.6172011Z http.https://github.com/.extraheader 2024-06-26T05:52:11.6180881Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-06-26T05:52:11.6211845Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T05:52:11.6447215Z Entering 'android/libs/fbjni' 2024-06-26T05:52:11.6468709Z http.https://github.com/.extraheader 2024-06-26T05:52:11.6495258Z Entering 'third_party/FP16' 2024-06-26T05:52:11.6518175Z http.https://github.com/.extraheader 2024-06-26T05:52:11.6544957Z Entering 'third_party/FXdiv' 2024-06-26T05:52:11.6569111Z http.https://github.com/.extraheader 2024-06-26T05:52:11.6595017Z Entering 'third_party/NNPACK' 2024-06-26T05:52:11.6618492Z http.https://github.com/.extraheader 2024-06-26T05:52:11.6645263Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T05:52:11.6672996Z http.https://github.com/.extraheader 2024-06-26T05:52:11.6699733Z Entering 'third_party/XNNPACK' 2024-06-26T05:52:11.6729492Z http.https://github.com/.extraheader 2024-06-26T05:52:11.6768464Z Entering 'third_party/benchmark' 2024-06-26T05:52:11.6791324Z http.https://github.com/.extraheader 2024-06-26T05:52:11.6817282Z Entering 'third_party/cpp-httplib' 2024-06-26T05:52:11.6841046Z http.https://github.com/.extraheader 2024-06-26T05:52:11.6868023Z Entering 'third_party/cpuinfo' 2024-06-26T05:52:11.6891758Z http.https://github.com/.extraheader 2024-06-26T05:52:11.6918687Z Entering 'third_party/cudnn_frontend' 2024-06-26T05:52:11.6943049Z http.https://github.com/.extraheader 2024-06-26T05:52:11.6968086Z Entering 'third_party/cutlass' 2024-06-26T05:52:11.6991967Z http.https://github.com/.extraheader 2024-06-26T05:52:11.7024084Z Entering 'third_party/eigen' 2024-06-26T05:52:11.7047304Z http.https://github.com/.extraheader 2024-06-26T05:52:11.7074997Z Entering 'third_party/fbgemm' 2024-06-26T05:52:11.7098533Z http.https://github.com/.extraheader 2024-06-26T05:52:11.7125129Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T05:52:11.7152869Z http.https://github.com/.extraheader 2024-06-26T05:52:11.7178498Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T05:52:11.7202476Z http.https://github.com/.extraheader 2024-06-26T05:52:11.7228342Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T05:52:11.7250807Z http.https://github.com/.extraheader 2024-06-26T05:52:11.7280657Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T05:52:11.7308191Z http.https://github.com/.extraheader 2024-06-26T05:52:11.7332187Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T05:52:11.7355747Z http.https://github.com/.extraheader 2024-06-26T05:52:11.7382685Z Entering 'third_party/flatbuffers' 2024-06-26T05:52:11.7407668Z http.https://github.com/.extraheader 2024-06-26T05:52:11.7435721Z Entering 'third_party/fmt' 2024-06-26T05:52:11.7459049Z http.https://github.com/.extraheader 2024-06-26T05:52:11.7493868Z Entering 'third_party/foxi' 2024-06-26T05:52:11.7518253Z http.https://github.com/.extraheader 2024-06-26T05:52:11.7545026Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T05:52:11.7569361Z http.https://github.com/.extraheader 2024-06-26T05:52:11.7595755Z Entering 'third_party/gloo' 2024-06-26T05:52:11.7619116Z http.https://github.com/.extraheader 2024-06-26T05:52:11.7648387Z Entering 'third_party/googletest' 2024-06-26T05:52:11.7672473Z http.https://github.com/.extraheader 2024-06-26T05:52:11.7698788Z Entering 'third_party/ideep' 2024-06-26T05:52:11.7727624Z http.https://github.com/.extraheader 2024-06-26T05:52:11.7756666Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T05:52:11.7779546Z http.https://github.com/.extraheader 2024-06-26T05:52:11.7810423Z Entering 'third_party/ittapi' 2024-06-26T05:52:11.7834026Z http.https://github.com/.extraheader 2024-06-26T05:52:11.7859386Z Entering 'third_party/kineto' 2024-06-26T05:52:11.7882285Z http.https://github.com/.extraheader 2024-06-26T05:52:11.7907854Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T05:52:11.7931017Z http.https://github.com/.extraheader 2024-06-26T05:52:11.7956708Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T05:52:11.7979102Z http.https://github.com/.extraheader 2024-06-26T05:52:11.8006216Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T05:52:11.8029088Z http.https://github.com/.extraheader 2024-06-26T05:52:11.8053546Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T05:52:11.8077165Z http.https://github.com/.extraheader 2024-06-26T05:52:11.8103645Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T05:52:11.8127166Z http.https://github.com/.extraheader 2024-06-26T05:52:11.8152750Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T05:52:11.8175485Z http.https://github.com/.extraheader 2024-06-26T05:52:11.8202558Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T05:52:11.8226367Z http.https://github.com/.extraheader 2024-06-26T05:52:11.8251779Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T05:52:11.8275567Z http.https://github.com/.extraheader 2024-06-26T05:52:11.8301879Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T05:52:11.8326571Z http.https://github.com/.extraheader 2024-06-26T05:52:11.8353631Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T05:52:11.8376848Z http.https://github.com/.extraheader 2024-06-26T05:52:11.8403226Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T05:52:11.8426194Z http.https://github.com/.extraheader 2024-06-26T05:52:11.8451087Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T05:52:11.8474744Z http.https://github.com/.extraheader 2024-06-26T05:52:11.8501060Z Entering 'third_party/mimalloc' 2024-06-26T05:52:11.8525238Z http.https://github.com/.extraheader 2024-06-26T05:52:11.8553846Z Entering 'third_party/nccl/nccl' 2024-06-26T05:52:11.8576958Z http.https://github.com/.extraheader 2024-06-26T05:52:11.8603080Z Entering 'third_party/nlohmann' 2024-06-26T05:52:11.8626720Z http.https://github.com/.extraheader 2024-06-26T05:52:11.8654227Z Entering 'third_party/onnx' 2024-06-26T05:52:11.8677275Z http.https://github.com/.extraheader 2024-06-26T05:52:11.8715728Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T05:52:11.8738958Z http.https://github.com/.extraheader 2024-06-26T05:52:11.8765209Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T05:52:11.8788732Z http.https://github.com/.extraheader 2024-06-26T05:52:11.8815617Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T05:52:11.8840107Z http.https://github.com/.extraheader 2024-06-26T05:52:11.8868050Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T05:52:11.8890793Z http.https://github.com/.extraheader 2024-06-26T05:52:11.8916919Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T05:52:11.8939094Z http.https://github.com/.extraheader 2024-06-26T05:52:11.8964280Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T05:52:11.8987659Z http.https://github.com/.extraheader 2024-06-26T05:52:11.9013719Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T05:52:11.9036632Z http.https://github.com/.extraheader 2024-06-26T05:52:11.9063839Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T05:52:11.9087827Z http.https://github.com/.extraheader 2024-06-26T05:52:11.9114404Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T05:52:11.9136010Z http.https://github.com/.extraheader 2024-06-26T05:52:11.9161597Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T05:52:11.9183410Z http.https://github.com/.extraheader 2024-06-26T05:52:11.9206520Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T05:52:11.9229242Z http.https://github.com/.extraheader 2024-06-26T05:52:11.9256582Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T05:52:11.9280184Z http.https://github.com/.extraheader 2024-06-26T05:52:11.9307711Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T05:52:11.9330530Z http.https://github.com/.extraheader 2024-06-26T05:52:11.9374444Z Entering 'third_party/pocketfft' 2024-06-26T05:52:11.9398655Z http.https://github.com/.extraheader 2024-06-26T05:52:11.9424143Z Entering 'third_party/protobuf' 2024-06-26T05:52:11.9448504Z http.https://github.com/.extraheader 2024-06-26T05:52:11.9477538Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T05:52:11.9500741Z http.https://github.com/.extraheader 2024-06-26T05:52:11.9525559Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T05:52:11.9548644Z http.https://github.com/.extraheader 2024-06-26T05:52:11.9577666Z Entering 'third_party/psimd' 2024-06-26T05:52:11.9600548Z http.https://github.com/.extraheader 2024-06-26T05:52:11.9627234Z Entering 'third_party/pthreadpool' 2024-06-26T05:52:11.9650332Z http.https://github.com/.extraheader 2024-06-26T05:52:11.9676218Z Entering 'third_party/pybind11' 2024-06-26T05:52:11.9699746Z http.https://github.com/.extraheader 2024-06-26T05:52:11.9725537Z Entering 'third_party/python-peachpy' 2024-06-26T05:52:11.9749554Z http.https://github.com/.extraheader 2024-06-26T05:52:11.9775646Z Entering 'third_party/sleef' 2024-06-26T05:52:11.9800094Z http.https://github.com/.extraheader 2024-06-26T05:52:11.9825411Z Entering 'third_party/tensorpipe' 2024-06-26T05:52:11.9849727Z http.https://github.com/.extraheader 2024-06-26T05:52:11.9877011Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T05:52:11.9899632Z http.https://github.com/.extraheader 2024-06-26T05:52:11.9925756Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T05:52:11.9948859Z http.https://github.com/.extraheader 2024-06-26T05:52:11.9973849Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T05:52:11.9995750Z http.https://github.com/.extraheader 2024-06-26T05:52:12.0021805Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T05:52:12.0044812Z http.https://github.com/.extraheader 2024-06-26T05:52:12.0068938Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T05:52:12.0092070Z http.https://github.com/.extraheader 2024-06-26T05:52:12.0198114Z A job completed hook has been configured by the self-hosted runner administrator 2024-06-26T05:52:12.0215902Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T05:52:12.0222576Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T05:52:12.0223114Z ##[endgroup] 2024-06-26T05:52:12.5667461Z Cleaning up orphan processes