2024-06-26T07:04:56.8027257Z Current runner version: '2.317.0' 2024-06-26T07:04:56.8033764Z Runner name: 'i-0c9d022330e7e8039' 2024-06-26T07:04:56.8034584Z Runner group name: 'Default' 2024-06-26T07:04:56.8035474Z Machine name: 'ip-10-0-49-201' 2024-06-26T07:04:56.8040443Z ##[group]GITHUB_TOKEN Permissions 2024-06-26T07:04:56.8042451Z Actions: read 2024-06-26T07:04:56.8043006Z Attestations: read 2024-06-26T07:04:56.8043558Z Checks: read 2024-06-26T07:04:56.8044067Z Contents: read 2024-06-26T07:04:56.8044601Z Deployments: read 2024-06-26T07:04:56.8045187Z Discussions: read 2024-06-26T07:04:56.8045720Z Issues: read 2024-06-26T07:04:56.8046236Z Metadata: read 2024-06-26T07:04:56.8046760Z Packages: read 2024-06-26T07:04:56.8047303Z Pages: read 2024-06-26T07:04:56.8047846Z PullRequests: read 2024-06-26T07:04:56.8048427Z RepositoryProjects: read 2024-06-26T07:04:56.8049077Z SecurityEvents: read 2024-06-26T07:04:56.8049622Z Statuses: read 2024-06-26T07:04:56.8050148Z ##[endgroup] 2024-06-26T07:04:56.8053526Z Secret source: Actions 2024-06-26T07:04:56.8054355Z Prepare workflow directory 2024-06-26T07:04:57.1849727Z Prepare all required actions 2024-06-26T07:04:57.2048876Z Getting action download info 2024-06-26T07:04:57.3897881Z Download action repository 'pytorch/test-infra@main' (SHA:43a2ce341cc31288e9a38b65ce600a7f43021bd5) 2024-06-26T07:04:57.6869063Z Download action repository 'pytorch/pytorch@main' (SHA:c422a9549d2b320b47a9391f6cbcf592b2876c88) 2024-06-26T07:05:00.4215426Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-06-26T07:05:00.5552816Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-06-26T07:05:00.8105821Z Getting action download info 2024-06-26T07:05:00.9028258Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-06-26T07:05:01.0615595Z Getting action download info 2024-06-26T07:05:01.2044677Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-06-26T07:05:01.3447182Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/129470 (b8c4c54d347aa776934c60784e35936878ef18dc) 2024-06-26T07:05:01.3449501Z ##[group] Inputs 2024-06-26T07:05:01.3449992Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:05:01.3456965Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:05:01.3464826Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:01.3466011Z sync-tag: 2024-06-26T07:05:01.3466835Z timeout-minutes: 240 2024-06-26T07:05:01.3467175Z use-gha: 2024-06-26T07:05:01.3467466Z dashboard-tag: 2024-06-26T07:05:01.3467789Z s3-bucket: gha-artifacts 2024-06-26T07:05:01.3468148Z aws-role-to-assume: 2024-06-26T07:05:01.3468488Z ##[endgroup] 2024-06-26T07:05:01.3469556Z Complete job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:05:01.4041139Z A job started hook has been configured by the self-hosted runner administrator 2024-06-26T07:05:01.4177594Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T07:05:01.4188083Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:01.4188617Z ##[endgroup] 2024-06-26T07:05:02.2458094Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-06-26T07:05:02.2458715Z with: 2024-06-26T07:05:02.2459500Z github-secret: *** 2024-06-26T07:05:02.2460524Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-06-26T07:05:02.2461588Z activate-with-label: false 2024-06-26T07:05:02.2462019Z label: with-ssh 2024-06-26T07:05:02.2462416Z remove-existing-keys: true 2024-06-26T07:05:02.2462848Z fail-silently: true 2024-06-26T07:05:02.2463233Z env: 2024-06-26T07:05:02.2463578Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:02.2463999Z ##[endgroup] 2024-06-26T07:05:02.3340029Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-06-26T07:05:02.3341149Z ciflow reference detected, attempting to extract PR number 2024-06-26T07:05:02.6459988Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-06-26T07:05:02.7361750Z No SSH keys found for user pytorch-bot[bot] 2024-06-26T07:05:02.7362654Z Grabbing public ssh keys from https://github.com/leslie-fang-intel.keys 2024-06-26T07:05:02.8102247Z No SSH keys found for user leslie-fang-intel 2024-06-26T07:05:02.8203533Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-06-26T07:05:02.8204086Z with: 2024-06-26T07:05:02.8204372Z submodules: recursive 2024-06-26T07:05:02.8204701Z fetch-depth: 0 2024-06-26T07:05:02.8204991Z env: 2024-06-26T07:05:02.8205261Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:02.8205595Z ##[endgroup] 2024-06-26T07:05:02.8407985Z ##[group]Run retry () { 2024-06-26T07:05:02.8408477Z retry () { 2024-06-26T07:05:02.8409048Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-06-26T07:05:02.8409636Z } 2024-06-26T07:05:02.8410023Z echo "${GITHUB_WORKSPACE}" 2024-06-26T07:05:02.8410491Z if [ -z "${NO_SUDO}" ]; then 2024-06-26T07:05:02.8411070Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:05:02.8411572Z else 2024-06-26T07:05:02.8411969Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:05:02.8412452Z fi 2024-06-26T07:05:02.8412891Z mkdir "${GITHUB_WORKSPACE}" 2024-06-26T07:05:02.8420935Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:02.8421491Z env: 2024-06-26T07:05:02.8421812Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:02.8422196Z NO_SUDO: 2024-06-26T07:05:02.8422521Z ##[endgroup] 2024-06-26T07:05:02.8475796Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:05:05.1110509Z ##[group]Run malfet/checkout@silent-checkout 2024-06-26T07:05:05.1110942Z with: 2024-06-26T07:05:05.1111485Z ref: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:05:05.1111937Z fetch-depth: 0 2024-06-26T07:05:05.1112250Z submodules: recursive 2024-06-26T07:05:05.1112595Z quiet-checkout: true 2024-06-26T07:05:05.1113106Z repository: pytorch/pytorch 2024-06-26T07:05:05.1113598Z token: *** 2024-06-26T07:05:05.1113888Z ssh-strict: true 2024-06-26T07:05:05.1114220Z persist-credentials: true 2024-06-26T07:05:05.1114570Z clean: true 2024-06-26T07:05:05.1114902Z sparse-checkout-cone-mode: true 2024-06-26T07:05:05.1115287Z lfs: false 2024-06-26T07:05:05.1115581Z set-safe-directory: true 2024-06-26T07:05:05.1116039Z env: 2024-06-26T07:05:05.1116312Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:05.1116656Z ##[endgroup] 2024-06-26T07:05:05.2051622Z Syncing repository: pytorch/pytorch 2024-06-26T07:05:05.2053195Z ##[group]Getting Git version info 2024-06-26T07:05:05.2053964Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:05:05.2054984Z [command]/usr/bin/git version 2024-06-26T07:05:05.2055483Z git version 2.40.1 2024-06-26T07:05:05.2077441Z ##[endgroup] 2024-06-26T07:05:05.2092233Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/830f55d9-ab54-4694-8e31-4bc7e30e8615' before making global git config changes 2024-06-26T07:05:05.2093501Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T07:05:05.2097986Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:05:05.2129352Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:05:05.2133435Z ##[group]Initializing the repository 2024-06-26T07:05:05.2136590Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:05:05.2157913Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-06-26T07:05:05.2158851Z hint: is subject to change. To configure the initial branch name to use in all 2024-06-26T07:05:05.2159655Z hint: of your new repositories, which will suppress this warning, call: 2024-06-26T07:05:05.2160437Z hint: 2024-06-26T07:05:05.2161131Z hint: git config --global init.defaultBranch 2024-06-26T07:05:05.2161641Z hint: 2024-06-26T07:05:05.2162158Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-06-26T07:05:05.2163029Z hint: 'development'. The just-created branch can be renamed via this command: 2024-06-26T07:05:05.2163963Z hint: 2024-06-26T07:05:05.2164343Z hint: git branch -m 2024-06-26T07:05:05.2165083Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-06-26T07:05:05.2167545Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-06-26T07:05:05.2187613Z ##[endgroup] 2024-06-26T07:05:05.2188208Z ##[group]Disabling automatic garbage collection 2024-06-26T07:05:05.2190409Z [command]/usr/bin/git config --local gc.auto 0 2024-06-26T07:05:05.2212250Z ##[endgroup] 2024-06-26T07:05:05.2212779Z ##[group]Setting up auth 2024-06-26T07:05:05.2218222Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T07:05:05.2239070Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T07:05:05.2442719Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T07:05:05.2464636Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T07:05:05.2669275Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:05:05.2708879Z ##[endgroup] 2024-06-26T07:05:05.2709445Z ##[group]Fetching the repository 2024-06-26T07:05:05.2716362Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-06-26T07:05:07.5505721Z remote: Enumerating objects: 987619 2024-06-26T07:05:07.5506501Z remote: Enumerating objects: 990220, done. 2024-06-26T07:05:07.5509974Z remote: Counting objects: 0% (1/2601) 2024-06-26T07:05:07.5511388Z remote: Counting objects: 1% (27/2601) 2024-06-26T07:05:07.5511954Z remote: Counting objects: 2% (53/2601) 2024-06-26T07:05:07.5512466Z remote: Counting objects: 3% (79/2601) 2024-06-26T07:05:07.5515438Z remote: Counting objects: 4% (105/2601) 2024-06-26T07:05:07.5516088Z remote: Counting objects: 5% (131/2601) 2024-06-26T07:05:07.5516727Z remote: Counting objects: 6% (157/2601) 2024-06-26T07:05:07.5517237Z remote: Counting objects: 7% (183/2601) 2024-06-26T07:05:07.5517791Z remote: Counting objects: 8% (209/2601) 2024-06-26T07:05:07.5518283Z remote: Counting objects: 9% (235/2601) 2024-06-26T07:05:07.5519057Z remote: Counting objects: 10% (261/2601) 2024-06-26T07:05:07.5520074Z remote: Counting objects: 11% (287/2601) 2024-06-26T07:05:07.5520793Z remote: Counting objects: 12% (313/2601) 2024-06-26T07:05:07.5521297Z remote: Counting objects: 13% (339/2601) 2024-06-26T07:05:07.5521807Z remote: Counting objects: 14% (365/2601) 2024-06-26T07:05:07.5522351Z remote: Counting objects: 15% (391/2601) 2024-06-26T07:05:07.5523041Z remote: Counting objects: 16% (417/2601) 2024-06-26T07:05:07.5523603Z remote: Counting objects: 17% (443/2601) 2024-06-26T07:05:07.5524177Z remote: Counting objects: 18% (469/2601) 2024-06-26T07:05:07.5524673Z remote: Counting objects: 19% (495/2601) 2024-06-26T07:05:07.5525217Z remote: Counting objects: 20% (521/2601) 2024-06-26T07:05:07.5526027Z remote: Counting objects: 21% (547/2601) 2024-06-26T07:05:07.5526633Z remote: Counting objects: 22% (573/2601) 2024-06-26T07:05:07.5527407Z remote: Counting objects: 23% (599/2601) 2024-06-26T07:05:07.5528086Z remote: Counting objects: 24% (625/2601) 2024-06-26T07:05:07.5528602Z remote: Counting objects: 25% (651/2601) 2024-06-26T07:05:07.5531692Z remote: Counting objects: 26% (677/2601) 2024-06-26T07:05:07.5532396Z remote: Counting objects: 27% (703/2601) 2024-06-26T07:05:07.5533368Z remote: Counting objects: 28% (729/2601) 2024-06-26T07:05:07.5533899Z remote: Counting objects: 29% (755/2601) 2024-06-26T07:05:07.5534557Z remote: Counting objects: 30% (781/2601) 2024-06-26T07:05:07.5535168Z remote: Counting objects: 31% (807/2601) 2024-06-26T07:05:07.5535805Z remote: Counting objects: 32% (833/2601) 2024-06-26T07:05:07.5536478Z remote: Counting objects: 33% (859/2601) 2024-06-26T07:05:07.5537134Z remote: Counting objects: 34% (885/2601) 2024-06-26T07:05:07.5537782Z remote: Counting objects: 35% (911/2601) 2024-06-26T07:05:07.5538459Z remote: Counting objects: 36% (937/2601) 2024-06-26T07:05:07.5539141Z remote: Counting objects: 37% (963/2601) 2024-06-26T07:05:07.5541354Z remote: Counting objects: 38% (989/2601) 2024-06-26T07:05:07.5542035Z remote: Counting objects: 39% (1015/2601) 2024-06-26T07:05:07.5542639Z remote: Counting objects: 40% (1041/2601) 2024-06-26T07:05:07.5543368Z remote: Counting objects: 41% (1067/2601) 2024-06-26T07:05:07.5543968Z remote: Counting objects: 42% (1093/2601) 2024-06-26T07:05:07.5544482Z remote: Counting objects: 43% (1119/2601) 2024-06-26T07:05:07.5545094Z remote: Counting objects: 44% (1145/2601) 2024-06-26T07:05:07.5546007Z remote: Counting objects: 45% (1171/2601) 2024-06-26T07:05:07.5546705Z remote: Counting objects: 46% (1197/2601) 2024-06-26T07:05:07.5547227Z remote: Counting objects: 47% (1223/2601) 2024-06-26T07:05:07.5547776Z remote: Counting objects: 48% (1249/2601) 2024-06-26T07:05:07.5548813Z remote: Counting objects: 49% (1275/2601) 2024-06-26T07:05:07.5549619Z remote: Counting objects: 50% (1301/2601) 2024-06-26T07:05:07.5550183Z remote: Counting objects: 51% (1327/2601) 2024-06-26T07:05:07.5550694Z remote: Counting objects: 52% (1353/2601) 2024-06-26T07:05:07.5551575Z remote: Counting objects: 53% (1379/2601) 2024-06-26T07:05:07.5552191Z remote: Counting objects: 54% (1405/2601) 2024-06-26T07:05:07.5552902Z remote: Counting objects: 55% (1431/2601) 2024-06-26T07:05:07.5553551Z remote: Counting objects: 56% (1457/2601) 2024-06-26T07:05:07.5554211Z remote: Counting objects: 57% (1483/2601) 2024-06-26T07:05:07.5554896Z remote: Counting objects: 58% (1509/2601) 2024-06-26T07:05:07.5555415Z remote: Counting objects: 59% (1535/2601) 2024-06-26T07:05:07.5556175Z remote: Counting objects: 60% (1561/2601) 2024-06-26T07:05:07.5556702Z remote: Counting objects: 61% (1587/2601) 2024-06-26T07:05:07.5557384Z remote: Counting objects: 62% (1613/2601) 2024-06-26T07:05:07.5557950Z remote: Counting objects: 63% (1639/2601) 2024-06-26T07:05:07.5558652Z remote: Counting objects: 64% (1665/2601) 2024-06-26T07:05:07.5559175Z remote: Counting objects: 65% (1691/2601) 2024-06-26T07:05:07.5559836Z remote: Counting objects: 66% (1717/2601) 2024-06-26T07:05:07.5560357Z remote: Counting objects: 67% (1743/2601) 2024-06-26T07:05:07.5561038Z remote: Counting objects: 68% (1769/2601) 2024-06-26T07:05:07.5561558Z remote: Counting objects: 69% (1795/2601) 2024-06-26T07:05:07.5562067Z remote: Counting objects: 70% (1821/2601) 2024-06-26T07:05:07.5562581Z remote: Counting objects: 71% (1847/2601) 2024-06-26T07:05:07.5563223Z remote: Counting objects: 72% (1873/2601) 2024-06-26T07:05:07.5563866Z remote: Counting objects: 73% (1899/2601) 2024-06-26T07:05:07.5564385Z remote: Counting objects: 74% (1925/2601) 2024-06-26T07:05:07.5565013Z remote: Counting objects: 75% (1951/2601) 2024-06-26T07:05:07.5565637Z remote: Counting objects: 76% (1977/2601) 2024-06-26T07:05:07.5566149Z remote: Counting objects: 77% (2003/2601) 2024-06-26T07:05:07.5566656Z remote: Counting objects: 78% (2029/2601) 2024-06-26T07:05:07.5567442Z remote: Counting objects: 79% (2055/2601) 2024-06-26T07:05:07.5568102Z remote: Counting objects: 80% (2081/2601) 2024-06-26T07:05:07.5568746Z remote: Counting objects: 81% (2107/2601) 2024-06-26T07:05:07.5569408Z remote: Counting objects: 82% (2133/2601) 2024-06-26T07:05:07.5570066Z remote: Counting objects: 83% (2159/2601) 2024-06-26T07:05:07.5570715Z remote: Counting objects: 84% (2185/2601) 2024-06-26T07:05:07.5571244Z remote: Counting objects: 85% (2211/2601) 2024-06-26T07:05:07.5571862Z remote: Counting objects: 86% (2237/2601) 2024-06-26T07:05:07.5572437Z remote: Counting objects: 87% (2263/2601) 2024-06-26T07:05:07.5572945Z remote: Counting objects: 88% (2289/2601) 2024-06-26T07:05:07.5573463Z remote: Counting objects: 89% (2315/2601) 2024-06-26T07:05:07.5573971Z remote: Counting objects: 90% (2341/2601) 2024-06-26T07:05:07.5574484Z remote: Counting objects: 91% (2367/2601) 2024-06-26T07:05:07.5575048Z remote: Counting objects: 92% (2393/2601) 2024-06-26T07:05:07.5575559Z remote: Counting objects: 93% (2419/2601) 2024-06-26T07:05:07.5576069Z remote: Counting objects: 94% (2445/2601) 2024-06-26T07:05:07.5576578Z remote: Counting objects: 95% (2471/2601) 2024-06-26T07:05:07.5577084Z remote: Counting objects: 96% (2497/2601) 2024-06-26T07:05:07.5577593Z remote: Counting objects: 97% (2523/2601) 2024-06-26T07:05:07.5578095Z remote: Counting objects: 98% (2549/2601) 2024-06-26T07:05:07.5578600Z remote: Counting objects: 99% (2575/2601) 2024-06-26T07:05:07.5579113Z remote: Counting objects: 100% (2601/2601) 2024-06-26T07:05:07.5579663Z remote: Counting objects: 100% (2601/2601), done. 2024-06-26T07:05:07.5708707Z remote: Compressing objects: 0% (1/1072) 2024-06-26T07:05:07.6145098Z remote: Compressing objects: 1% (11/1072) 2024-06-26T07:05:07.6703199Z remote: Compressing objects: 2% (22/1072) 2024-06-26T07:05:07.7206227Z remote: Compressing objects: 3% (33/1072) 2024-06-26T07:05:07.7500915Z remote: Compressing objects: 4% (43/1072) 2024-06-26T07:05:07.7776680Z remote: Compressing objects: 5% (54/1072) 2024-06-26T07:05:07.7992084Z remote: Compressing objects: 6% (65/1072) 2024-06-26T07:05:07.8363950Z remote: Compressing objects: 7% (76/1072) 2024-06-26T07:05:07.9243018Z remote: Compressing objects: 8% (86/1072) 2024-06-26T07:05:08.0040536Z remote: Compressing objects: 9% (97/1072) 2024-06-26T07:05:08.0270866Z remote: Compressing objects: 10% (108/1072) 2024-06-26T07:05:08.0456209Z remote: Compressing objects: 11% (118/1072) 2024-06-26T07:05:08.0559138Z remote: Compressing objects: 12% (129/1072) 2024-06-26T07:05:08.0642558Z remote: Compressing objects: 13% (140/1072) 2024-06-26T07:05:08.0686894Z remote: Compressing objects: 14% (151/1072) 2024-06-26T07:05:08.0687473Z remote: Compressing objects: 15% (161/1072) 2024-06-26T07:05:08.0755857Z remote: Compressing objects: 16% (172/1072) 2024-06-26T07:05:08.0756543Z remote: Compressing objects: 17% (183/1072) 2024-06-26T07:05:08.0757091Z remote: Compressing objects: 18% (193/1072) 2024-06-26T07:05:08.0774200Z remote: Compressing objects: 19% (204/1072) 2024-06-26T07:05:08.0804197Z remote: Compressing objects: 20% (215/1072) 2024-06-26T07:05:08.0816210Z remote: Compressing objects: 21% (226/1072) 2024-06-26T07:05:08.0826889Z remote: Compressing objects: 22% (236/1072) 2024-06-26T07:05:08.0851039Z remote: Compressing objects: 23% (247/1072) 2024-06-26T07:05:08.0859332Z remote: Compressing objects: 24% (258/1072) 2024-06-26T07:05:08.0872035Z remote: Compressing objects: 25% (268/1072) 2024-06-26T07:05:08.0882940Z remote: Compressing objects: 26% (279/1072) 2024-06-26T07:05:08.0884930Z remote: Compressing objects: 27% (290/1072) 2024-06-26T07:05:08.0887081Z remote: Compressing objects: 28% (301/1072) 2024-06-26T07:05:08.0899847Z remote: Compressing objects: 29% (311/1072) 2024-06-26T07:05:08.0908242Z remote: Compressing objects: 30% (322/1072) 2024-06-26T07:05:08.0913703Z remote: Compressing objects: 31% (333/1072) 2024-06-26T07:05:08.0917582Z remote: Compressing objects: 32% (344/1072) 2024-06-26T07:05:08.0935062Z remote: Compressing objects: 33% (354/1072) 2024-06-26T07:05:08.0951572Z remote: Compressing objects: 34% (365/1072) 2024-06-26T07:05:08.0960151Z remote: Compressing objects: 35% (376/1072) 2024-06-26T07:05:08.0973345Z remote: Compressing objects: 36% (386/1072) 2024-06-26T07:05:08.0990124Z remote: Compressing objects: 37% (397/1072) 2024-06-26T07:05:08.1010583Z remote: Compressing objects: 38% (408/1072) 2024-06-26T07:05:08.1023048Z remote: Compressing objects: 39% (419/1072) 2024-06-26T07:05:08.1042450Z remote: Compressing objects: 40% (429/1072) 2024-06-26T07:05:08.1057532Z remote: Compressing objects: 41% (440/1072) 2024-06-26T07:05:08.1070395Z remote: Compressing objects: 42% (451/1072) 2024-06-26T07:05:08.1091081Z remote: Compressing objects: 43% (461/1072) 2024-06-26T07:05:08.1092689Z remote: Compressing objects: 44% (472/1072) 2024-06-26T07:05:08.1093699Z remote: Compressing objects: 45% (483/1072) 2024-06-26T07:05:08.1104537Z remote: Compressing objects: 46% (494/1072) 2024-06-26T07:05:08.1120652Z remote: Compressing objects: 47% (504/1072) 2024-06-26T07:05:08.1130454Z remote: Compressing objects: 48% (515/1072) 2024-06-26T07:05:08.1135441Z remote: Compressing objects: 49% (526/1072) 2024-06-26T07:05:08.1143806Z remote: Compressing objects: 50% (536/1072) 2024-06-26T07:05:08.1154309Z remote: Compressing objects: 51% (547/1072) 2024-06-26T07:05:08.1168190Z remote: Compressing objects: 52% (558/1072) 2024-06-26T07:05:08.1187111Z remote: Compressing objects: 53% (569/1072) 2024-06-26T07:05:08.1198613Z remote: Compressing objects: 54% (579/1072) 2024-06-26T07:05:08.1216644Z remote: Compressing objects: 55% (590/1072) 2024-06-26T07:05:08.1229665Z remote: Compressing objects: 56% (601/1072) 2024-06-26T07:05:08.1233466Z remote: Compressing objects: 57% (612/1072) 2024-06-26T07:05:08.1248060Z remote: Compressing objects: 58% (622/1072) 2024-06-26T07:05:08.1252369Z remote: Compressing objects: 59% (633/1072) 2024-06-26T07:05:08.1260309Z remote: Compressing objects: 60% (644/1072) 2024-06-26T07:05:08.1276871Z remote: Compressing objects: 61% (654/1072) 2024-06-26T07:05:08.1283540Z remote: Compressing objects: 62% (665/1072) 2024-06-26T07:05:08.1297476Z remote: Compressing objects: 63% (676/1072) 2024-06-26T07:05:08.1307339Z remote: Compressing objects: 64% (687/1072) 2024-06-26T07:05:08.1312216Z remote: Compressing objects: 65% (697/1072) 2024-06-26T07:05:08.1314038Z remote: Compressing objects: 66% (708/1072) 2024-06-26T07:05:08.1318798Z remote: Compressing objects: 67% (719/1072) 2024-06-26T07:05:08.1322757Z remote: Compressing objects: 68% (729/1072) 2024-06-26T07:05:08.1324794Z remote: Compressing objects: 69% (740/1072) 2024-06-26T07:05:08.1329312Z remote: Compressing objects: 70% (751/1072) 2024-06-26T07:05:08.1332278Z remote: Compressing objects: 71% (762/1072) 2024-06-26T07:05:08.1333200Z remote: Compressing objects: 72% (772/1072) 2024-06-26T07:05:08.1334689Z remote: Compressing objects: 73% (783/1072) 2024-06-26T07:05:08.1336036Z remote: Compressing objects: 74% (794/1072) 2024-06-26T07:05:08.1336804Z remote: Compressing objects: 75% (804/1072) 2024-06-26T07:05:08.1337343Z remote: Compressing objects: 76% (815/1072) 2024-06-26T07:05:08.1337997Z remote: Compressing objects: 77% (826/1072) 2024-06-26T07:05:08.1338539Z remote: Compressing objects: 78% (837/1072) 2024-06-26T07:05:08.1346790Z remote: Compressing objects: 79% (847/1072) 2024-06-26T07:05:08.1347328Z remote: Compressing objects: 80% (858/1072) 2024-06-26T07:05:08.1348153Z remote: Compressing objects: 81% (869/1072) 2024-06-26T07:05:08.1349165Z remote: Compressing objects: 82% (880/1072) 2024-06-26T07:05:08.1350142Z remote: Compressing objects: 83% (890/1072) 2024-06-26T07:05:08.1354681Z remote: Compressing objects: 84% (901/1072) 2024-06-26T07:05:08.1359013Z remote: Compressing objects: 85% (912/1072) 2024-06-26T07:05:08.1359565Z remote: Compressing objects: 86% (922/1072) 2024-06-26T07:05:08.1361494Z remote: Compressing objects: 87% (933/1072) 2024-06-26T07:05:08.1363926Z remote: Compressing objects: 88% (944/1072) 2024-06-26T07:05:08.1364468Z remote: Compressing objects: 89% (955/1072) 2024-06-26T07:05:08.1366299Z remote: Compressing objects: 90% (965/1072) 2024-06-26T07:05:08.1366842Z remote: Compressing objects: 91% (976/1072) 2024-06-26T07:05:08.1368884Z remote: Compressing objects: 92% (987/1072) 2024-06-26T07:05:08.1369425Z remote: Compressing objects: 93% (997/1072) 2024-06-26T07:05:08.1373313Z remote: Compressing objects: 94% (1008/1072) 2024-06-26T07:05:08.1373875Z remote: Compressing objects: 95% (1019/1072) 2024-06-26T07:05:08.1374427Z remote: Compressing objects: 96% (1030/1072) 2024-06-26T07:05:08.1376486Z remote: Compressing objects: 97% (1040/1072) 2024-06-26T07:05:08.1377039Z remote: Compressing objects: 98% (1051/1072) 2024-06-26T07:05:08.1379618Z remote: Compressing objects: 99% (1062/1072) 2024-06-26T07:05:08.1380316Z remote: Compressing objects: 100% (1072/1072) 2024-06-26T07:05:08.1380898Z remote: Compressing objects: 100% (1072/1072), done. 2024-06-26T07:05:28.0785728Z remote: Total 990220 (delta 2066), reused 1868 (delta 1528), pack-reused 987619 2024-06-26T07:05:47.8237264Z [command]/usr/bin/git rev-parse --verify --quiet b8c4c54d347aa776934c60784e35936878ef18dc^{object} 2024-06-26T07:05:47.8255208Z b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:05:47.8258813Z ##[endgroup] 2024-06-26T07:05:47.8259366Z ##[group]Determining the checkout info 2024-06-26T07:05:47.8261373Z ##[endgroup] 2024-06-26T07:05:47.8261911Z ##[group]Checking out the ref 2024-06-26T07:05:47.8263676Z [command]/usr/bin/git checkout --quiet --force b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:05:49.0449567Z ##[endgroup] 2024-06-26T07:05:49.0450167Z ##[group]Setting up auth for fetching submodules 2024-06-26T07:05:49.0455168Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:05:49.0493788Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-06-26T07:05:49.0514014Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-06-26T07:05:49.0539098Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-06-26T07:05:49.0557100Z ##[endgroup] 2024-06-26T07:05:49.0557627Z ##[group]Fetching submodules 2024-06-26T07:05:49.0560393Z [command]/usr/bin/git submodule sync --recursive 2024-06-26T07:05:49.0775786Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-06-26T07:05:49.0990195Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-06-26T07:05:49.0991952Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-06-26T07:05:49.0993397Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-06-26T07:05:49.0994985Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-06-26T07:05:49.0997116Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-06-26T07:05:49.0998900Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-06-26T07:05:49.1001287Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-06-26T07:05:49.1003462Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-06-26T07:05:49.1005707Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-06-26T07:05:49.1008204Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-06-26T07:05:49.1010507Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-06-26T07:05:49.1012975Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-06-26T07:05:49.1015543Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-06-26T07:05:49.1018231Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-06-26T07:05:49.1020739Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-06-26T07:05:49.1023488Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-06-26T07:05:49.1026471Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:05:49.1029159Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-06-26T07:05:49.1032554Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-06-26T07:05:49.1035481Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-06-26T07:05:49.1038723Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-06-26T07:05:49.1041941Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-06-26T07:05:49.1045262Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-06-26T07:05:49.1048470Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-06-26T07:05:49.1051857Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-06-26T07:05:49.1055225Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-06-26T07:05:49.1058988Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-06-26T07:05:49.1062338Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-06-26T07:05:49.1065952Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-06-26T07:05:49.1069676Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-06-26T07:05:49.1074457Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-06-26T07:05:49.1078333Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-06-26T07:05:49.1082139Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-06-26T07:05:49.1085916Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-06-26T07:05:49.1090017Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-06-26T07:05:49.1110294Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-06-26T07:05:49.3581779Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-06-26T07:05:49.5220165Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-06-26T07:05:49.6830199Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-06-26T07:05:51.1659199Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-06-26T07:05:53.2164485Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-06-26T07:06:02.8780024Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-06-26T07:06:03.2826838Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-06-26T07:06:03.7748663Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-06-26T07:06:04.3568403Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-06-26T07:06:05.5453337Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-06-26T07:06:07.2263174Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-06-26T07:06:13.0179316Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-06-26T07:06:14.3791052Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-06-26T07:06:16.0204033Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-06-26T07:06:17.1815986Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-06-26T07:06:17.4177375Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-06-26T07:06:17.7831294Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-06-26T07:06:18.1470089Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-06-26T07:06:19.1749113Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-06-26T07:06:19.5175109Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-06-26T07:06:19.7534947Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-06-26T07:06:21.2422189Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-06-26T07:06:21.9445488Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-06-26T07:06:22.6805891Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-06-26T07:06:28.1842385Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-06-26T07:06:30.5053622Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-06-26T07:06:36.0607328Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-06-26T07:06:36.2484863Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-06-26T07:06:44.1918619Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-06-26T07:06:44.3524547Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-06-26T07:06:44.5289389Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-06-26T07:06:45.5507237Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-06-26T07:06:45.8133751Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-06-26T07:06:46.3893731Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-06-26T07:06:46.7920449Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-06-26T07:06:46.8002457Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-06-26T07:06:46.8064145Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-06-26T07:06:46.8245289Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-06-26T07:06:46.8551831Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-06-26T07:06:47.6310041Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-06-26T07:06:47.6482853Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-06-26T07:06:47.6841232Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-06-26T07:06:47.7693414Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-06-26T07:06:47.7965035Z Submodule path 'third_party/cudnn_frontend': checked out 'aa3abd4bc689d6412979c7f55f9cd132848c9c6a' 2024-06-26T07:06:48.2283336Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-06-26T07:06:48.4422266Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-06-26T07:06:48.5060495Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-06-26T07:06:48.5071625Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:06:48.5073468Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:06:48.5075287Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:06:48.5077419Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:06:48.5079876Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:06:48.5096547Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-06-26T07:06:49.3554625Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-06-26T07:06:49.8972937Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-06-26T07:06:51.5536077Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-06-26T07:06:52.4906632Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-06-26T07:06:52.7741237Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-06-26T07:06:52.8580527Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-06-26T07:06:53.2005639Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-06-26T07:06:53.2548386Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-06-26T07:06:53.2634146Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-06-26T07:06:53.3570533Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-06-26T07:06:53.3849540Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-06-26T07:06:53.3918195Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-06-26T07:06:53.4221655Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-06-26T07:06:53.4417637Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-06-26T07:06:53.4786802Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:06:53.4885160Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-06-26T07:06:53.4896307Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-06-26T07:06:53.4916625Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-06-26T07:07:05.7518274Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-06-26T07:07:05.7655627Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-06-26T07:07:05.8436109Z Submodule path 'third_party/kineto': checked out '8681ff11e1fa54da39023076c5c43eddd87b7a8a' 2024-06-26T07:07:05.8446711Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:07:05.8448403Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:07:05.8450389Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:07:05.8465535Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-06-26T07:07:06.3827769Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-06-26T07:07:07.5877972Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-06-26T07:07:08.5968621Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-06-26T07:07:08.5978181Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:07:08.5979937Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:07:08.5981927Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:07:08.5983745Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:07:08.5985620Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:07:08.5987867Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:07:08.5989722Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:07:08.5992116Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:07:08.6011404Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-06-26T07:07:09.6241569Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-06-26T07:07:09.9716274Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-06-26T07:07:11.1817881Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-06-26T07:07:11.4457727Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-06-26T07:07:11.9607450Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-06-26T07:07:12.9222731Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-06-26T07:07:18.2709517Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-06-26T07:07:18.6257415Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-06-26T07:07:18.6393203Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-06-26T07:07:18.6685347Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-06-26T07:07:18.6781631Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-06-26T07:07:18.6791082Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:07:18.6810369Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-06-26T07:07:18.9526684Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-06-26T07:07:18.9663740Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-06-26T07:07:19.0002642Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-06-26T07:07:19.0807230Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-06-26T07:07:19.0927842Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-06-26T07:07:19.1223425Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-06-26T07:07:19.1718255Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-06-26T07:07:19.2024104Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-06-26T07:07:19.2221432Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-06-26T07:07:19.3083251Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-06-26T07:07:19.5871778Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-06-26T07:07:19.5894831Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-06-26T07:07:19.5896449Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-06-26T07:07:19.5912516Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-06-26T07:07:19.9654726Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-06-26T07:07:21.1183736Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-06-26T07:07:21.1458570Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-06-26T07:07:21.1992942Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-06-26T07:07:21.2004608Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:07:21.2011748Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:07:21.2013371Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:07:21.2015430Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:07:21.2017573Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:07:21.2019858Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:07:21.2021995Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:07:21.2023791Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:07:21.2035932Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-06-26T07:07:21.6747089Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-06-26T07:07:22.6069084Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-06-26T07:07:22.9379685Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-06-26T07:07:29.2668295Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-06-26T07:07:29.5372028Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-06-26T07:07:29.7229017Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-06-26T07:07:30.0715668Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-06-26T07:07:37.5979167Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-06-26T07:07:37.6315964Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-06-26T07:07:37.6441687Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-06-26T07:07:37.7284919Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-06-26T07:07:37.7378636Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-06-26T07:07:37.7484949Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-06-26T07:07:37.7597642Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-06-26T07:07:37.7608421Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:07:37.7610450Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:07:37.7626051Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-06-26T07:07:39.6422435Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-06-26T07:07:40.8184320Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-06-26T07:07:40.8554085Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:07:41.2173602Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-06-26T07:07:41.2256230Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-06-26T07:07:41.4438516Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-06-26T07:07:41.4452862Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:07:41.4454785Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-06-26T07:07:41.4473286Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-06-26T07:07:41.8495982Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-06-26T07:07:42.8172655Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-06-26T07:07:42.8795191Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-06-26T07:07:42.8856203Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-06-26T07:07:42.8947766Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-06-26T07:07:42.9231439Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-06-26T07:07:42.9457382Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-06-26T07:07:42.9793861Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-06-26T07:07:42.9996587Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-06-26T07:07:43.0009562Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:07:43.0011360Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:07:43.0015282Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:07:43.0017777Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:07:43.0034484Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-06-26T07:07:44.0061906Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-06-26T07:07:44.2348901Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-06-26T07:07:45.2755649Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-06-26T07:07:46.3632040Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-06-26T07:07:46.3743921Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-06-26T07:07:46.4307324Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-06-26T07:07:46.4540905Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-06-26T07:07:46.4551692Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:07:46.4568960Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-06-26T07:07:46.6755918Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-06-26T07:07:46.6779965Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-06-26T07:07:46.6989706Z Entering 'android/libs/fbjni' 2024-06-26T07:07:46.7018354Z Entering 'third_party/FP16' 2024-06-26T07:07:46.7046081Z Entering 'third_party/FXdiv' 2024-06-26T07:07:46.7072748Z Entering 'third_party/NNPACK' 2024-06-26T07:07:46.7102576Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:07:46.7131993Z Entering 'third_party/XNNPACK' 2024-06-26T07:07:46.7169163Z Entering 'third_party/benchmark' 2024-06-26T07:07:46.7196701Z Entering 'third_party/cpp-httplib' 2024-06-26T07:07:46.7224235Z Entering 'third_party/cpuinfo' 2024-06-26T07:07:46.7252289Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:07:46.7278977Z Entering 'third_party/cutlass' 2024-06-26T07:07:46.7312115Z Entering 'third_party/eigen' 2024-06-26T07:07:46.7340790Z Entering 'third_party/fbgemm' 2024-06-26T07:07:46.7370341Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:07:46.7398430Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:07:46.7426041Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:07:46.7459745Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:07:46.7487518Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:07:46.7516278Z Entering 'third_party/flatbuffers' 2024-06-26T07:07:46.7545399Z Entering 'third_party/fmt' 2024-06-26T07:07:46.7572914Z Entering 'third_party/foxi' 2024-06-26T07:07:46.7601899Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:07:46.7628201Z Entering 'third_party/gloo' 2024-06-26T07:07:46.7657490Z Entering 'third_party/googletest' 2024-06-26T07:07:46.7684593Z Entering 'third_party/ideep' 2024-06-26T07:07:46.7711633Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:07:46.7744844Z Entering 'third_party/ittapi' 2024-06-26T07:07:46.7772171Z Entering 'third_party/kineto' 2024-06-26T07:07:46.7802784Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:07:46.7829360Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:07:46.7859665Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:07:46.7885362Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:07:46.7911384Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:07:46.7939063Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:07:46.7967522Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:07:46.7993701Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:07:46.8022458Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:07:46.8050445Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:07:46.8079012Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:07:46.8105966Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:07:46.8135957Z Entering 'third_party/mimalloc' 2024-06-26T07:07:46.8163299Z Entering 'third_party/nccl/nccl' 2024-06-26T07:07:46.8189785Z Entering 'third_party/nlohmann' 2024-06-26T07:07:46.8220025Z Entering 'third_party/onnx' 2024-06-26T07:07:46.8259589Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:07:46.8291147Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:07:46.8321789Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:07:46.8350101Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:07:46.8378830Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:07:46.8405869Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:07:46.8432307Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:07:46.8459686Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:07:46.8486960Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:07:46.8512806Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:07:46.8540369Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:07:46.8569155Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:07:46.8598567Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:07:46.8642031Z Entering 'third_party/pocketfft' 2024-06-26T07:07:46.8671696Z Entering 'third_party/protobuf' 2024-06-26T07:07:46.8702839Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:07:46.8730338Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:07:46.8758469Z Entering 'third_party/psimd' 2024-06-26T07:07:46.8787238Z Entering 'third_party/pthreadpool' 2024-06-26T07:07:46.8816152Z Entering 'third_party/pybind11' 2024-06-26T07:07:46.8843999Z Entering 'third_party/python-peachpy' 2024-06-26T07:07:46.8872665Z Entering 'third_party/sleef' 2024-06-26T07:07:46.8901976Z Entering 'third_party/tensorpipe' 2024-06-26T07:07:46.8930878Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:07:46.8958734Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:07:46.8984904Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:07:46.9012917Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:07:46.9040938Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:07:46.9082406Z ##[endgroup] 2024-06-26T07:07:46.9084602Z ##[group]Persisting credentials for submodules 2024-06-26T07:07:46.9086197Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-06-26T07:07:46.9295457Z Entering 'android/libs/fbjni' 2024-06-26T07:07:46.9330316Z Entering 'third_party/FP16' 2024-06-26T07:07:46.9363758Z Entering 'third_party/FXdiv' 2024-06-26T07:07:46.9398688Z Entering 'third_party/NNPACK' 2024-06-26T07:07:46.9434492Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:07:46.9465867Z Entering 'third_party/XNNPACK' 2024-06-26T07:07:46.9511451Z Entering 'third_party/benchmark' 2024-06-26T07:07:46.9546220Z Entering 'third_party/cpp-httplib' 2024-06-26T07:07:46.9580753Z Entering 'third_party/cpuinfo' 2024-06-26T07:07:46.9614093Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:07:46.9646688Z Entering 'third_party/cutlass' 2024-06-26T07:07:46.9686543Z Entering 'third_party/eigen' 2024-06-26T07:07:46.9723762Z Entering 'third_party/fbgemm' 2024-06-26T07:07:46.9760182Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:07:46.9794939Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:07:46.9829346Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:07:46.9867886Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:07:46.9902366Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:07:46.9938628Z Entering 'third_party/flatbuffers' 2024-06-26T07:07:46.9977978Z Entering 'third_party/fmt' 2024-06-26T07:07:47.0011602Z Entering 'third_party/foxi' 2024-06-26T07:07:47.0043821Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:07:47.0079992Z Entering 'third_party/gloo' 2024-06-26T07:07:47.0113393Z Entering 'third_party/googletest' 2024-06-26T07:07:47.0147656Z Entering 'third_party/ideep' 2024-06-26T07:07:47.0182278Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:07:47.0221554Z Entering 'third_party/ittapi' 2024-06-26T07:07:47.0257500Z Entering 'third_party/kineto' 2024-06-26T07:07:47.0294391Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:07:47.0327945Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:07:47.0363506Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:07:47.0398368Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:07:47.0432035Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:07:47.0466375Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:07:47.0501292Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:07:47.0535873Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:07:47.0571109Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:07:47.0606392Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:07:47.0642772Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:07:47.0677453Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:07:47.0713497Z Entering 'third_party/mimalloc' 2024-06-26T07:07:47.0749113Z Entering 'third_party/nccl/nccl' 2024-06-26T07:07:47.0785760Z Entering 'third_party/nlohmann' 2024-06-26T07:07:47.0821854Z Entering 'third_party/onnx' 2024-06-26T07:07:47.0868543Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:07:47.0904376Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:07:47.0941243Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:07:47.0978228Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:07:47.1013785Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:07:47.1047316Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:07:47.1081252Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:07:47.1117217Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:07:47.1151498Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:07:47.1185991Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:07:47.1220178Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:07:47.1256402Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:07:47.1291611Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:07:47.1339084Z Entering 'third_party/pocketfft' 2024-06-26T07:07:47.1375808Z Entering 'third_party/protobuf' 2024-06-26T07:07:47.1412544Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:07:47.1447767Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:07:47.1482447Z Entering 'third_party/psimd' 2024-06-26T07:07:47.1519567Z Entering 'third_party/pthreadpool' 2024-06-26T07:07:47.1553328Z Entering 'third_party/pybind11' 2024-06-26T07:07:47.1592613Z Entering 'third_party/python-peachpy' 2024-06-26T07:07:47.1628715Z Entering 'third_party/sleef' 2024-06-26T07:07:47.1664560Z Entering 'third_party/tensorpipe' 2024-06-26T07:07:47.1701135Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:07:47.1735248Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:07:47.1768316Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:07:47.1803506Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:07:47.1836639Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:07:47.1885776Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-06-26T07:07:47.2091558Z Entering 'android/libs/fbjni' 2024-06-26T07:07:47.2123063Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-06-26T07:07:47.2136284Z Entering 'third_party/FP16' 2024-06-26T07:07:47.2168065Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-06-26T07:07:47.2180857Z Entering 'third_party/FXdiv' 2024-06-26T07:07:47.2213851Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-06-26T07:07:47.2226028Z Entering 'third_party/NNPACK' 2024-06-26T07:07:47.2258672Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-06-26T07:07:47.2271228Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:07:47.2304477Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-06-26T07:07:47.2318322Z Entering 'third_party/XNNPACK' 2024-06-26T07:07:47.2351011Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-06-26T07:07:47.2375013Z Entering 'third_party/benchmark' 2024-06-26T07:07:47.2406698Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:07:47.2420190Z Entering 'third_party/cpp-httplib' 2024-06-26T07:07:47.2453372Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-06-26T07:07:47.2465114Z Entering 'third_party/cpuinfo' 2024-06-26T07:07:47.2495766Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:07:47.2509527Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:07:47.2541540Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-06-26T07:07:47.2555672Z Entering 'third_party/cutlass' 2024-06-26T07:07:47.2587389Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:07:47.2606501Z Entering 'third_party/eigen' 2024-06-26T07:07:47.2636507Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-06-26T07:07:47.2652435Z Entering 'third_party/fbgemm' 2024-06-26T07:07:47.2685646Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-06-26T07:07:47.2698439Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:07:47.2730706Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-06-26T07:07:47.2742752Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:07:47.2775518Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:07:47.2788688Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:07:47.2822771Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:07:47.2841080Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:07:47.2872634Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:07:47.2885430Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:07:47.2917159Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-06-26T07:07:47.2931771Z Entering 'third_party/flatbuffers' 2024-06-26T07:07:47.2965477Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-06-26T07:07:47.2980104Z Entering 'third_party/fmt' 2024-06-26T07:07:47.3013352Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:07:47.3025981Z Entering 'third_party/foxi' 2024-06-26T07:07:47.3056732Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-06-26T07:07:47.3067395Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:07:47.3101145Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-06-26T07:07:47.3114766Z Entering 'third_party/gloo' 2024-06-26T07:07:47.3146091Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-06-26T07:07:47.3160376Z Entering 'third_party/googletest' 2024-06-26T07:07:47.3191403Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:07:47.3206523Z Entering 'third_party/ideep' 2024-06-26T07:07:47.3238210Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-06-26T07:07:47.3251077Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:07:47.3282392Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-06-26T07:07:47.3300560Z Entering 'third_party/ittapi' 2024-06-26T07:07:47.3331984Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-06-26T07:07:47.3346001Z Entering 'third_party/kineto' 2024-06-26T07:07:47.3380645Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-06-26T07:07:47.3394806Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:07:47.3426828Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-06-26T07:07:47.3439045Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:07:47.3471685Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-06-26T07:07:47.3485925Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:07:47.3518375Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-06-26T07:07:47.3529932Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:07:47.3562072Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:07:47.3575477Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:07:47.3607111Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-06-26T07:07:47.3618785Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:07:47.3651920Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-06-26T07:07:47.3666545Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:07:47.3697782Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-06-26T07:07:47.3708631Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:07:47.3741398Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:07:47.3754283Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:07:47.3786045Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-06-26T07:07:47.3799715Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:07:47.3833222Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-06-26T07:07:47.3847631Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:07:47.3878376Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-06-26T07:07:47.3890212Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:07:47.3921306Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-06-26T07:07:47.3934486Z Entering 'third_party/mimalloc' 2024-06-26T07:07:47.3966332Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-06-26T07:07:47.3980205Z Entering 'third_party/nccl/nccl' 2024-06-26T07:07:47.4013519Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-06-26T07:07:47.4025917Z Entering 'third_party/nlohmann' 2024-06-26T07:07:47.4058809Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-06-26T07:07:47.4073600Z Entering 'third_party/onnx' 2024-06-26T07:07:47.4107126Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-06-26T07:07:47.4132053Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:07:47.4163935Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:07:47.4177443Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:07:47.4209017Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:07:47.4224302Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:07:47.4258140Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-06-26T07:07:47.4272103Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:07:47.4304405Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:07:47.4318048Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:07:47.4349155Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:07:47.4362448Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:07:47.4393665Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-06-26T07:07:47.4406156Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:07:47.4438849Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-06-26T07:07:47.4451642Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:07:47.4482647Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-06-26T07:07:47.4493349Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:07:47.4525479Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-06-26T07:07:47.4538028Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:07:47.4571463Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-06-26T07:07:47.4582528Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:07:47.4614106Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-06-26T07:07:47.4628245Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:07:47.4659247Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-06-26T07:07:47.4672361Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:07:47.4703309Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-06-26T07:07:47.4731234Z Entering 'third_party/pocketfft' 2024-06-26T07:07:47.4762569Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-06-26T07:07:47.4776413Z Entering 'third_party/protobuf' 2024-06-26T07:07:47.4808370Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-06-26T07:07:47.4824888Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:07:47.4857231Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:07:47.4868946Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:07:47.4899916Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:07:47.4914750Z Entering 'third_party/psimd' 2024-06-26T07:07:47.4947086Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-06-26T07:07:47.4959846Z Entering 'third_party/pthreadpool' 2024-06-26T07:07:47.4990889Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-06-26T07:07:47.5004244Z Entering 'third_party/pybind11' 2024-06-26T07:07:47.5036124Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:07:47.5049167Z Entering 'third_party/python-peachpy' 2024-06-26T07:07:47.5079713Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-06-26T07:07:47.5091973Z Entering 'third_party/sleef' 2024-06-26T07:07:47.5122809Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-06-26T07:07:47.5135142Z Entering 'third_party/tensorpipe' 2024-06-26T07:07:47.5166755Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-06-26T07:07:47.5181119Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:07:47.5214317Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:07:47.5227262Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:07:47.5260899Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-06-26T07:07:47.5274198Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:07:47.5305495Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-06-26T07:07:47.5319475Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:07:47.5349967Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:07:47.5362090Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:07:47.5391967Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-06-26T07:07:47.6128533Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-06-26T07:07:47.6338855Z Entering 'android/libs/fbjni' 2024-06-26T07:07:47.6364966Z Entering 'third_party/FP16' 2024-06-26T07:07:47.6394312Z Entering 'third_party/FXdiv' 2024-06-26T07:07:47.6420347Z Entering 'third_party/NNPACK' 2024-06-26T07:07:47.6449007Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:07:47.6477724Z Entering 'third_party/XNNPACK' 2024-06-26T07:07:47.6516043Z Entering 'third_party/benchmark' 2024-06-26T07:07:47.6544114Z Entering 'third_party/cpp-httplib' 2024-06-26T07:07:47.6570734Z Entering 'third_party/cpuinfo' 2024-06-26T07:07:47.6599643Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:07:47.6626974Z Entering 'third_party/cutlass' 2024-06-26T07:07:47.6661777Z Entering 'third_party/eigen' 2024-06-26T07:07:47.6691295Z Entering 'third_party/fbgemm' 2024-06-26T07:07:47.6720240Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:07:47.6745341Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:07:47.6773929Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:07:47.6807756Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:07:47.6834381Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:07:47.6861459Z Entering 'third_party/flatbuffers' 2024-06-26T07:07:47.6891373Z Entering 'third_party/fmt' 2024-06-26T07:07:47.6919351Z Entering 'third_party/foxi' 2024-06-26T07:07:47.6945858Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:07:47.6975019Z Entering 'third_party/gloo' 2024-06-26T07:07:47.7000425Z Entering 'third_party/googletest' 2024-06-26T07:07:47.7028238Z Entering 'third_party/ideep' 2024-06-26T07:07:47.7054522Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:07:47.7086348Z Entering 'third_party/ittapi' 2024-06-26T07:07:47.7115261Z Entering 'third_party/kineto' 2024-06-26T07:07:47.7142139Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:07:47.7171578Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:07:47.7199638Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:07:47.7226271Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:07:47.7254867Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:07:47.7282400Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:07:47.7313629Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:07:47.7342280Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:07:47.7371302Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:07:47.7400712Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:07:47.7429126Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:07:47.7457744Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:07:47.7485070Z Entering 'third_party/mimalloc' 2024-06-26T07:07:47.7510943Z Entering 'third_party/nccl/nccl' 2024-06-26T07:07:47.7540379Z Entering 'third_party/nlohmann' 2024-06-26T07:07:47.7567140Z Entering 'third_party/onnx' 2024-06-26T07:07:47.7604555Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:07:47.7633888Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:07:47.7663684Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:07:47.7692478Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:07:47.7719876Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:07:47.7744858Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:07:47.7772819Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:07:47.7801982Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:07:47.7826556Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:07:47.7854905Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:07:47.7880665Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:07:47.7909180Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:07:47.7939799Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:07:47.7981980Z Entering 'third_party/pocketfft' 2024-06-26T07:07:47.8012138Z Entering 'third_party/protobuf' 2024-06-26T07:07:47.8043178Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:07:47.8070609Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:07:47.8100554Z Entering 'third_party/psimd' 2024-06-26T07:07:47.8126047Z Entering 'third_party/pthreadpool' 2024-06-26T07:07:47.8150575Z Entering 'third_party/pybind11' 2024-06-26T07:07:47.8180210Z Entering 'third_party/python-peachpy' 2024-06-26T07:07:47.8206399Z Entering 'third_party/sleef' 2024-06-26T07:07:47.8236982Z Entering 'third_party/tensorpipe' 2024-06-26T07:07:47.8265184Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:07:47.8292997Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:07:47.8319015Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:07:47.8344993Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:07:47.8373497Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:07:47.8415413Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-06-26T07:07:47.8625227Z Entering 'android/libs/fbjni' 2024-06-26T07:07:47.8652766Z Entering 'third_party/FP16' 2024-06-26T07:07:47.8683206Z Entering 'third_party/FXdiv' 2024-06-26T07:07:47.8712743Z Entering 'third_party/NNPACK' 2024-06-26T07:07:47.8741374Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:07:47.8768069Z Entering 'third_party/XNNPACK' 2024-06-26T07:07:47.8807552Z Entering 'third_party/benchmark' 2024-06-26T07:07:47.8837561Z Entering 'third_party/cpp-httplib' 2024-06-26T07:07:47.8865258Z Entering 'third_party/cpuinfo' 2024-06-26T07:07:47.8896423Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:07:47.8924200Z Entering 'third_party/cutlass' 2024-06-26T07:07:47.8959047Z Entering 'third_party/eigen' 2024-06-26T07:07:47.8990058Z Entering 'third_party/fbgemm' 2024-06-26T07:07:47.9021724Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:07:47.9048304Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:07:47.9073393Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:07:47.9106167Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:07:47.9134342Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:07:47.9164390Z Entering 'third_party/flatbuffers' 2024-06-26T07:07:47.9194226Z Entering 'third_party/fmt' 2024-06-26T07:07:47.9222106Z Entering 'third_party/foxi' 2024-06-26T07:07:47.9251463Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:07:47.9280018Z Entering 'third_party/gloo' 2024-06-26T07:07:47.9308593Z Entering 'third_party/googletest' 2024-06-26T07:07:47.9336490Z Entering 'third_party/ideep' 2024-06-26T07:07:47.9364369Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:07:47.9397202Z Entering 'third_party/ittapi' 2024-06-26T07:07:47.9425620Z Entering 'third_party/kineto' 2024-06-26T07:07:47.9455497Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:07:47.9483862Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:07:47.9509980Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:07:47.9538169Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:07:47.9566764Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:07:47.9592810Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:07:47.9619340Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:07:47.9646465Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:07:47.9675169Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:07:47.9702682Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:07:47.9732958Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:07:47.9759150Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:07:47.9786391Z Entering 'third_party/mimalloc' 2024-06-26T07:07:47.9816929Z Entering 'third_party/nccl/nccl' 2024-06-26T07:07:47.9843771Z Entering 'third_party/nlohmann' 2024-06-26T07:07:47.9874023Z Entering 'third_party/onnx' 2024-06-26T07:07:47.9913062Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:07:47.9942466Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:07:47.9971183Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:07:48.0002915Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:07:48.0031832Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:07:48.0059825Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:07:48.0086134Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:07:48.0114690Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:07:48.0141100Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:07:48.0167462Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:07:48.0193761Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:07:48.0221424Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:07:48.0249476Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:07:48.0291501Z Entering 'third_party/pocketfft' 2024-06-26T07:07:48.0322321Z Entering 'third_party/protobuf' 2024-06-26T07:07:48.0350892Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:07:48.0379545Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:07:48.0409190Z Entering 'third_party/psimd' 2024-06-26T07:07:48.0437746Z Entering 'third_party/pthreadpool' 2024-06-26T07:07:48.0466619Z Entering 'third_party/pybind11' 2024-06-26T07:07:48.0496885Z Entering 'third_party/python-peachpy' 2024-06-26T07:07:48.0525317Z Entering 'third_party/sleef' 2024-06-26T07:07:48.0553646Z Entering 'third_party/tensorpipe' 2024-06-26T07:07:48.0581736Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:07:48.0609921Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:07:48.0635245Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:07:48.0666999Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:07:48.0694270Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:07:48.0730753Z ##[endgroup] 2024-06-26T07:07:48.0764166Z [command]/usr/bin/git log -1 --format='%H' 2024-06-26T07:07:48.0786106Z 'b8c4c54d347aa776934c60784e35936878ef18dc' 2024-06-26T07:07:48.0961581Z Prepare all required actions 2024-06-26T07:07:48.0962042Z Getting action download info 2024-06-26T07:07:48.2520576Z ##[group]Run ./.github/actions/setup-linux 2024-06-26T07:07:48.2521003Z env: 2024-06-26T07:07:48.2521284Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:48.2521632Z ##[endgroup] 2024-06-26T07:07:48.2579280Z ##[group]Run set -euo pipefail 2024-06-26T07:07:48.2579711Z set -euo pipefail 2024-06-26T07:07:48.2580108Z function get_ec2_metadata() { 2024-06-26T07:07:48.2580646Z  # Pulled from instance metadata endpoint for EC2 2024-06-26T07:07:48.2581583Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-06-26T07:07:48.2582359Z  category=$1 2024-06-26T07:07:48.2582887Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-06-26T07:07:48.2583528Z  runner_name_str=i-0c9d022330e7e8039 2024-06-26T07:07:48.2584028Z  if [[ -f /.inarc ]]; then 2024-06-26T07:07:48.2584522Z  echo "ARC Runner, no info on ec2 metadata" 2024-06-26T07:07:48.2585097Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-06-26T07:07:48.2585802Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-06-26T07:07:48.2586404Z  else 2024-06-26T07:07:48.2586891Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-06-26T07:07:48.2587473Z  fi 2024-06-26T07:07:48.2587761Z } 2024-06-26T07:07:48.2588123Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-06-26T07:07:48.2588702Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-06-26T07:07:48.2589356Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-06-26T07:07:48.2589927Z echo "system info $(uname -a)" 2024-06-26T07:07:48.2598136Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:48.2598623Z env: 2024-06-26T07:07:48.2598908Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:48.2599255Z ##[endgroup] 2024-06-26T07:07:48.2671271Z ami-id: ami-0ce0c36d7a00b20e2 2024-06-26T07:07:48.2717887Z instance-id: i-0c9d022330e7e8039 2024-06-26T07:07:48.2758898Z instance-type: g5.4xlarge 2024-06-26T07:07:48.2765460Z system info Linux ip-10-0-49-201.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-06-26T07:07:48.2785196Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:07:48.2786111Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:07:48.2794305Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:48.2794800Z env: 2024-06-26T07:07:48.2795074Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:48.2795419Z ##[endgroup] 2024-06-26T07:07:48.2855711Z ##[group]Run if systemctl is-active --quiet docker; then 2024-06-26T07:07:48.2856322Z if systemctl is-active --quiet docker; then 2024-06-26T07:07:48.2856867Z  echo "Docker daemon is running..."; 2024-06-26T07:07:48.2857438Z else 2024-06-26T07:07:48.2857931Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-06-26T07:07:48.2858516Z fi 2024-06-26T07:07:48.2865397Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:48.2865885Z env: 2024-06-26T07:07:48.2866159Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:48.2866504Z ##[endgroup] 2024-06-26T07:07:48.2903411Z Docker daemon is running... 2024-06-26T07:07:48.2946654Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:07:48.2947215Z with: 2024-06-26T07:07:48.2947486Z shell: bash 2024-06-26T07:07:48.2947788Z timeout_minutes: 5 2024-06-26T07:07:48.2948115Z max_attempts: 3 2024-06-26T07:07:48.2948432Z retry_wait_seconds: 30 2024-06-26T07:07:48.2949952Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-06-26T07:07:48.2951552Z polling_interval_seconds: 1 2024-06-26T07:07:48.2951931Z warning_on_retry: true 2024-06-26T07:07:48.2952286Z continue_on_error: false 2024-06-26T07:07:48.2952624Z env: 2024-06-26T07:07:48.2952890Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:48.2953251Z AWS_RETRY_MODE: standard 2024-06-26T07:07:48.2953604Z AWS_MAX_ATTEMPTS: 5 2024-06-26T07:07:48.2953941Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:07:48.2954315Z ##[endgroup] 2024-06-26T07:07:49.1683363Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:07:49.1684400Z Configure a credential helper to remove this warning. See 2024-06-26T07:07:49.1685228Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:07:49.1685748Z 2024-06-26T07:07:49.1685894Z Login Succeeded 2024-06-26T07:07:49.3450939Z Command completed after 1 attempt(s). 2024-06-26T07:07:49.3491690Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:07:49.3492430Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:07:49.3493088Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:07:49.3500895Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:49.3501407Z env: 2024-06-26T07:07:49.3501727Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:49.3502074Z ##[endgroup] 2024-06-26T07:07:49.3562406Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:07:49.3563186Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:07:49.3563776Z # shellcheck disable=SC2046 2024-06-26T07:07:49.3564241Z docker stop $(docker ps -q) || true 2024-06-26T07:07:49.3564739Z # Prune all of the docker images 2024-06-26T07:07:49.3565190Z docker system prune -af 2024-06-26T07:07:49.3572289Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:49.3572807Z env: 2024-06-26T07:07:49.3573078Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:49.3573425Z ##[endgroup] 2024-06-26T07:07:49.3822895Z "docker stop" requires at least 1 argument. 2024-06-26T07:07:49.3823554Z See 'docker stop --help'. 2024-06-26T07:07:49.3823794Z 2024-06-26T07:07:49.3824019Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-06-26T07:07:49.3824402Z 2024-06-26T07:07:49.3825352Z Stop one or more running containers 2024-06-26T07:07:49.3976083Z Total reclaimed space: 0B 2024-06-26T07:07:49.4015664Z ##[group]Run set +e 2024-06-26T07:07:49.4016180Z set +e 2024-06-26T07:07:49.4016605Z set -x 2024-06-26T07:07:49.4017026Z  2024-06-26T07:07:49.4017499Z PT_DOMAIN=download.pytorch.org 2024-06-26T07:07:49.4018718Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-06-26T07:07:49.4020444Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-06-26T07:07:49.4021811Z # one is returned at random 2024-06-26T07:07:49.4022605Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-06-26T07:07:49.4023394Z  2024-06-26T07:07:49.4023873Z if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:07:49.4024808Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-06-26T07:07:49.4025965Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-06-26T07:07:49.4026796Z  2024-06-26T07:07:49.4027274Z  if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:07:49.4028114Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-06-26T07:07:49.4028881Z  exit 1 2024-06-26T07:07:49.4029326Z  fi 2024-06-26T07:07:49.4029736Z fi 2024-06-26T07:07:49.4030377Z  2024-06-26T07:07:49.4030923Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-06-26T07:07:49.4032049Z  # Clean up any old records first 2024-06-26T07:07:49.4032817Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-06-26T07:07:49.4033507Z fi 2024-06-26T07:07:49.4033911Z  2024-06-26T07:07:49.4034530Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-06-26T07:07:49.4035336Z cat /etc/hosts 2024-06-26T07:07:49.4045504Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:49.4046241Z env: 2024-06-26T07:07:49.4046643Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:49.4047151Z ##[endgroup] 2024-06-26T07:07:49.4097297Z + PT_DOMAIN=download.pytorch.org 2024-06-26T07:07:49.4101166Z ++ dig -4 +short download.pytorch.org 2024-06-26T07:07:49.4101609Z ++ tail -n1 2024-06-26T07:07:49.4197781Z + RESOLVED_IP=18.67.76.42 2024-06-26T07:07:49.4198204Z + '[' -z 18.67.76.42 ']' 2024-06-26T07:07:49.4198649Z + grep -r download.pytorch.org /etc/hosts 2024-06-26T07:07:49.4203982Z 18.160.10.36 download.pytorch.org 2024-06-26T07:07:49.4204620Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-06-26T07:07:49.4301437Z + echo '18.67.76.42 download.pytorch.org' 2024-06-26T07:07:49.4302020Z + sudo tee -a /etc/hosts 2024-06-26T07:07:49.4704263Z 18.67.76.42 download.pytorch.org 2024-06-26T07:07:49.4717669Z + cat /etc/hosts 2024-06-26T07:07:49.4722511Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-06-26T07:07:49.4735694Z ::1 localhost6 localhost6.localdomain6 2024-06-26T07:07:49.4736247Z 18.67.76.42 download.pytorch.org 2024-06-26T07:07:49.4950703Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-06-26T07:07:49.4951599Z with: 2024-06-26T07:07:49.4952795Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:49.4954060Z docker-build-dir: .ci/docker 2024-06-26T07:07:49.4954452Z working-directory: . 2024-06-26T07:07:49.4954934Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:49.4955467Z force-push: false 2024-06-26T07:07:49.4955839Z env: 2024-06-26T07:07:49.4956102Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:49.4956438Z ##[endgroup] 2024-06-26T07:07:49.4975464Z ##[group]Run set -ex 2024-06-26T07:07:49.4975840Z set -ex 2024-06-26T07:07:49.4976133Z  2024-06-26T07:07:49.4976685Z # If the docker build directory or the build script doesn't exist, the action will 2024-06-26T07:07:49.4977683Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-06-26T07:07:49.4978497Z # job could then download the pre-built image as usual 2024-06-26T07:07:49.4979242Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-06-26T07:07:49.4979912Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:49.4980547Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:49.4981221Z  2024-06-26T07:07:49.4981724Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-06-26T07:07:49.4982345Z  exit 0 2024-06-26T07:07:49.4982640Z else 2024-06-26T07:07:49.4983005Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:49.4983451Z fi 2024-06-26T07:07:49.4983724Z  2024-06-26T07:07:49.4984196Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-06-26T07:07:49.4985056Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-06-26T07:07:49.4985833Z  # use it as it is, but first let's extract the tag 2024-06-26T07:07:49.4986533Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-06-26T07:07:49.4987256Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:49.4987966Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:49.4988538Z else 2024-06-26T07:07:49.4988974Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-06-26T07:07:49.4989636Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:49.4990526Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:49.4991414Z fi 2024-06-26T07:07:49.4999137Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:49.4999630Z env: 2024-06-26T07:07:49.4999906Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:49.5000260Z REPO_NAME: pytorch 2024-06-26T07:07:49.5001398Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:49.5002649Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:07:49.5003152Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:49.5003681Z ##[endgroup] 2024-06-26T07:07:49.5023768Z + [[ ! -d .ci/docker ]] 2024-06-26T07:07:49.5024207Z + [[ ! -f .ci/docker/build.sh ]] 2024-06-26T07:07:49.5024591Z + echo skip=false 2024-06-26T07:07:49.5026560Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-06-26T07:07:49.5030242Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:49.5032316Z ++ awk -F '[:,]' '{print $2}' 2024-06-26T07:07:49.5036146Z + DOCKER_TAG=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:49.5036821Z + echo docker-tag=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:49.5038486Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:49.5065000Z ##[group]Run set +e 2024-06-26T07:07:49.5065378Z set +e 2024-06-26T07:07:49.5065690Z set -x 2024-06-26T07:07:49.5065983Z  2024-06-26T07:07:49.5066275Z login() { 2024-06-26T07:07:49.5066956Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:07:49.5067681Z } 2024-06-26T07:07:49.5067957Z  2024-06-26T07:07:49.5068242Z retry () { 2024-06-26T07:07:49.5068644Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:07:49.5069106Z } 2024-06-26T07:07:49.5069388Z  2024-06-26T07:07:49.5069720Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:07:49.5070150Z  2024-06-26T07:07:49.5070624Z # Check if image already exists, if it does then skip building it 2024-06-26T07:07:49.5071749Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-06-26T07:07:49.5072275Z  exit 0 2024-06-26T07:07:49.5072588Z fi 2024-06-26T07:07:49.5072868Z  2024-06-26T07:07:49.5073381Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-06-26T07:07:49.5074220Z # be empty. The default action would be to continue rebuild the image 2024-06-26T07:07:49.5074977Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-06-26T07:07:49.5075651Z  # if we're on the base branch then use the parent commit 2024-06-26T07:07:49.5076338Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-06-26T07:07:49.5076782Z else 2024-06-26T07:07:49.5077262Z  # otherwise we're on a PR, so use the most recent base commit 2024-06-26T07:07:49.5077950Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-06-26T07:07:49.5078479Z fi 2024-06-26T07:07:49.5078766Z  2024-06-26T07:07:49.5079100Z if [[ -z "${MERGE_BASE}" ]]; then 2024-06-26T07:07:49.5079605Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:49.5080070Z  2024-06-26T07:07:49.5080733Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-06-26T07:07:49.5081506Z  exit 0 2024-06-26T07:07:49.5081806Z fi 2024-06-26T07:07:49.5082091Z  2024-06-26T07:07:49.5082542Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-06-26T07:07:49.5083523Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-06-26T07:07:49.5084322Z  exit 1 2024-06-26T07:07:49.5084631Z fi 2024-06-26T07:07:49.5084917Z  2024-06-26T07:07:49.5085417Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-06-26T07:07:49.5086375Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-06-26T07:07:49.5087254Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-06-26T07:07:49.5088230Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-06-26T07:07:49.5089333Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-06-26T07:07:49.5090089Z fi 2024-06-26T07:07:49.5090379Z  2024-06-26T07:07:49.5090749Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:49.5098438Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:49.5098946Z env: 2024-06-26T07:07:49.5099229Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:49.5099607Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:07:49.5100087Z BASE_REVISION: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:07:49.5101384Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:49.5102634Z DOCKER_TAG: 91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:49.5103259Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:49.5103801Z ##[endgroup] 2024-06-26T07:07:49.5123190Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:49.5124097Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:49.5124931Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:07:49.5126038Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:49.9317454Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:07:49.9318436Z Configure a credential helper to remove this warning. See 2024-06-26T07:07:49.9319589Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:07:49.9320110Z 2024-06-26T07:07:49.9320243Z Login Succeeded 2024-06-26T07:07:49.9328735Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:50.1380740Z { 2024-06-26T07:07:50.1381137Z "schemaVersion": 2, 2024-06-26T07:07:50.1381892Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-06-26T07:07:50.1382684Z "config": { 2024-06-26T07:07:50.1383295Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-06-26T07:07:50.1384043Z "size": 48681, 2024-06-26T07:07:50.1384841Z "digest": "sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd" 2024-06-26T07:07:50.1385832Z }, 2024-06-26T07:07:50.1386166Z "layers": [ 2024-06-26T07:07:50.1386533Z { 2024-06-26T07:07:50.1387175Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1387976Z "size": 28580681, 2024-06-26T07:07:50.1388759Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-06-26T07:07:50.1389561Z }, 2024-06-26T07:07:50.1389896Z { 2024-06-26T07:07:50.1390505Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1391649Z "size": 7943451, 2024-06-26T07:07:50.1392475Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-06-26T07:07:50.1393375Z }, 2024-06-26T07:07:50.1393701Z { 2024-06-26T07:07:50.1394318Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1395091Z "size": 55728572, 2024-06-26T07:07:50.1395954Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-06-26T07:07:50.1396772Z }, 2024-06-26T07:07:50.1397101Z { 2024-06-26T07:07:50.1397578Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1398217Z "size": 186, 2024-06-26T07:07:50.1398967Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-06-26T07:07:50.1399789Z }, 2024-06-26T07:07:50.1400112Z { 2024-06-26T07:07:50.1400678Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1401387Z "size": 6886, 2024-06-26T07:07:50.1402132Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-06-26T07:07:50.1402756Z }, 2024-06-26T07:07:50.1403182Z { 2024-06-26T07:07:50.1403652Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1404480Z + exit 0 2024-06-26T07:07:50.1404872Z "size": 1329001756, 2024-06-26T07:07:50.1405682Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-06-26T07:07:50.1406629Z }, 2024-06-26T07:07:50.1406966Z { 2024-06-26T07:07:50.1407552Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1408296Z "size": 62501, 2024-06-26T07:07:50.1409022Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-06-26T07:07:50.1409779Z }, 2024-06-26T07:07:50.1410030Z { 2024-06-26T07:07:50.1410453Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1411006Z "size": 1684, 2024-06-26T07:07:50.1411537Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-06-26T07:07:50.1412136Z }, 2024-06-26T07:07:50.1412389Z { 2024-06-26T07:07:50.1412913Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1413669Z "size": 1523, 2024-06-26T07:07:50.1414251Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-06-26T07:07:50.1415027Z }, 2024-06-26T07:07:50.1415371Z { 2024-06-26T07:07:50.1415980Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1416727Z "size": 2528295403, 2024-06-26T07:07:50.1417620Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-06-26T07:07:50.1418353Z }, 2024-06-26T07:07:50.1418603Z { 2024-06-26T07:07:50.1419045Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1419590Z "size": 86016, 2024-06-26T07:07:50.1420333Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-06-26T07:07:50.1421158Z }, 2024-06-26T07:07:50.1421485Z { 2024-06-26T07:07:50.1422051Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1422695Z "size": 1823, 2024-06-26T07:07:50.1423399Z "digest": "sha256:309b872da805fcdf74137a887464876fa0615dc8b24199307119eb65fc2bfb49" 2024-06-26T07:07:50.1424186Z }, 2024-06-26T07:07:50.1424504Z { 2024-06-26T07:07:50.1425076Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1425817Z "size": 246737542, 2024-06-26T07:07:50.1426513Z "digest": "sha256:e82932cb002c06a782077b8e5965f8733eb8563406a23cab260d9d8c4a994f7d" 2024-06-26T07:07:50.1427129Z }, 2024-06-26T07:07:50.1427377Z { 2024-06-26T07:07:50.1427808Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1428350Z "size": 545, 2024-06-26T07:07:50.1428893Z "digest": "sha256:ca9dd73e3e9378225ebdcfd12199ef44649e34fe55003cfbc8afe2e8a4527c5b" 2024-06-26T07:07:50.1429518Z }, 2024-06-26T07:07:50.1429769Z { 2024-06-26T07:07:50.1430196Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1430750Z "size": 1283, 2024-06-26T07:07:50.1431491Z "digest": "sha256:16d76a79d5d6616e228537785ec83a869f68a402520bbc6e826201621ca6cb98" 2024-06-26T07:07:50.1432101Z }, 2024-06-26T07:07:50.1432345Z { 2024-06-26T07:07:50.1432779Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1433334Z "size": 485, 2024-06-26T07:07:50.1433872Z "digest": "sha256:4bdf41c9ecbe1651e0986ccbea4663a0556af27b7145a679cfa45c35dc96b891" 2024-06-26T07:07:50.1434496Z }, 2024-06-26T07:07:50.1434747Z { 2024-06-26T07:07:50.1435179Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1435829Z "size": 91710937, 2024-06-26T07:07:50.1436411Z "digest": "sha256:f4863801e30cdbcd1d4e6cb1cbbcde7dbcaf6ae317ca80429295741ce60ac31d" 2024-06-26T07:07:50.1437035Z }, 2024-06-26T07:07:50.1437288Z { 2024-06-26T07:07:50.1437715Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1438388Z "size": 3233, 2024-06-26T07:07:50.1438944Z "digest": "sha256:d37dd9ed8948a0d02e1198bbedadbca36477a0980dd553196385aab0c025581d" 2024-06-26T07:07:50.1439553Z }, 2024-06-26T07:07:50.1439804Z { 2024-06-26T07:07:50.1440233Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1440785Z "size": 1905, 2024-06-26T07:07:50.1441330Z "digest": "sha256:cf2f34f225dbed17c1a9a8b9ea4ad030aba797774a716dc0a82fe212324aa751" 2024-06-26T07:07:50.1441958Z }, 2024-06-26T07:07:50.1442204Z { 2024-06-26T07:07:50.1442645Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1443189Z "size": 702, 2024-06-26T07:07:50.1443733Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:07:50.1444351Z }, 2024-06-26T07:07:50.1444600Z { 2024-06-26T07:07:50.1445023Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1445576Z "size": 2784789002, 2024-06-26T07:07:50.1446146Z "digest": "sha256:3d914950f8addfc4d657a2365c76e1143eb80b710cd3f6e9ae05640217fc6347" 2024-06-26T07:07:50.1446758Z }, 2024-06-26T07:07:50.1447008Z { 2024-06-26T07:07:50.1447441Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1447990Z "size": 381, 2024-06-26T07:07:50.1448522Z "digest": "sha256:96d0e2032184c57e62a652aaffcf0241f0d0c7a23e894ca0ad192c4a64a36583" 2024-06-26T07:07:50.1449129Z }, 2024-06-26T07:07:50.1449448Z { 2024-06-26T07:07:50.1449878Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1450419Z "size": 11879, 2024-06-26T07:07:50.1450956Z "digest": "sha256:06e12ce59790cb8b091e5d619c2a62a28329008ee184a8b9a562e3db12471f30" 2024-06-26T07:07:50.1451569Z }, 2024-06-26T07:07:50.1451845Z { 2024-06-26T07:07:50.1452281Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1452838Z "size": 803, 2024-06-26T07:07:50.1453386Z "digest": "sha256:ed712f63cde7070bdec2a6179c4c35e752c6991ddbc69fe9f2724a9903250ceb" 2024-06-26T07:07:50.1453996Z }, 2024-06-26T07:07:50.1454241Z { 2024-06-26T07:07:50.1454673Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1455227Z "size": 106, 2024-06-26T07:07:50.1455746Z "digest": "sha256:a97a71ef8a4e66234dfc2098834f8395ac5630240e60164d1a251fdb0115ba85" 2024-06-26T07:07:50.1456347Z }, 2024-06-26T07:07:50.1456591Z { 2024-06-26T07:07:50.1457025Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1457565Z "size": 503, 2024-06-26T07:07:50.1458095Z "digest": "sha256:25cd0c482ad612f936b35b4f220c580e16be982226aac8801a6a849a6bfdc221" 2024-06-26T07:07:50.1458712Z }, 2024-06-26T07:07:50.1458955Z { 2024-06-26T07:07:50.1459383Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1459935Z "size": 121477177, 2024-06-26T07:07:50.1460498Z "digest": "sha256:9bf94bc456584365fda21d0ed83b7cc690b08a71727b71c4c1e667e15028a386" 2024-06-26T07:07:50.1461098Z }, 2024-06-26T07:07:50.1461349Z { 2024-06-26T07:07:50.1461777Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1462326Z "size": 109, 2024-06-26T07:07:50.1462851Z "digest": "sha256:de45a14c9b227e4b0c9d45d42223b50a71c03522ddd9e2ca0d9517835206c823" 2024-06-26T07:07:50.1463459Z }, 2024-06-26T07:07:50.1463712Z { 2024-06-26T07:07:50.1464133Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1464690Z "size": 491, 2024-06-26T07:07:50.1465219Z "digest": "sha256:e99241e77209b0b35a0fb59a61c0821fc161190178af7e4af45e6eeaa043375e" 2024-06-26T07:07:50.1465880Z }, 2024-06-26T07:07:50.1466128Z { 2024-06-26T07:07:50.1466558Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1467102Z "size": 291, 2024-06-26T07:07:50.1467649Z "digest": "sha256:35e8f4e2a42f9df0c969dedd2199cf71719cf5ecfa676dfa014bbdeb4fdb295f" 2024-06-26T07:07:50.1468338Z }, 2024-06-26T07:07:50.1468592Z { 2024-06-26T07:07:50.1469012Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1469563Z "size": 101, 2024-06-26T07:07:50.1470103Z "digest": "sha256:a823ffa4c541394b52953391dc0a6867e26c37fe4a71fa022a2cc3f07f0ba70d" 2024-06-26T07:07:50.1470708Z }, 2024-06-26T07:07:50.1470952Z { 2024-06-26T07:07:50.1471608Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1472334Z "size": 1473, 2024-06-26T07:07:50.1472922Z "digest": "sha256:44ba272eae2064603605a6238b3f35cc6578859d660068e702a047a9cd8ab2fd" 2024-06-26T07:07:50.1473521Z }, 2024-06-26T07:07:50.1473766Z { 2024-06-26T07:07:50.1474192Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1474733Z "size": 446710781, 2024-06-26T07:07:50.1475289Z "digest": "sha256:da02d278e5240c285bc855ce4e42cd6121d5b903787f5c14f03e575ad1268650" 2024-06-26T07:07:50.1475998Z }, 2024-06-26T07:07:50.1476252Z { 2024-06-26T07:07:50.1476678Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1477226Z "size": 161, 2024-06-26T07:07:50.1477750Z "digest": "sha256:73a0ce681c295135456e190578733273276b38b4dd5babbb39298f384d43b1c6" 2024-06-26T07:07:50.1478344Z }, 2024-06-26T07:07:50.1478592Z { 2024-06-26T07:07:50.1479027Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1479577Z "size": 564, 2024-06-26T07:07:50.1480235Z "digest": "sha256:49737bd5ae709dbf7956e21fc8b05ba009e9e6c9b130b0126a73bd3d3f5efc78" 2024-06-26T07:07:50.1480852Z }, 2024-06-26T07:07:50.1481101Z { 2024-06-26T07:07:50.1481527Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1482118Z "size": 35874600, 2024-06-26T07:07:50.1482675Z "digest": "sha256:9cd4c745f8e4f9d5c9bb8050923575afad21f5b7852975a4a1981bc9eff282c5" 2024-06-26T07:07:50.1483287Z }, 2024-06-26T07:07:50.1483541Z { 2024-06-26T07:07:50.1483980Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1484527Z "size": 104, 2024-06-26T07:07:50.1485051Z "digest": "sha256:b2950a0365345d4c17a48627c905fd8930b3b3154c727f0526eb9ca547102bb7" 2024-06-26T07:07:50.1485642Z }, 2024-06-26T07:07:50.1485893Z { 2024-06-26T07:07:50.1486328Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1486874Z "size": 425, 2024-06-26T07:07:50.1487390Z "digest": "sha256:25e19d81d9fc081962ff4cbc09e098f4035419822712c19a410f5282300dfb9e" 2024-06-26T07:07:50.1487994Z }, 2024-06-26T07:07:50.1488245Z { 2024-06-26T07:07:50.1488674Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1489214Z "size": 20262084, 2024-06-26T07:07:50.1489775Z "digest": "sha256:60156b40fcbec2c25f0170b0a82fa8cd9303e0daae8bb1811fd0fc6562e88e8e" 2024-06-26T07:07:50.1490390Z }, 2024-06-26T07:07:50.1490631Z { 2024-06-26T07:07:50.1491062Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1491616Z "size": 439, 2024-06-26T07:07:50.1492140Z "digest": "sha256:7a868239227fd19b30a19927bbc8f058d247403fe5b343c55f4b4b8987918911" 2024-06-26T07:07:50.1492731Z }, 2024-06-26T07:07:50.1492981Z { 2024-06-26T07:07:50.1493412Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1493960Z "size": 702, 2024-06-26T07:07:50.1494503Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:07:50.1495126Z }, 2024-06-26T07:07:50.1495376Z { 2024-06-26T07:07:50.1495809Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1496348Z "size": 143, 2024-06-26T07:07:50.1496875Z "digest": "sha256:83911d7c7ecf11b5932bb7449975c52e7f737a5b867691935e1ccb85ceb25b52" 2024-06-26T07:07:50.1497479Z }, 2024-06-26T07:07:50.1497721Z { 2024-06-26T07:07:50.1498154Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1498810Z "size": 135, 2024-06-26T07:07:50.1499366Z "digest": "sha256:b3c5d66c679f58c71febdd502889bd8cd3d36e5001754cdf1e30eabe21135cce" 2024-06-26T07:07:50.1499977Z }, 2024-06-26T07:07:50.1500228Z { 2024-06-26T07:07:50.1500660Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1501217Z "size": 4911495247, 2024-06-26T07:07:50.1501791Z "digest": "sha256:84b2c7a899820f6acdb22148c5e8346d8b26c33269fd01842c9573515251e1e1" 2024-06-26T07:07:50.1502424Z }, 2024-06-26T07:07:50.1502670Z { 2024-06-26T07:07:50.1503094Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1503645Z "size": 190, 2024-06-26T07:07:50.1504191Z "digest": "sha256:c1068d03a811d8dead5da9e82fdc8869205fc4ee9cb92d4f7a4acfbb5ba83950" 2024-06-26T07:07:50.1504820Z }, 2024-06-26T07:07:50.1505062Z { 2024-06-26T07:07:50.1505488Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1506035Z "size": 565, 2024-06-26T07:07:50.1506568Z "digest": "sha256:671b4a301d81074fca17d36851f81801552c4ecb2f90ff3412aa9ab6a1a91371" 2024-06-26T07:07:50.1507161Z }, 2024-06-26T07:07:50.1507417Z { 2024-06-26T07:07:50.1507844Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1508388Z "size": 43163095, 2024-06-26T07:07:50.1508937Z "digest": "sha256:67760cf5ec46ab58d3ca98637fd9b26cb47a67446bfd0db176fe5e023606aa0d" 2024-06-26T07:07:50.1509546Z }, 2024-06-26T07:07:50.1509851Z { 2024-06-26T07:07:50.1510269Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1510815Z "size": 106, 2024-06-26T07:07:50.1511590Z "digest": "sha256:ddc9740bcea2b301e99b50525059aea8d9c7121a1c8eab566561a8968438a8c2" 2024-06-26T07:07:50.1512260Z }, 2024-06-26T07:07:50.1512496Z { 2024-06-26T07:07:50.1512921Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1513482Z "size": 1213, 2024-06-26T07:07:50.1514017Z "digest": "sha256:fb2e977685609fcc01de170ba7f50a573e73e6520e47253717e92a28952c5ee8" 2024-06-26T07:07:50.1514614Z }, 2024-06-26T07:07:50.1514866Z { 2024-06-26T07:07:50.1515291Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1515949Z "size": 702, 2024-06-26T07:07:50.1516478Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:07:50.1517087Z }, 2024-06-26T07:07:50.1517332Z { 2024-06-26T07:07:50.1517757Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1518305Z "size": 138, 2024-06-26T07:07:50.1518845Z "digest": "sha256:b599c5a8c7cb461cbd7b3c98b99960b8eb04506e05e79db1cc0cfadb14a47e91" 2024-06-26T07:07:50.1519462Z }, 2024-06-26T07:07:50.1519702Z { 2024-06-26T07:07:50.1520127Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1520668Z "size": 120, 2024-06-26T07:07:50.1521209Z "digest": "sha256:e08e88c1366827455ee7ddbf92b6ae615ceb3cbd3fb11cd03ddd31bd3fa94a9c" 2024-06-26T07:07:50.1521854Z }, 2024-06-26T07:07:50.1522108Z { 2024-06-26T07:07:50.1522534Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1523073Z "size": 1904157495, 2024-06-26T07:07:50.1523638Z "digest": "sha256:12ba7431ff38de8cf6af3052a2a52e79f4e712b8c99f5bafde3c20e63b1f570f" 2024-06-26T07:07:50.1524253Z }, 2024-06-26T07:07:50.1524500Z { 2024-06-26T07:07:50.1524919Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1525472Z "size": 172, 2024-06-26T07:07:50.1526009Z "digest": "sha256:cf7c2cb896cba6a594028cc0fd246e127a2f4481d48905a3aa12c5f5872332d4" 2024-06-26T07:07:50.1526625Z }, 2024-06-26T07:07:50.1526863Z { 2024-06-26T07:07:50.1527288Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1527835Z "size": 908, 2024-06-26T07:07:50.1528381Z "digest": "sha256:3a778ddfac2af5b3a11f49ddf2f3f1ad1731f74ba691f434ab69966d1e2a189f" 2024-06-26T07:07:50.1529096Z }, 2024-06-26T07:07:50.1529349Z { 2024-06-26T07:07:50.1529776Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1530316Z "size": 702, 2024-06-26T07:07:50.1530859Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:07:50.1531469Z }, 2024-06-26T07:07:50.1531716Z { 2024-06-26T07:07:50.1532132Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1532687Z "size": 134, 2024-06-26T07:07:50.1533212Z "digest": "sha256:236c8f0245f4b40616a4322eaf1e1b3e178d5990db1fcaba39622ea4e79c06a1" 2024-06-26T07:07:50.1533817Z }, 2024-06-26T07:07:50.1534057Z { 2024-06-26T07:07:50.1534484Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1535028Z "size": 32, 2024-06-26T07:07:50.1535562Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:07:50.1536169Z }, 2024-06-26T07:07:50.1536429Z { 2024-06-26T07:07:50.1536857Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1537392Z "size": 156, 2024-06-26T07:07:50.1537937Z "digest": "sha256:dda1ce302b9a9df0c8f1145d2e3eaa03c96909bb45a792451f1fcb555b06738d" 2024-06-26T07:07:50.1538556Z }, 2024-06-26T07:07:50.1538802Z { 2024-06-26T07:07:50.1539223Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1539771Z "size": 1841, 2024-06-26T07:07:50.1540378Z "digest": "sha256:e797fe86caef19319e03f8dca92cf96838ab3bb3644f4447c6dfc04230118afa" 2024-06-26T07:07:50.1540988Z }, 2024-06-26T07:07:50.1541226Z { 2024-06-26T07:07:50.1541654Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1542203Z "size": 7529784, 2024-06-26T07:07:50.1542742Z "digest": "sha256:967190076fc0ab2bb2fc0d74e58edd3d662a537d702ff3f8b6e876230154a16f" 2024-06-26T07:07:50.1543351Z }, 2024-06-26T07:07:50.1543605Z { 2024-06-26T07:07:50.1544034Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1544574Z "size": 165, 2024-06-26T07:07:50.1545107Z "digest": "sha256:4fb0a83f06855534ef82b80dcf7c685303edb2bcbd446196066dc2ad2dd71281" 2024-06-26T07:07:50.1545718Z }, 2024-06-26T07:07:50.1545965Z { 2024-06-26T07:07:50.1546384Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1546928Z "size": 7944, 2024-06-26T07:07:50.1547468Z "digest": "sha256:68af2ec4e30a33f26947ece9f3a31b7d3b04c2bca5865d4b48618457832a1da2" 2024-06-26T07:07:50.1548079Z }, 2024-06-26T07:07:50.1548316Z { 2024-06-26T07:07:50.1548737Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1549289Z "size": 8070, 2024-06-26T07:07:50.1549814Z "digest": "sha256:cc91933e985a044c6df53caa15488d2285896f5e991f38ec068c253ceec53f14" 2024-06-26T07:07:50.1550416Z }, 2024-06-26T07:07:50.1550668Z { 2024-06-26T07:07:50.1551271Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1551849Z "size": 303, 2024-06-26T07:07:50.1552378Z "digest": "sha256:9f3187916e7dcad33cc572f8f0e949d2f507b887f99cfd6acefd14826b44960d" 2024-06-26T07:07:50.1552986Z }, 2024-06-26T07:07:50.1553228Z { 2024-06-26T07:07:50.1553644Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1554186Z "size": 7629071, 2024-06-26T07:07:50.1554706Z "digest": "sha256:52608399485738d10d779254946c9e1d536010dd45593c69bf1eb9f62f85dc24" 2024-06-26T07:07:50.1555285Z }, 2024-06-26T07:07:50.1555529Z { 2024-06-26T07:07:50.1556052Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1556595Z "size": 108, 2024-06-26T07:07:50.1557116Z "digest": "sha256:05a6f6dcc1c00bac6660923a22adc7d7c34416718f8c89daf9b3351d9365f694" 2024-06-26T07:07:50.1557723Z }, 2024-06-26T07:07:50.1557963Z { 2024-06-26T07:07:50.1558387Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1559025Z "size": 54145775, 2024-06-26T07:07:50.1559584Z "digest": "sha256:6bc9dc8315b09b27c976bfba9700767ba412d72fd0528b18bbe8f8e62005ec22" 2024-06-26T07:07:50.1560189Z }, 2024-06-26T07:07:50.1560423Z { 2024-06-26T07:07:50.1560850Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1561393Z "size": 474, 2024-06-26T07:07:50.1561925Z "digest": "sha256:1b0e27c35f07b3fbf0aeb548d1949bf1d60a420136da21789d94d196c891d981" 2024-06-26T07:07:50.1562541Z }, 2024-06-26T07:07:50.1562790Z { 2024-06-26T07:07:50.1563209Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1563757Z "size": 1374858747, 2024-06-26T07:07:50.1564293Z "digest": "sha256:95237e7311a1c5ce6937db7196c866632f44e8788514a5c0eb4a3f04da4017e6" 2024-06-26T07:07:50.1564886Z }, 2024-06-26T07:07:50.1565130Z { 2024-06-26T07:07:50.1565548Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1566081Z "size": 106, 2024-06-26T07:07:50.1566626Z "digest": "sha256:789eff6c51aada70bf75eb1acafed63c0d839b768638001500616ff6e53b854a" 2024-06-26T07:07:50.1567247Z }, 2024-06-26T07:07:50.1567480Z { 2024-06-26T07:07:50.1567900Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1568441Z "size": 558, 2024-06-26T07:07:50.1568977Z "digest": "sha256:de0f8a8c45c6077b3c5ba3b84d7f07de98353fd618dad9a1e2f368ffe00493c7" 2024-06-26T07:07:50.1569581Z }, 2024-06-26T07:07:50.1569901Z { 2024-06-26T07:07:50.1570326Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1570871Z "size": 46248513, 2024-06-26T07:07:50.1571400Z "digest": "sha256:2060028f74888d218bebdd10d723f66c33ee1c708935c4ffd4654d34912ac4dc" 2024-06-26T07:07:50.1571996Z }, 2024-06-26T07:07:50.1572235Z { 2024-06-26T07:07:50.1572648Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1573193Z "size": 111, 2024-06-26T07:07:50.1573727Z "digest": "sha256:e412c7f3b2ded440992955bd92512345cbece8a8894cd3592f714ebc758ad1a0" 2024-06-26T07:07:50.1574327Z }, 2024-06-26T07:07:50.1574565Z { 2024-06-26T07:07:50.1574990Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1575526Z "size": 32, 2024-06-26T07:07:50.1576054Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:07:50.1576654Z }, 2024-06-26T07:07:50.1576895Z { 2024-06-26T07:07:50.1577320Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1577858Z "size": 32, 2024-06-26T07:07:50.1578376Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:07:50.1578985Z }, 2024-06-26T07:07:50.1579225Z { 2024-06-26T07:07:50.1579637Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1580186Z "size": 32, 2024-06-26T07:07:50.1580716Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:07:50.1581323Z }, 2024-06-26T07:07:50.1581556Z { 2024-06-26T07:07:50.1581980Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:07:50.1582529Z "size": 32, 2024-06-26T07:07:50.1583063Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:07:50.1583665Z } 2024-06-26T07:07:50.1583908Z ] 2024-06-26T07:07:50.1584147Z } 2024-06-26T07:07:50.1697255Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:07:50.1697748Z tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:07:50.1698303Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-06-26T07:07:50.1706157Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:50.1706660Z env: 2024-06-26T07:07:50.1706943Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:50.1708095Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:50.1709263Z ##[endgroup] 2024-06-26T07:07:50.1732167Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:50.1780827Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-06-26T07:07:50.1781405Z with: 2024-06-26T07:07:50.1782479Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:50.1783817Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:50.1784327Z env: 2024-06-26T07:07:50.1784603Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:50.1784953Z ##[endgroup] 2024-06-26T07:07:50.1799913Z ##[group]Run set -x 2024-06-26T07:07:50.1800261Z set -x 2024-06-26T07:07:50.1800563Z set +e 2024-06-26T07:07:50.1800857Z  2024-06-26T07:07:50.1801132Z login() { 2024-06-26T07:07:50.1801798Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:07:50.1802519Z } 2024-06-26T07:07:50.1802805Z  2024-06-26T07:07:50.1803108Z retry () { 2024-06-26T07:07:50.1803506Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:07:50.1803959Z } 2024-06-26T07:07:50.1804238Z  2024-06-26T07:07:50.1804559Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:07:50.1805079Z  2024-06-26T07:07:50.1805354Z set -e 2024-06-26T07:07:50.1805833Z # ignore output since only exit code is used for conditional 2024-06-26T07:07:50.1806535Z # only pull docker image if it's not available locally 2024-06-26T07:07:50.1807299Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-06-26T07:07:50.1807993Z  retry docker pull "${DOCKER_IMAGE}" 2024-06-26T07:07:50.1808431Z fi 2024-06-26T07:07:50.1815964Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:50.1816459Z env: 2024-06-26T07:07:50.1816733Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:50.1817869Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:50.1819163Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:50.1819686Z ##[endgroup] 2024-06-26T07:07:50.1839414Z + set +e 2024-06-26T07:07:50.1840092Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:50.1840959Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:50.1842186Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:07:50.1842951Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:07:50.6011795Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:07:50.6012747Z Configure a credential helper to remove this warning. See 2024-06-26T07:07:50.6013779Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:07:50.6014386Z 2024-06-26T07:07:50.6014546Z Login Succeeded 2024-06-26T07:07:50.6022076Z + set -e 2024-06-26T07:07:50.6023446Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:50.6153559Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:50.6155886Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:50.8249049Z 91382da70d5719cd7007b6b80b71d2f48398f6b7: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-06-26T07:07:50.8250219Z 7a2c55901189: Pulling fs layer 2024-06-26T07:07:50.8250749Z 224fe954d725: Pulling fs layer 2024-06-26T07:07:50.8251455Z 75722010b82e: Pulling fs layer 2024-06-26T07:07:50.8251875Z d527cbbb87e3: Pulling fs layer 2024-06-26T07:07:50.8252351Z b57676e46aee: Pulling fs layer 2024-06-26T07:07:50.8252875Z a8c1e85b5e14: Pulling fs layer 2024-06-26T07:07:50.8253411Z a41a8d1c11c8: Pulling fs layer 2024-06-26T07:07:50.8253910Z 0c1227890755: Pulling fs layer 2024-06-26T07:07:50.8254434Z d8d1234baab3: Pulling fs layer 2024-06-26T07:07:50.8254934Z 7ed32bc8e469: Pulling fs layer 2024-06-26T07:07:50.8255398Z ec1e7978c1fe: Pulling fs layer 2024-06-26T07:07:50.8255776Z 309b872da805: Pulling fs layer 2024-06-26T07:07:50.8256155Z e82932cb002c: Pulling fs layer 2024-06-26T07:07:50.8256568Z ca9dd73e3e93: Pulling fs layer 2024-06-26T07:07:50.8256945Z 16d76a79d5d6: Pulling fs layer 2024-06-26T07:07:50.8257327Z 4bdf41c9ecbe: Pulling fs layer 2024-06-26T07:07:50.8257706Z f4863801e30c: Pulling fs layer 2024-06-26T07:07:50.8258074Z d37dd9ed8948: Pulling fs layer 2024-06-26T07:07:50.8258456Z cf2f34f225db: Pulling fs layer 2024-06-26T07:07:50.8258847Z df546d7b3e34: Pulling fs layer 2024-06-26T07:07:50.8259230Z 3d914950f8ad: Pulling fs layer 2024-06-26T07:07:50.8259599Z 96d0e2032184: Pulling fs layer 2024-06-26T07:07:50.8259976Z 06e12ce59790: Pulling fs layer 2024-06-26T07:07:50.8260495Z ed712f63cde7: Pulling fs layer 2024-06-26T07:07:50.8260873Z a97a71ef8a4e: Pulling fs layer 2024-06-26T07:07:50.8261244Z 25cd0c482ad6: Pulling fs layer 2024-06-26T07:07:50.8261607Z 0c1227890755: Waiting 2024-06-26T07:07:50.8261929Z 7ed32bc8e469: Waiting 2024-06-26T07:07:50.8262268Z 9bf94bc45658: Pulling fs layer 2024-06-26T07:07:50.8262634Z d8d1234baab3: Waiting 2024-06-26T07:07:50.8262972Z de45a14c9b22: Pulling fs layer 2024-06-26T07:07:50.8263335Z e82932cb002c: Waiting 2024-06-26T07:07:50.8263662Z e99241e77209: Pulling fs layer 2024-06-26T07:07:50.8264021Z ec1e7978c1fe: Waiting 2024-06-26T07:07:50.8264364Z 35e8f4e2a42f: Pulling fs layer 2024-06-26T07:07:50.8264734Z f4863801e30c: Waiting 2024-06-26T07:07:50.8265061Z a823ffa4c541: Pulling fs layer 2024-06-26T07:07:50.8265426Z 309b872da805: Waiting 2024-06-26T07:07:50.8265763Z 44ba272eae20: Pulling fs layer 2024-06-26T07:07:50.8266115Z cf2f34f225db: Waiting 2024-06-26T07:07:50.8266444Z 3d914950f8ad: Waiting 2024-06-26T07:07:50.8266879Z 06e12ce59790: Waiting 2024-06-26T07:07:50.8267318Z d37dd9ed8948: Waiting 2024-06-26T07:07:50.8267785Z da02d278e524: Pulling fs layer 2024-06-26T07:07:50.8268330Z 4bdf41c9ecbe: Waiting 2024-06-26T07:07:50.8268770Z ca9dd73e3e93: Waiting 2024-06-26T07:07:50.8269093Z df546d7b3e34: Waiting 2024-06-26T07:07:50.8269492Z 73a0ce681c29: Pulling fs layer 2024-06-26T07:07:50.8269857Z 16d76a79d5d6: Waiting 2024-06-26T07:07:50.8270217Z 49737bd5ae70: Pulling fs layer 2024-06-26T07:07:50.8270589Z 9cd4c745f8e4: Pulling fs layer 2024-06-26T07:07:50.8270948Z 96d0e2032184: Waiting 2024-06-26T07:07:50.8271574Z b2950a036534: Pulling fs layer 2024-06-26T07:07:50.8271974Z 25cd0c482ad6: Waiting 2024-06-26T07:07:50.8272297Z a8c1e85b5e14: Waiting 2024-06-26T07:07:50.8272632Z 25e19d81d9fc: Pulling fs layer 2024-06-26T07:07:50.8273005Z ed712f63cde7: Waiting 2024-06-26T07:07:50.8273333Z 60156b40fcbe: Pulling fs layer 2024-06-26T07:07:50.8273711Z 7a868239227f: Pulling fs layer 2024-06-26T07:07:50.8274070Z 9bf94bc45658: Waiting 2024-06-26T07:07:50.8274407Z 83911d7c7ecf: Pulling fs layer 2024-06-26T07:07:50.8274782Z b3c5d66c679f: Pulling fs layer 2024-06-26T07:07:50.8275156Z 84b2c7a89982: Pulling fs layer 2024-06-26T07:07:50.8275515Z de45a14c9b22: Waiting 2024-06-26T07:07:50.8275942Z da02d278e524: Waiting 2024-06-26T07:07:50.8276253Z 35e8f4e2a42f: Waiting 2024-06-26T07:07:50.8276583Z c1068d03a811: Pulling fs layer 2024-06-26T07:07:50.8276943Z 9cd4c745f8e4: Waiting 2024-06-26T07:07:50.8277265Z 671b4a301d81: Pulling fs layer 2024-06-26T07:07:50.8277625Z 49737bd5ae70: Waiting 2024-06-26T07:07:50.8277956Z 67760cf5ec46: Pulling fs layer 2024-06-26T07:07:50.8278381Z b2950a036534: Waiting 2024-06-26T07:07:50.8278712Z ddc9740bcea2: Pulling fs layer 2024-06-26T07:07:50.8279072Z a823ffa4c541: Waiting 2024-06-26T07:07:50.8279582Z fb2e97768560: Pulling fs layer 2024-06-26T07:07:50.8279965Z b599c5a8c7cb: Pulling fs layer 2024-06-26T07:07:50.8280332Z e08e88c13668: Pulling fs layer 2024-06-26T07:07:50.8280710Z 12ba7431ff38: Pulling fs layer 2024-06-26T07:07:50.8281208Z 7a868239227f: Waiting 2024-06-26T07:07:50.8281651Z d527cbbb87e3: Waiting 2024-06-26T07:07:50.8282119Z cf7c2cb896cb: Pulling fs layer 2024-06-26T07:07:50.8282641Z 44ba272eae20: Waiting 2024-06-26T07:07:50.8283047Z 84b2c7a89982: Waiting 2024-06-26T07:07:50.8283378Z 3a778ddfac2a: Pulling fs layer 2024-06-26T07:07:50.8283739Z c1068d03a811: Waiting 2024-06-26T07:07:50.8284054Z 25e19d81d9fc: Waiting 2024-06-26T07:07:50.8284366Z ddc9740bcea2: Waiting 2024-06-26T07:07:50.8284683Z 73a0ce681c29: Waiting 2024-06-26T07:07:50.8284998Z 671b4a301d81: Waiting 2024-06-26T07:07:50.8285308Z 67760cf5ec46: Waiting 2024-06-26T07:07:50.8285615Z b3c5d66c679f: Waiting 2024-06-26T07:07:50.8285948Z 236c8f0245f4: Pulling fs layer 2024-06-26T07:07:50.8286324Z 4f4fb700ef54: Pulling fs layer 2024-06-26T07:07:50.8286695Z 3a778ddfac2a: Waiting 2024-06-26T07:07:50.8287030Z dda1ce302b9a: Pulling fs layer 2024-06-26T07:07:50.8287391Z 12ba7431ff38: Waiting 2024-06-26T07:07:50.8287724Z e797fe86caef: Pulling fs layer 2024-06-26T07:07:50.8288077Z 236c8f0245f4: Waiting 2024-06-26T07:07:50.8288514Z 967190076fc0: Pulling fs layer 2024-06-26T07:07:50.8288886Z 4fb0a83f0685: Pulling fs layer 2024-06-26T07:07:50.8289244Z b57676e46aee: Waiting 2024-06-26T07:07:50.8289570Z 68af2ec4e30a: Pulling fs layer 2024-06-26T07:07:50.8289932Z 4f4fb700ef54: Waiting 2024-06-26T07:07:50.8290244Z 967190076fc0: Waiting 2024-06-26T07:07:50.8290567Z cc91933e985a: Pulling fs layer 2024-06-26T07:07:50.8290931Z e797fe86caef: Waiting 2024-06-26T07:07:50.8291326Z 9f3187916e7d: Pulling fs layer 2024-06-26T07:07:50.8291695Z 526083994857: Pulling fs layer 2024-06-26T07:07:50.8292043Z 83911d7c7ecf: Waiting 2024-06-26T07:07:50.8292384Z 05a6f6dcc1c0: Pulling fs layer 2024-06-26T07:07:50.8292742Z 4fb0a83f0685: Waiting 2024-06-26T07:07:50.8293063Z cc91933e985a: Waiting 2024-06-26T07:07:50.8293387Z 6bc9dc8315b0: Pulling fs layer 2024-06-26T07:07:50.8293793Z 9f3187916e7d: Waiting 2024-06-26T07:07:50.8294109Z 526083994857: Waiting 2024-06-26T07:07:50.8294432Z 1b0e27c35f07: Pulling fs layer 2024-06-26T07:07:50.8294814Z 95237e7311a1: Pulling fs layer 2024-06-26T07:07:50.8295188Z 789eff6c51aa: Pulling fs layer 2024-06-26T07:07:50.8295565Z de0f8a8c45c6: Pulling fs layer 2024-06-26T07:07:50.8295915Z e08e88c13668: Waiting 2024-06-26T07:07:50.8296271Z 95237e7311a1: Waiting 2024-06-26T07:07:50.8296727Z 2060028f7488: Pulling fs layer 2024-06-26T07:07:50.8297246Z 6bc9dc8315b0: Waiting 2024-06-26T07:07:50.8297681Z b599c5a8c7cb: Waiting 2024-06-26T07:07:50.8298164Z e412c7f3b2de: Pulling fs layer 2024-06-26T07:07:50.8298682Z de0f8a8c45c6: Waiting 2024-06-26T07:07:50.8299112Z e412c7f3b2de: Waiting 2024-06-26T07:07:50.8299552Z 1b0e27c35f07: Waiting 2024-06-26T07:07:50.9665706Z 224fe954d725: Verifying Checksum 2024-06-26T07:07:50.9666176Z 224fe954d725: Download complete 2024-06-26T07:07:51.0694670Z d527cbbb87e3: Verifying Checksum 2024-06-26T07:07:51.0695101Z d527cbbb87e3: Download complete 2024-06-26T07:07:51.1541695Z b57676e46aee: Verifying Checksum 2024-06-26T07:07:51.1542123Z b57676e46aee: Download complete 2024-06-26T07:07:51.1703741Z 7a2c55901189: Verifying Checksum 2024-06-26T07:07:51.1704301Z 7a2c55901189: Download complete 2024-06-26T07:07:51.2553805Z a41a8d1c11c8: Download complete 2024-06-26T07:07:51.3387813Z 0c1227890755: Verifying Checksum 2024-06-26T07:07:51.3388322Z 0c1227890755: Download complete 2024-06-26T07:07:51.4363719Z d8d1234baab3: Download complete 2024-06-26T07:07:51.5932619Z 75722010b82e: Verifying Checksum 2024-06-26T07:07:51.5933118Z 75722010b82e: Download complete 2024-06-26T07:07:51.6858557Z ec1e7978c1fe: Verifying Checksum 2024-06-26T07:07:51.6859104Z ec1e7978c1fe: Download complete 2024-06-26T07:07:51.7531953Z 309b872da805: Verifying Checksum 2024-06-26T07:07:51.7532471Z 309b872da805: Download complete 2024-06-26T07:07:52.2025838Z 7a2c55901189: Pull complete 2024-06-26T07:07:52.5116640Z 224fe954d725: Pull complete 2024-06-26T07:07:53.4741043Z 75722010b82e: Pull complete 2024-06-26T07:07:53.5121851Z d527cbbb87e3: Pull complete 2024-06-26T07:07:53.5626608Z b57676e46aee: Pull complete 2024-06-26T07:07:54.2836747Z e82932cb002c: Verifying Checksum 2024-06-26T07:07:54.2837295Z e82932cb002c: Download complete 2024-06-26T07:07:54.3957553Z ca9dd73e3e93: Download complete 2024-06-26T07:07:54.4817018Z 16d76a79d5d6: Verifying Checksum 2024-06-26T07:07:54.4818051Z 16d76a79d5d6: Download complete 2024-06-26T07:07:54.5582941Z 4bdf41c9ecbe: Verifying Checksum 2024-06-26T07:07:54.5583534Z 4bdf41c9ecbe: Download complete 2024-06-26T07:07:55.5364896Z f4863801e30c: Verifying Checksum 2024-06-26T07:07:55.5365376Z f4863801e30c: Download complete 2024-06-26T07:07:55.6127840Z d37dd9ed8948: Verifying Checksum 2024-06-26T07:07:55.6128295Z d37dd9ed8948: Download complete 2024-06-26T07:07:55.6803646Z cf2f34f225db: Verifying Checksum 2024-06-26T07:07:55.6804282Z cf2f34f225db: Download complete 2024-06-26T07:07:55.7376493Z df546d7b3e34: Verifying Checksum 2024-06-26T07:07:55.7377086Z df546d7b3e34: Download complete 2024-06-26T07:08:04.5501511Z a8c1e85b5e14: Verifying Checksum 2024-06-26T07:08:04.5501991Z a8c1e85b5e14: Download complete 2024-06-26T07:08:04.6324824Z 96d0e2032184: Verifying Checksum 2024-06-26T07:08:04.6325465Z 96d0e2032184: Download complete 2024-06-26T07:08:04.7084958Z 06e12ce59790: Verifying Checksum 2024-06-26T07:08:04.7085459Z 06e12ce59790: Download complete 2024-06-26T07:08:04.7941988Z ed712f63cde7: Verifying Checksum 2024-06-26T07:08:04.7942455Z ed712f63cde7: Download complete 2024-06-26T07:08:04.8831423Z a97a71ef8a4e: Download complete 2024-06-26T07:08:04.9692522Z 25cd0c482ad6: Verifying Checksum 2024-06-26T07:08:04.9693084Z 25cd0c482ad6: Download complete 2024-06-26T07:08:06.3164943Z 9bf94bc45658: Verifying Checksum 2024-06-26T07:08:06.3165449Z 9bf94bc45658: Download complete 2024-06-26T07:08:06.3862206Z de45a14c9b22: Verifying Checksum 2024-06-26T07:08:06.3862678Z de45a14c9b22: Download complete 2024-06-26T07:08:06.4693745Z e99241e77209: Download complete 2024-06-26T07:08:06.5434307Z 35e8f4e2a42f: Download complete 2024-06-26T07:08:06.6449250Z a823ffa4c541: Download complete 2024-06-26T07:08:06.7740530Z 44ba272eae20: Download complete 2024-06-26T07:08:11.2929221Z da02d278e524: Verifying Checksum 2024-06-26T07:08:11.2929731Z da02d278e524: Download complete 2024-06-26T07:08:11.3801033Z 73a0ce681c29: Verifying Checksum 2024-06-26T07:08:11.3801485Z 73a0ce681c29: Download complete 2024-06-26T07:08:11.4724230Z 49737bd5ae70: Verifying Checksum 2024-06-26T07:08:11.4724797Z 49737bd5ae70: Download complete 2024-06-26T07:08:11.8955518Z 9cd4c745f8e4: Verifying Checksum 2024-06-26T07:08:11.8956347Z 9cd4c745f8e4: Download complete 2024-06-26T07:08:11.9767775Z b2950a036534: Verifying Checksum 2024-06-26T07:08:11.9768267Z b2950a036534: Download complete 2024-06-26T07:08:12.0580855Z 25e19d81d9fc: Download complete 2024-06-26T07:08:12.3191584Z 60156b40fcbe: Verifying Checksum 2024-06-26T07:08:12.3192327Z 60156b40fcbe: Download complete 2024-06-26T07:08:12.4024570Z 7a868239227f: Download complete 2024-06-26T07:08:12.5084742Z 83911d7c7ecf: Download complete 2024-06-26T07:08:12.5844883Z b3c5d66c679f: Verifying Checksum 2024-06-26T07:08:12.5845540Z b3c5d66c679f: Download complete 2024-06-26T07:08:16.7906699Z 7ed32bc8e469: Verifying Checksum 2024-06-26T07:08:16.7907387Z 7ed32bc8e469: Download complete 2024-06-26T07:08:16.8703981Z c1068d03a811: Verifying Checksum 2024-06-26T07:08:16.8704439Z c1068d03a811: Download complete 2024-06-26T07:08:16.9568174Z 671b4a301d81: Verifying Checksum 2024-06-26T07:08:16.9568755Z 671b4a301d81: Download complete 2024-06-26T07:08:17.5737072Z a8c1e85b5e14: Pull complete 2024-06-26T07:08:17.7981458Z a41a8d1c11c8: Pull complete 2024-06-26T07:08:17.8038195Z 67760cf5ec46: Verifying Checksum 2024-06-26T07:08:17.8038616Z 67760cf5ec46: Download complete 2024-06-26T07:08:17.8751379Z ddc9740bcea2: Verifying Checksum 2024-06-26T07:08:17.8751824Z ddc9740bcea2: Download complete 2024-06-26T07:08:17.9595987Z fb2e97768560: Download complete 2024-06-26T07:08:18.0246686Z 0c1227890755: Pull complete 2024-06-26T07:08:18.0308929Z b599c5a8c7cb: Verifying Checksum 2024-06-26T07:08:18.0309360Z b599c5a8c7cb: Download complete 2024-06-26T07:08:18.1055517Z e08e88c13668: Verifying Checksum 2024-06-26T07:08:18.1055945Z e08e88c13668: Download complete 2024-06-26T07:08:18.2572818Z d8d1234baab3: Pull complete 2024-06-26T07:08:24.0065796Z 3d914950f8ad: Verifying Checksum 2024-06-26T07:08:24.0066309Z 3d914950f8ad: Download complete 2024-06-26T07:08:24.0962411Z cf7c2cb896cb: Verifying Checksum 2024-06-26T07:08:24.0962945Z cf7c2cb896cb: Download complete 2024-06-26T07:08:24.1786489Z 3a778ddfac2a: Download complete 2024-06-26T07:08:24.2639253Z 236c8f0245f4: Verifying Checksum 2024-06-26T07:08:24.2639728Z 236c8f0245f4: Download complete 2024-06-26T07:08:24.2729806Z 4f4fb700ef54: Verifying Checksum 2024-06-26T07:08:24.2730373Z 4f4fb700ef54: Download complete 2024-06-26T07:08:24.3436789Z dda1ce302b9a: Verifying Checksum 2024-06-26T07:08:24.3437216Z dda1ce302b9a: Download complete 2024-06-26T07:08:24.4153376Z e797fe86caef: Download complete 2024-06-26T07:08:24.5515827Z 967190076fc0: Verifying Checksum 2024-06-26T07:08:24.5516446Z 967190076fc0: Download complete 2024-06-26T07:08:24.6357071Z 4fb0a83f0685: Verifying Checksum 2024-06-26T07:08:24.6357630Z 4fb0a83f0685: Download complete 2024-06-26T07:08:24.7083211Z 68af2ec4e30a: Download complete 2024-06-26T07:08:24.7844312Z cc91933e985a: Verifying Checksum 2024-06-26T07:08:24.7844857Z cc91933e985a: Download complete 2024-06-26T07:08:24.8598050Z 9f3187916e7d: Download complete 2024-06-26T07:08:25.0114802Z 526083994857: Verifying Checksum 2024-06-26T07:08:25.0115288Z 526083994857: Download complete 2024-06-26T07:08:25.0807794Z 05a6f6dcc1c0: Verifying Checksum 2024-06-26T07:08:25.0808294Z 05a6f6dcc1c0: Download complete 2024-06-26T07:08:25.8972304Z 6bc9dc8315b0: Verifying Checksum 2024-06-26T07:08:25.8972784Z 6bc9dc8315b0: Download complete 2024-06-26T07:08:25.9750399Z 1b0e27c35f07: Download complete 2024-06-26T07:08:48.5342512Z 12ba7431ff38: Verifying Checksum 2024-06-26T07:08:48.5342985Z 12ba7431ff38: Download complete 2024-06-26T07:08:48.6015816Z 789eff6c51aa: Verifying Checksum 2024-06-26T07:08:48.6016488Z 789eff6c51aa: Download complete 2024-06-26T07:08:48.6881605Z de0f8a8c45c6: Download complete 2024-06-26T07:08:50.1451761Z 2060028f7488: Verifying Checksum 2024-06-26T07:08:50.1452334Z 2060028f7488: Download complete 2024-06-26T07:08:50.2212787Z e412c7f3b2de: Verifying Checksum 2024-06-26T07:08:50.2213279Z e412c7f3b2de: Download complete 2024-06-26T07:08:53.8883701Z 95237e7311a1: Verifying Checksum 2024-06-26T07:08:53.8884250Z 95237e7311a1: Download complete 2024-06-26T07:09:16.7190776Z 84b2c7a89982: Verifying Checksum 2024-06-26T07:09:16.7191619Z 84b2c7a89982: Download complete 2024-06-26T07:09:34.1185395Z 7ed32bc8e469: Pull complete 2024-06-26T07:09:34.3517313Z ec1e7978c1fe: Pull complete 2024-06-26T07:09:34.5855984Z 309b872da805: Pull complete 2024-06-26T07:09:42.0155636Z e82932cb002c: Pull complete 2024-06-26T07:09:42.2358270Z ca9dd73e3e93: Pull complete 2024-06-26T07:09:42.4342343Z 16d76a79d5d6: Pull complete 2024-06-26T07:09:42.6431605Z 4bdf41c9ecbe: Pull complete 2024-06-26T07:09:45.0299773Z f4863801e30c: Pull complete 2024-06-26T07:09:45.1766386Z d37dd9ed8948: Pull complete 2024-06-26T07:09:45.3496486Z cf2f34f225db: Pull complete 2024-06-26T07:09:45.5638111Z df546d7b3e34: Pull complete 2024-06-26T07:10:34.9005778Z 3d914950f8ad: Pull complete 2024-06-26T07:10:35.0621586Z 96d0e2032184: Pull complete 2024-06-26T07:10:35.1997771Z 06e12ce59790: Pull complete 2024-06-26T07:10:35.2870216Z ed712f63cde7: Pull complete 2024-06-26T07:10:35.3760804Z a97a71ef8a4e: Pull complete 2024-06-26T07:10:35.4720084Z 25cd0c482ad6: Pull complete 2024-06-26T07:10:38.9347348Z 9bf94bc45658: Pull complete 2024-06-26T07:10:39.0209298Z de45a14c9b22: Pull complete 2024-06-26T07:10:39.1912571Z e99241e77209: Pull complete 2024-06-26T07:10:39.4051514Z 35e8f4e2a42f: Pull complete 2024-06-26T07:10:39.6172310Z a823ffa4c541: Pull complete 2024-06-26T07:10:39.8219898Z 44ba272eae20: Pull complete 2024-06-26T07:10:48.1468590Z da02d278e524: Pull complete 2024-06-26T07:10:48.3339337Z 73a0ce681c29: Pull complete 2024-06-26T07:10:48.5546882Z 49737bd5ae70: Pull complete 2024-06-26T07:10:49.4915201Z 9cd4c745f8e4: Pull complete 2024-06-26T07:10:49.6619508Z b2950a036534: Pull complete 2024-06-26T07:10:49.8438019Z 25e19d81d9fc: Pull complete 2024-06-26T07:10:50.2998734Z 60156b40fcbe: Pull complete 2024-06-26T07:10:50.4647690Z 7a868239227f: Pull complete 2024-06-26T07:10:50.8365751Z 83911d7c7ecf: Pull complete 2024-06-26T07:10:51.0509886Z b3c5d66c679f: Pull complete 2024-06-26T07:11:30.0328019Z 84b2c7a89982: Pull complete 2024-06-26T07:11:30.2503747Z c1068d03a811: Pull complete 2024-06-26T07:11:30.3954743Z 671b4a301d81: Pull complete 2024-06-26T07:11:32.3778541Z 67760cf5ec46: Pull complete 2024-06-26T07:11:32.4327718Z ddc9740bcea2: Pull complete 2024-06-26T07:11:32.4677506Z fb2e97768560: Pull complete 2024-06-26T07:11:32.5795408Z b599c5a8c7cb: Pull complete 2024-06-26T07:11:32.6313329Z e08e88c13668: Pull complete 2024-06-26T07:12:06.7037733Z 12ba7431ff38: Pull complete 2024-06-26T07:12:06.8302850Z cf7c2cb896cb: Pull complete 2024-06-26T07:12:06.9469908Z 3a778ddfac2a: Pull complete 2024-06-26T07:12:07.3629733Z 236c8f0245f4: Pull complete 2024-06-26T07:12:07.5042579Z 4f4fb700ef54: Pull complete 2024-06-26T07:12:07.6212420Z dda1ce302b9a: Pull complete 2024-06-26T07:12:07.8446699Z e797fe86caef: Pull complete 2024-06-26T07:12:08.2085334Z 967190076fc0: Pull complete 2024-06-26T07:12:08.4129910Z 4fb0a83f0685: Pull complete 2024-06-26T07:12:08.6219282Z 68af2ec4e30a: Pull complete 2024-06-26T07:12:08.8091865Z cc91933e985a: Pull complete 2024-06-26T07:12:09.0176583Z 9f3187916e7d: Pull complete 2024-06-26T07:12:10.0678256Z 526083994857: Pull complete 2024-06-26T07:12:10.2273070Z 05a6f6dcc1c0: Pull complete 2024-06-26T07:12:12.3037750Z 6bc9dc8315b0: Pull complete 2024-06-26T07:12:12.4685660Z 1b0e27c35f07: Pull complete 2024-06-26T07:12:25.7389999Z 95237e7311a1: Pull complete 2024-06-26T07:12:25.8094698Z 789eff6c51aa: Pull complete 2024-06-26T07:12:25.8999290Z de0f8a8c45c6: Pull complete 2024-06-26T07:12:26.7284601Z 2060028f7488: Pull complete 2024-06-26T07:12:26.9229548Z e412c7f3b2de: Pull complete 2024-06-26T07:12:27.8850593Z Digest: sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T07:12:27.9310341Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:12:27.9511355Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:12:27.9563840Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:12:27.9564772Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:12:27.9572754Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:12:27.9573252Z env: 2024-06-26T07:12:27.9573530Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:27.9573874Z ##[endgroup] 2024-06-26T07:12:27.9727103Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-06-26T07:12:27.9727651Z with: 2024-06-26T07:12:27.9727935Z driver-version: 550.54.15 2024-06-26T07:12:27.9728283Z env: 2024-06-26T07:12:27.9728546Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:27.9728891Z ##[endgroup] 2024-06-26T07:12:27.9862786Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:12:27.9863356Z with: 2024-06-26T07:12:27.9863641Z timeout_minutes: 10 2024-06-26T07:12:27.9863968Z max_attempts: 3 2024-06-26T07:12:27.9895228Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-06-26T07:12:27.9926649Z retry_wait_seconds: 10 2024-06-26T07:12:27.9927017Z polling_interval_seconds: 1 2024-06-26T07:12:27.9927382Z warning_on_retry: true 2024-06-26T07:12:27.9927733Z continue_on_error: false 2024-06-26T07:12:27.9928064Z env: 2024-06-26T07:12:27.9928333Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:12:27.9928763Z DRIVER_VERSION: 550.54.15 2024-06-26T07:12:27.9929111Z ##[endgroup] 2024-06-26T07:12:28.0454265Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-06-26T07:12:28.0455017Z + pre_install_nvidia_driver_amzn2 2024-06-26T07:12:28.0455681Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-06-26T07:12:28.3562753Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:12:28.3948749Z No Match for argument: nvidia-driver-latest-dkms 2024-06-26T07:12:28.4193221Z No Packages marked for removal 2024-06-26T07:12:28.4316262Z + install_nvidia_driver_common 2024-06-26T07:12:28.4319350Z + echo 'Before installing NVIDIA driver' 2024-06-26T07:12:28.4320007Z + lspci 2024-06-26T07:12:28.4320429Z Before installing NVIDIA driver 2024-06-26T07:12:28.4396899Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:12:28.4398066Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:12:28.4399345Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:12:28.4400394Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:12:28.4401162Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:12:28.4401922Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:12:28.4402645Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:12:28.4403430Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:12:28.4404024Z + lsmod 2024-06-26T07:12:28.4411651Z Module Size Used by 2024-06-26T07:12:28.4412279Z xt_nat 16384 0 2024-06-26T07:12:28.4412868Z ib_core 266240 0 2024-06-26T07:12:28.4413403Z nvidia_modeset 1339392 0 2024-06-26T07:12:28.4413778Z veth 16384 0 2024-06-26T07:12:28.4414154Z nvidia_uvm 4599808 0 2024-06-26T07:12:28.4415069Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:12:28.4415827Z drm 421888 1 nvidia 2024-06-26T07:12:28.4416378Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:12:28.4417012Z backlight 16384 1 nvidia_modeset 2024-06-26T07:12:28.4417527Z xt_conntrack 16384 1 2024-06-26T07:12:28.4418061Z ipt_MASQUERADE 16384 1 2024-06-26T07:12:28.4418665Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:12:28.4419289Z nf_conntrack_netlink 49152 0 2024-06-26T07:12:28.4419870Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:12:28.4420491Z xfrm_user 45056 1 2024-06-26T07:12:28.4420952Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:12:28.4421511Z iptable_nat 16384 1 2024-06-26T07:12:28.4422108Z nf_conntrack_ipv4 16384 3 2024-06-26T07:12:28.4422644Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:12:28.4423287Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:12:28.4424012Z nf_nat 32768 3 nf_nat_masquerade_ipv4,nf_nat_ipv4,xt_nat 2024-06-26T07:12:28.4425481Z nf_conntrack 155648 8 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink 2024-06-26T07:12:28.4426852Z xt_addrtype 16384 2 2024-06-26T07:12:28.4427427Z iptable_filter 16384 1 2024-06-26T07:12:28.4427951Z br_netfilter 24576 0 2024-06-26T07:12:28.4428376Z bridge 172032 1 br_netfilter 2024-06-26T07:12:28.4428831Z stp 16384 1 bridge 2024-06-26T07:12:28.4429257Z llc 16384 2 bridge,stp 2024-06-26T07:12:28.4429686Z overlay 86016 0 2024-06-26T07:12:28.4430067Z sunrpc 393216 1 2024-06-26T07:12:28.4430459Z dm_mirror 28672 0 2024-06-26T07:12:28.4430859Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:12:28.4431677Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:12:28.4432376Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:12:28.4432914Z dax 69632 1 dm_mod 2024-06-26T07:12:28.4433351Z crc32_pclmul 16384 0 2024-06-26T07:12:28.4433758Z ghash_clmulni_intel 16384 0 2024-06-26T07:12:28.4434144Z pcbc 16384 0 2024-06-26T07:12:28.4434514Z aesni_intel 188416 0 2024-06-26T07:12:28.4434920Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:12:28.4435388Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:12:28.4435895Z glue_helper 16384 1 aesni_intel 2024-06-26T07:12:28.4436600Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:12:28.4437153Z mousedev 24576 0 2024-06-26T07:12:28.4437537Z evdev 20480 3 2024-06-26T07:12:28.4437920Z psmouse 32768 0 2024-06-26T07:12:28.4438290Z button 16384 0 2024-06-26T07:12:28.4438662Z ena 139264 0 2024-06-26T07:12:28.4439043Z ptp 20480 1 ena 2024-06-26T07:12:28.4439446Z pps_core 20480 1 ptp 2024-06-26T07:12:28.4439846Z crc32c_intel 24576 0 2024-06-26T07:12:28.4440219Z autofs4 49152 2 2024-06-26T07:12:28.4440585Z + modinfo nvidia 2024-06-26T07:12:28.4441278Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:12:28.4441986Z alias: char-major-195-* 2024-06-26T07:12:28.4442373Z version: 550.54.15 2024-06-26T07:12:28.4442717Z supported: external 2024-06-26T07:12:28.4443052Z license: NVIDIA 2024-06-26T07:12:28.4443434Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:12:28.4443928Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:12:28.4444394Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:12:28.4444860Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:12:28.4445365Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:12:28.4446040Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:12:28.4446530Z depends: i2c-core,drm 2024-06-26T07:12:28.4446893Z retpoline: Y 2024-06-26T07:12:28.4447201Z name: nvidia 2024-06-26T07:12:28.4447759Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:12:28.4448416Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:12:28.4449072Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:12:28.4449678Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:12:28.4450127Z parm: NVreg_RmLogonRC:int 2024-06-26T07:12:28.4450558Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:12:28.4451018Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:12:28.4451458Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:12:28.4451908Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:12:28.4452435Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:12:28.4453015Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:12:28.4453506Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:12:28.4453940Z parm: NVreg_EnableMSI:int 2024-06-26T07:12:28.4454349Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:12:28.4454810Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:12:28.4455343Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:12:28.4455924Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:12:28.4456498Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:12:28.4457114Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:12:28.4457728Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:12:28.4458339Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:12:28.4458937Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:12:28.4459446Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:12:28.4460087Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:12:28.4460633Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:12:28.4461133Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:12:28.4461601Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:12:28.4462087Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:12:28.4462554Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:12:28.4463007Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:12:28.4463518Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:12:28.4464051Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:12:28.4464526Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:12:28.4465019Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:12:28.4465529Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:12:28.4466022Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:12:28.4466522Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:12:28.4467013Z parm: NVreg_RmMsg:charp 2024-06-26T07:12:28.4467427Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:12:28.4467903Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:12:28.4468379Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:12:28.4468842Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:12:28.4469325Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:12:28.4469800Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:12:28.4470279Z parm: rm_firmware_active:charp 2024-06-26T07:12:28.4470709Z + HAS_NVIDIA_DRIVER=0 2024-06-26T07:12:28.4471419Z ++ command -v nvidia-smi 2024-06-26T07:12:28.4471889Z + '[' -x /usr/bin/nvidia-smi ']' 2024-06-26T07:12:28.4472260Z + set +e 2024-06-26T07:12:28.4472766Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-06-26T07:12:30.4264937Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-06-26T07:12:30.4265640Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:12:30.4266613Z + '[' 0 -ne 0 ']' 2024-06-26T07:12:30.4267220Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-06-26T07:12:30.4267617Z + HAS_NVIDIA_DRIVER=1 2024-06-26T07:12:30.4268335Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-06-26T07:12:30.4269030Z + set -e 2024-06-26T07:12:30.4269329Z + '[' 1 -eq 0 ']' 2024-06-26T07:12:30.4269907Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-06-26T07:12:30.4270593Z + post_install_nvidia_driver_common 2024-06-26T07:12:30.4270997Z + sudo modprobe nvidia 2024-06-26T07:12:30.4365023Z + echo 'After installing NVIDIA driver' 2024-06-26T07:12:30.4365602Z + lspci 2024-06-26T07:12:30.4366010Z After installing NVIDIA driver 2024-06-26T07:12:30.4448202Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:12:30.4449229Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:12:30.4450175Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:12:30.4450978Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:12:30.4451724Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:12:30.4452483Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:12:30.4453208Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:12:30.4453988Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:12:30.4454569Z + lsmod 2024-06-26T07:12:30.4462378Z Module Size Used by 2024-06-26T07:12:30.4463033Z xt_nat 16384 0 2024-06-26T07:12:30.4463544Z ib_core 266240 0 2024-06-26T07:12:30.4464030Z nvidia_modeset 1339392 0 2024-06-26T07:12:30.4464522Z veth 16384 0 2024-06-26T07:12:30.4464898Z nvidia_uvm 4599808 0 2024-06-26T07:12:30.4465358Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:12:30.4466066Z drm 421888 1 nvidia 2024-06-26T07:12:30.4466492Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:12:30.4466958Z backlight 16384 1 nvidia_modeset 2024-06-26T07:12:30.4467396Z xt_conntrack 16384 1 2024-06-26T07:12:30.4467776Z ipt_MASQUERADE 16384 1 2024-06-26T07:12:30.4468212Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:12:30.4468680Z nf_conntrack_netlink 49152 0 2024-06-26T07:12:30.4469141Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:12:30.4469774Z xfrm_user 45056 1 2024-06-26T07:12:30.4470345Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:12:30.4470892Z iptable_nat 16384 1 2024-06-26T07:12:30.4471612Z nf_conntrack_ipv4 16384 3 2024-06-26T07:12:30.4472194Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:12:30.4472839Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:12:30.4473571Z nf_nat 32768 3 nf_nat_masquerade_ipv4,nf_nat_ipv4,xt_nat 2024-06-26T07:12:30.4474948Z nf_conntrack 155648 8 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink 2024-06-26T07:12:30.4476181Z xt_addrtype 16384 2 2024-06-26T07:12:30.4476714Z iptable_filter 16384 1 2024-06-26T07:12:30.4477224Z br_netfilter 24576 0 2024-06-26T07:12:30.4477766Z bridge 172032 1 br_netfilter 2024-06-26T07:12:30.4478362Z stp 16384 1 bridge 2024-06-26T07:12:30.4478918Z llc 16384 2 bridge,stp 2024-06-26T07:12:30.4479344Z overlay 86016 0 2024-06-26T07:12:30.4479714Z sunrpc 393216 1 2024-06-26T07:12:30.4480138Z dm_mirror 28672 0 2024-06-26T07:12:30.4480540Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:12:30.4481040Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:12:30.4481703Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:12:30.4482175Z dax 69632 1 dm_mod 2024-06-26T07:12:30.4482588Z crc32_pclmul 16384 0 2024-06-26T07:12:30.4482978Z ghash_clmulni_intel 16384 0 2024-06-26T07:12:30.4483360Z pcbc 16384 0 2024-06-26T07:12:30.4483736Z aesni_intel 188416 0 2024-06-26T07:12:30.4484141Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:12:30.4484591Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:12:30.4485047Z glue_helper 16384 1 aesni_intel 2024-06-26T07:12:30.4485609Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:12:30.4486157Z mousedev 24576 0 2024-06-26T07:12:30.4486526Z evdev 20480 3 2024-06-26T07:12:30.4486901Z psmouse 32768 0 2024-06-26T07:12:30.4487276Z button 16384 0 2024-06-26T07:12:30.4487643Z ena 139264 0 2024-06-26T07:12:30.4488017Z ptp 20480 1 ena 2024-06-26T07:12:30.4488423Z pps_core 20480 1 ptp 2024-06-26T07:12:30.4488829Z crc32c_intel 24576 0 2024-06-26T07:12:30.4489198Z autofs4 49152 2 2024-06-26T07:12:30.4489562Z + modinfo nvidia 2024-06-26T07:12:30.4490365Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:12:30.4491081Z alias: char-major-195-* 2024-06-26T07:12:30.4491466Z version: 550.54.15 2024-06-26T07:12:30.4491809Z supported: external 2024-06-26T07:12:30.4492156Z license: NVIDIA 2024-06-26T07:12:30.4492538Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:12:30.4493030Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:12:30.4493488Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:12:30.4493958Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:12:30.4494466Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:12:30.4494963Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:12:30.4495539Z depends: i2c-core,drm 2024-06-26T07:12:30.4495902Z retpoline: Y 2024-06-26T07:12:30.4496213Z name: nvidia 2024-06-26T07:12:30.4496763Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:12:30.4497434Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:12:30.4498099Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:12:30.4498703Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:12:30.4499148Z parm: NVreg_RmLogonRC:int 2024-06-26T07:12:30.4499587Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:12:30.4500049Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:12:30.4500485Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:12:30.4500922Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:12:30.4501459Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:12:30.4502033Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:12:30.4502524Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:12:30.4502955Z parm: NVreg_EnableMSI:int 2024-06-26T07:12:30.4503374Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:12:30.4503835Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:12:30.4504366Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:12:30.4504957Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:12:30.4505526Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:12:30.4506144Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:12:30.4506744Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:12:30.4507365Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:12:30.4507966Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:12:30.4508462Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:12:30.4509062Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:12:30.4509626Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:12:30.4510133Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:12:30.4510612Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:12:30.4511305Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:12:30.4511850Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:12:30.4512305Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:12:30.4512814Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:12:30.4513341Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:12:30.4513825Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:12:30.4514318Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:12:30.4514822Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:12:30.4515305Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:12:30.4515818Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:12:30.4516413Z parm: NVreg_RmMsg:charp 2024-06-26T07:12:30.4516829Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:12:30.4517305Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:12:30.4517781Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:12:30.4518246Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:12:30.4518723Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:12:30.4519204Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:12:30.4519662Z parm: rm_firmware_active:charp 2024-06-26T07:12:30.4520071Z + set +e 2024-06-26T07:12:30.4520369Z + nvidia-smi 2024-06-26T07:12:32.0038652Z Wed Jun 26 07:12:31 2024 2024-06-26T07:12:32.0039609Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:12:32.0040481Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-06-26T07:12:32.0041290Z |-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:12:32.0042345Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-06-26T07:12:32.0043233Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-06-26T07:12:32.0043933Z | | | MIG M. | 2024-06-26T07:12:32.0044486Z |=========================================+========================+======================| 2024-06-26T07:12:32.0134057Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-06-26T07:12:32.0135004Z | 0% 33C P0 57W / 300W | 0MiB / 23028MiB | 0% Default | 2024-06-26T07:12:32.0135834Z | | | N/A | 2024-06-26T07:12:32.0136697Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:12:32.0137312Z 2024-06-26T07:12:32.0138104Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:12:32.0138897Z | Processes: | 2024-06-26T07:12:32.0139607Z | GPU GI CI PID Type Process name GPU Memory | 2024-06-26T07:12:32.0140317Z | ID ID Usage | 2024-06-26T07:12:32.0140877Z |=========================================================================================| 2024-06-26T07:12:32.0141535Z | No running processes found | 2024-06-26T07:12:32.0142294Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:12:32.6353359Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-06-26T07:12:34.1920453Z NVIDIA A10G 2024-06-26T07:12:34.6476248Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:12:34.6476855Z + '[' 0 -eq 0 ']' 2024-06-26T07:12:34.6477259Z + echo 'INFO: Ignoring allowed status 0' 2024-06-26T07:12:34.6477678Z + set -e 2024-06-26T07:12:34.6477975Z INFO: Ignoring allowed status 0 2024-06-26T07:12:34.6480373Z == Installing nvidia container toolkit for amzn2 == 2024-06-26T07:12:34.6482571Z + sudo yum install -y yum-utils 2024-06-26T07:12:34.9483107Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:12:36.5915049Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-06-26T07:12:36.5915770Z Nothing to do 2024-06-26T07:12:36.7442757Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]] 2024-06-26T07:12:36.7443769Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:12:36.7461321Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:12:37.0661046Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:12:37.0916674Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:12:37.0918358Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:12:37.0919298Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:12:37.1033016Z + sudo yum install -y nvidia-docker2 2024-06-26T07:12:37.4031750Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:12:38.8594118Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-06-26T07:12:38.8594862Z Nothing to do 2024-06-26T07:12:39.0103344Z + sudo systemctl restart docker 2024-06-26T07:13:15.0914077Z Command completed after 1 attempt(s). 2024-06-26T07:13:15.0973127Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:13:15.0973913Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:13:15.0974737Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-06-26T07:13:15.0975406Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:13:15.0983268Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:13:15.0983771Z env: 2024-06-26T07:13:15.0984055Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:15.0984508Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:15.0984989Z ##[endgroup] 2024-06-26T07:13:15.3206071Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:13:15.3355846Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-06-26T07:13:15.3454766Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-06-26T07:13:15.4377851Z Prepare all required actions 2024-06-26T07:13:15.4378358Z Getting action download info 2024-06-26T07:13:15.6279312Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-06-26T07:13:15.9315043Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-06-26T07:13:16.0621846Z ##[group]Run ./.github/actions/download-build-artifacts 2024-06-26T07:13:16.0622329Z with: 2024-06-26T07:13:16.0622666Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:13:16.0623126Z s3-bucket: gha-artifacts 2024-06-26T07:13:16.0623466Z env: 2024-06-26T07:13:16.0623735Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:16.0624197Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:16.0624684Z ##[endgroup] 2024-06-26T07:13:16.9030981Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:13:16.9031784Z with: 2024-06-26T07:13:16.9032173Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:13:16.9032632Z s3-bucket: gha-artifacts 2024-06-26T07:13:16.9033005Z region: us-east-1 2024-06-26T07:13:16.9033338Z env: 2024-06-26T07:13:16.9033622Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:16.9034066Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:16.9034557Z ##[endgroup] 2024-06-26T07:13:17.3244799Z (node:14203) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:13:17.3245534Z 2024-06-26T07:13:17.3245839Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:13:17.3246832Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:13:17.3248065Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:13:17.3952311Z Found 1 objects with prefix pytorch/pytorch/9673646806/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-06-26T07:13:17.3953411Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:13:27.8850483Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:13:27.8856390Z Artifact download has finished successfully 2024-06-26T07:13:27.9004508Z ##[group]Run unzip -o artifacts.zip 2024-06-26T07:13:27.9004951Z unzip -o artifacts.zip 2024-06-26T07:13:27.9012680Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:13:27.9013189Z env: 2024-06-26T07:13:27.9013471Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:27.9013928Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:27.9014413Z ##[endgroup] 2024-06-26T07:13:27.9043338Z Archive: artifacts.zip 2024-06-26T07:13:27.9045243Z creating: dist/ 2024-06-26T07:13:29.8755075Z inflating: dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:13:29.8755792Z creating: build/custom_test_artifacts/ 2024-06-26T07:13:29.8756496Z creating: build/custom_test_artifacts/custom-op-build/ 2024-06-26T07:13:29.8757243Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-06-26T07:13:29.8758273Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:13:29.8762796Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:13:29.8763738Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-06-26T07:13:29.8764680Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:13:29.8765678Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:13:29.8766647Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:13:29.8767753Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:13:29.8769093Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:13:29.8770110Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:13:29.8771121Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:13:29.8772274Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:13:29.8773450Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:13:29.8774604Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:13:29.8775753Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:13:29.8776928Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:13:29.8778101Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:13:29.8779152Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:13:29.8780222Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:13:29.8817184Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:13:29.8855140Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:13:29.8856580Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:13:29.8899346Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:13:29.8900766Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:13:29.8902206Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:13:29.8903694Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:13:29.8905116Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:13:29.8906508Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:13:29.8907917Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:13:29.8909323Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:13:29.8910692Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:13:29.8912349Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:13:29.8913582Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:13:29.8914814Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:13:29.8916155Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:13:29.8917360Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:13:29.8918582Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:13:29.8969408Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:13:29.9027329Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:13:29.9028526Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:13:29.9029525Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:13:29.9030577Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:13:29.9031759Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-06-26T07:13:29.9032712Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-06-26T07:13:29.9033745Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-06-26T07:13:29.9034908Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-06-26T07:13:29.9036120Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-06-26T07:13:29.9037169Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-06-26T07:13:29.9038256Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-06-26T07:13:29.9039340Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-06-26T07:13:29.9040425Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-06-26T07:13:29.9041495Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-06-26T07:13:29.9042562Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-06-26T07:13:29.9055003Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-06-26T07:13:29.9183354Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-06-26T07:13:29.9185729Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-06-26T07:13:29.9188237Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-06-26T07:13:29.9190105Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-06-26T07:13:29.9191551Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-06-26T07:13:29.9192666Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-06-26T07:13:29.9193795Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-06-26T07:13:29.9194942Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-06-26T07:13:29.9196181Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-06-26T07:13:29.9197473Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-06-26T07:13:29.9198592Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-06-26T07:13:29.9206129Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-06-26T07:13:29.9285239Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-06-26T07:13:29.9286778Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:13:29.9288143Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:13:29.9289250Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-06-26T07:13:29.9290460Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-06-26T07:13:29.9291654Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-06-26T07:13:29.9292574Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-06-26T07:13:29.9293431Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-06-26T07:13:29.9294217Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-06-26T07:13:29.9295026Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-06-26T07:13:29.9399924Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-06-26T07:13:29.9460211Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-06-26T07:13:29.9461134Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-06-26T07:13:29.9461832Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-06-26T07:13:29.9462650Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:13:29.9467687Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:13:29.9468773Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-06-26T07:13:29.9469763Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:13:29.9471012Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:13:29.9472398Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:13:29.9473504Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:13:29.9474841Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:13:29.9476155Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:13:29.9477267Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:13:29.9478637Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:13:29.9480139Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:13:29.9481557Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:13:29.9482683Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:13:29.9483850Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:13:29.9485005Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:13:29.9486039Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:13:29.9487184Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:13:29.9520583Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:13:29.9558396Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:13:29.9559957Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:13:29.9602593Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:13:29.9604566Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:13:29.9606580Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:13:29.9608619Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:13:29.9610316Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:13:29.9611764Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:13:29.9613140Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:13:29.9614495Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:13:29.9615825Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:13:29.9617086Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:13:29.9618302Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:13:29.9619534Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:13:29.9620745Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:13:29.9621927Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:13:29.9623120Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:13:29.9668801Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:13:29.9726463Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:13:29.9728102Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:13:29.9729496Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:13:29.9730621Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:13:29.9731866Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-06-26T07:13:29.9732954Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-06-26T07:13:29.9734029Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-06-26T07:13:29.9735210Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-06-26T07:13:29.9736360Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-06-26T07:13:29.9737550Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-06-26T07:13:29.9738650Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-06-26T07:13:29.9739760Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-06-26T07:13:29.9740869Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-06-26T07:13:29.9741965Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-06-26T07:13:29.9743054Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-06-26T07:13:29.9753575Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-06-26T07:13:29.9814342Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-06-26T07:13:29.9815574Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:13:29.9816648Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:13:29.9817613Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-06-26T07:13:29.9818520Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-06-26T07:13:29.9819415Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-06-26T07:13:29.9820304Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-06-26T07:13:29.9821141Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-06-26T07:13:29.9821913Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-06-26T07:13:29.9822721Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-06-26T07:13:29.9869062Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-06-26T07:13:29.9869821Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-06-26T07:13:29.9870574Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-06-26T07:13:29.9871661Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:13:29.9877037Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:13:29.9878134Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-06-26T07:13:29.9894461Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:13:29.9895711Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:13:29.9896839Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:13:29.9898029Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:13:29.9899195Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:13:29.9900253Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:13:29.9901300Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:13:29.9902499Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:13:29.9903711Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:13:29.9904893Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:13:29.9906079Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:13:29.9907461Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:13:29.9908668Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:13:29.9909750Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:13:29.9910803Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:13:29.9928386Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:13:29.9966131Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:13:29.9967736Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:13:30.0010488Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:13:30.0011977Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:13:30.0013477Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:13:30.0015005Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:13:30.0016487Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:13:30.0017953Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:13:30.0019474Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:13:30.0020929Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:13:30.0022342Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:13:30.0023701Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:13:30.0024992Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:13:30.0026277Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:13:30.0027567Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:13:30.0028830Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:13:30.0030095Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:13:30.0080896Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:13:30.0138796Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:13:30.0141229Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:13:30.0143143Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:13:30.0144806Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:13:30.0146560Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-06-26T07:13:30.0148628Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-06-26T07:13:30.0150073Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-06-26T07:13:30.0151479Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-06-26T07:13:30.0152682Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-06-26T07:13:30.0153818Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-06-26T07:13:30.0154992Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-06-26T07:13:30.0156331Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-06-26T07:13:30.0157526Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-06-26T07:13:30.0158705Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-06-26T07:13:30.0159917Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-06-26T07:13:30.0161162Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-06-26T07:13:30.0270493Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-06-26T07:13:30.0271888Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-06-26T07:13:30.0273072Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-06-26T07:13:30.0274385Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-06-26T07:13:30.0275650Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-06-26T07:13:30.0276907Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-06-26T07:13:30.0278146Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-06-26T07:13:30.0279373Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-06-26T07:13:30.0280606Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-06-26T07:13:30.0281839Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-06-26T07:13:30.0283063Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-06-26T07:13:30.0295772Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-06-26T07:13:30.0349456Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-06-26T07:13:30.0350752Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:13:30.0352013Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:13:30.0353042Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-06-26T07:13:30.0354001Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-06-26T07:13:30.0354962Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-06-26T07:13:30.0355918Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-06-26T07:13:30.0357072Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-06-26T07:13:30.0357918Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-06-26T07:13:30.0358773Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-06-26T07:13:30.0456662Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-06-26T07:13:30.0497944Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-06-26T07:13:30.0498584Z creating: build/lib/ 2024-06-26T07:13:30.0584145Z inflating: build/lib/libprotobuf-lite.a 2024-06-26T07:13:30.1021309Z inflating: build/lib/libprotobuf.a 2024-06-26T07:13:30.1030891Z inflating: build/lib/libpthreadpool.a 2024-06-26T07:13:30.1039147Z inflating: build/lib/libcpuinfo.a 2024-06-26T07:13:30.1046944Z inflating: build/lib/libcpuinfo_internals.a 2024-06-26T07:13:30.1048082Z inflating: build/lib/libclog.a 2024-06-26T07:13:30.1065791Z inflating: build/lib/libnnpack.a 2024-06-26T07:13:30.1068074Z inflating: build/lib/libnnpack_reference_layers.a 2024-06-26T07:13:30.1130786Z inflating: build/lib/libgtest.a 2024-06-26T07:13:30.1203126Z inflating: build/lib/libbenchmark.a 2024-06-26T07:13:30.1264959Z inflating: build/lib/libasmjit.a 2024-06-26T07:13:30.1272532Z inflating: build/lib/libittnotify.a 2024-06-26T07:13:30.1299417Z inflating: build/lib/libtensorpipe_uv.a 2024-06-26T07:13:30.1421992Z inflating: build/lib/libgloo.a 2024-06-26T07:13:30.1443523Z inflating: build/lib/libfmt.a 2024-06-26T07:13:30.1532501Z inflating: build/lib/libc10.so 2024-06-26T07:13:30.1534265Z inflating: build/lib/libcaffe2_nvrtc.so 2024-06-26T07:13:30.1534962Z inflating: build/lib/libfoxi_loader.a 2024-06-26T07:13:30.1536557Z inflating: build/lib/libtorch_global_deps.so 2024-06-26T07:13:30.1555774Z inflating: build/lib/libpytorch_qnnpack.a 2024-06-26T07:13:30.1573455Z inflating: build/lib/libgmock.a 2024-06-26T07:13:30.1574107Z inflating: build/lib/libgtest_main.a 2024-06-26T07:13:30.1574787Z inflating: build/lib/libbenchmark_main.a 2024-06-26T07:13:30.2059431Z inflating: build/lib/libprotoc.a 2024-06-26T07:13:30.2428901Z inflating: build/lib/libgloo_cuda.a 2024-06-26T07:13:31.2144758Z inflating: build/lib/libdnnl.a 2024-06-26T07:13:31.2693280Z inflating: build/lib/libtensorpipe.a 2024-06-26T07:13:31.2746480Z inflating: build/lib/libc10_cuda.so 2024-06-26T07:13:31.2747139Z inflating: build/lib/libgmock_main.a 2024-06-26T07:13:31.3950022Z inflating: build/lib/libfbgemm.a 2024-06-26T07:13:31.4195726Z inflating: build/lib/libtensorpipe_cuda.a 2024-06-26T07:13:31.4695358Z inflating: build/lib/libkineto.a 2024-06-26T07:13:31.4873094Z inflating: build/lib/libXNNPACK.a 2024-06-26T07:13:31.4914214Z inflating: build/lib/libonnx_proto.a 2024-06-26T07:13:31.5579088Z inflating: build/lib/libonnx.a 2024-06-26T07:13:33.8827758Z inflating: build/lib/libtorch_cpu.so 2024-06-26T07:13:33.8832396Z inflating: build/lib/libunbox_lib.a 2024-06-26T07:13:33.8836709Z inflating: build/lib/libshm.so 2024-06-26T07:13:35.8676378Z inflating: build/lib/libtorch_cuda.so 2024-06-26T07:13:35.8677105Z inflating: build/lib/libtorch.so 2024-06-26T07:13:36.6611696Z inflating: build/lib/libtorch_cuda_linalg.so 2024-06-26T07:13:36.6613688Z inflating: build/lib/libc10d_cuda_test.so 2024-06-26T07:13:36.8508523Z inflating: build/lib/libtorch_python.so 2024-06-26T07:13:36.8575055Z inflating: build/lib/libtorchbind_test.so 2024-06-26T07:13:36.8595581Z inflating: build/lib/libjitbackend_test.so 2024-06-26T07:13:36.8620409Z inflating: build/lib/libbackend_with_compiler.so 2024-06-26T07:13:36.8654377Z inflating: build/lib/libnnapi_backend.so 2024-06-26T07:13:36.8655058Z creating: build/bin/ 2024-06-26T07:13:36.8703692Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-06-26T07:13:36.8754206Z inflating: build/bin/c10_DeviceGuard_test 2024-06-26T07:13:36.8803154Z inflating: build/bin/c10_Device_test 2024-06-26T07:13:36.8860833Z inflating: build/bin/c10_DispatchKeySet_test 2024-06-26T07:13:36.8913169Z inflating: build/bin/c10_Scalar_test 2024-06-26T07:13:36.8960991Z inflating: build/bin/c10_StreamGuard_test 2024-06-26T07:13:36.9013950Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-06-26T07:13:36.9063800Z inflating: build/bin/c10_SymInt_test 2024-06-26T07:13:36.9118366Z inflating: build/bin/c10_InlineStreamGuard_test 2024-06-26T07:13:36.9173550Z inflating: build/bin/c10_SizesAndStrides_test 2024-06-26T07:13:36.9225126Z inflating: build/bin/c10_Bitset_test 2024-06-26T07:13:36.9294461Z inflating: build/bin/c10_cow_test 2024-06-26T07:13:36.9342276Z inflating: build/bin/c10_ConstexprCrc_test 2024-06-26T07:13:36.9391482Z inflating: build/bin/c10_DeadlockDetection_test 2024-06-26T07:13:36.9440852Z inflating: build/bin/c10_Half_test 2024-06-26T07:13:36.9495655Z inflating: build/bin/c10_LeftRight_test 2024-06-26T07:13:36.9549678Z inflating: build/bin/c10_Metaprogramming_test 2024-06-26T07:13:36.9598751Z inflating: build/bin/c10_Synchronized_test 2024-06-26T07:13:36.9653374Z inflating: build/bin/c10_ThreadLocal_test 2024-06-26T07:13:36.9703719Z inflating: build/bin/c10_TypeIndex_test 2024-06-26T07:13:36.9753741Z inflating: build/bin/c10_TypeList_test 2024-06-26T07:13:36.9801638Z inflating: build/bin/c10_TypeTraits_test 2024-06-26T07:13:36.9852166Z inflating: build/bin/c10_accumulate_test 2024-06-26T07:13:36.9906790Z inflating: build/bin/c10_bfloat16_test 2024-06-26T07:13:36.9956525Z inflating: build/bin/c10_bit_cast_test 2024-06-26T07:13:37.0011476Z inflating: build/bin/c10_complex_math_test 2024-06-26T07:13:37.0065847Z inflating: build/bin/c10_complex_test 2024-06-26T07:13:37.0118051Z inflating: build/bin/c10_exception_test 2024-06-26T07:13:37.0167146Z inflating: build/bin/c10_flags_test 2024-06-26T07:13:37.0216053Z inflating: build/bin/c10_generic_math_test 2024-06-26T07:13:37.0374766Z inflating: build/bin/c10_intrusive_ptr_test 2024-06-26T07:13:37.0424410Z inflating: build/bin/c10_irange_test 2024-06-26T07:13:37.0476970Z inflating: build/bin/c10_lazy_test 2024-06-26T07:13:37.0532869Z inflating: build/bin/c10_logging_test 2024-06-26T07:13:37.0605828Z inflating: build/bin/c10_optional_test 2024-06-26T07:13:37.0666469Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-06-26T07:13:37.0719319Z inflating: build/bin/c10_registry_test 2024-06-26T07:13:37.0865592Z inflating: build/bin/c10_small_vector_test 2024-06-26T07:13:37.0915913Z inflating: build/bin/c10_ssize_test 2024-06-26T07:13:37.0966995Z inflating: build/bin/c10_string_util_test 2024-06-26T07:13:37.1024428Z inflating: build/bin/c10_string_view_test 2024-06-26T07:13:37.1073776Z inflating: build/bin/c10_tempfile_test 2024-06-26T07:13:37.1121414Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-06-26T07:13:37.1175973Z inflating: build/bin/c10_typeid_test 2024-06-26T07:13:37.1609756Z inflating: build/bin/protoc-3.13.0.0 2024-06-26T07:13:37.2042716Z inflating: build/bin/protoc 2024-06-26T07:13:37.2094413Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-06-26T07:13:37.2146367Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-06-26T07:13:37.2198097Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-06-26T07:13:37.2248665Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-06-26T07:13:37.2300451Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-06-26T07:13:37.2352313Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-06-26T07:13:37.2400459Z inflating: build/bin/c10_cuda_CUDATest 2024-06-26T07:13:37.2451853Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-06-26T07:13:37.2771962Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-06-26T07:13:37.3105064Z inflating: build/bin/vec_test_all_types_AVX512 2024-06-26T07:13:37.3451176Z inflating: build/bin/vec_test_all_types_AVX2 2024-06-26T07:13:37.3502998Z inflating: build/bin/FileStoreTest 2024-06-26T07:13:37.3557259Z inflating: build/bin/TCPStoreTest 2024-06-26T07:13:37.3609377Z inflating: build/bin/HashStoreTest 2024-06-26T07:13:37.3622795Z inflating: build/bin/ProcessGroupMPITest 2024-06-26T07:13:37.3675694Z inflating: build/bin/test_edge_op_registration 2024-06-26T07:13:37.3680213Z inflating: build/bin/torch_shm_manager 2024-06-26T07:13:37.3683306Z inflating: build/bin/example_allreduce 2024-06-26T07:13:37.3737268Z inflating: build/bin/test_dist_autograd 2024-06-26T07:13:37.3805638Z inflating: build/bin/test_cpp_rpc 2024-06-26T07:13:37.3808111Z inflating: build/bin/parallel_benchmark 2024-06-26T07:13:37.3873675Z inflating: build/bin/test_mobile_nnc 2024-06-26T07:13:37.3882825Z inflating: build/bin/aot_model_compiler_test 2024-06-26T07:13:37.4216285Z inflating: build/bin/test_lazy 2024-06-26T07:13:37.5360944Z inflating: build/bin/test_api 2024-06-26T07:13:37.5432744Z inflating: build/bin/Dict_test 2024-06-26T07:13:37.5483706Z inflating: build/bin/Dimname_test 2024-06-26T07:13:37.5546787Z inflating: build/bin/MaybeOwned_test 2024-06-26T07:13:37.5603040Z inflating: build/bin/NamedTensor_test 2024-06-26T07:13:37.5660731Z inflating: build/bin/apply_utils_test 2024-06-26T07:13:37.5718469Z inflating: build/bin/atest 2024-06-26T07:13:37.5779945Z inflating: build/bin/basic 2024-06-26T07:13:37.5833927Z inflating: build/bin/broadcast_test 2024-06-26T07:13:37.5883472Z inflating: build/bin/cpu_allocator_test 2024-06-26T07:13:37.5940320Z inflating: build/bin/cpu_generator_test 2024-06-26T07:13:37.5992488Z inflating: build/bin/cpu_profiling_allocator_test 2024-06-26T07:13:37.6083283Z inflating: build/bin/cpu_rng_test 2024-06-26T07:13:37.6132225Z inflating: build/bin/dispatch_key_set_test 2024-06-26T07:13:37.6181754Z inflating: build/bin/dlconvertor_test 2024-06-26T07:13:37.6238691Z inflating: build/bin/extension_backend_test 2024-06-26T07:13:37.6292514Z inflating: build/bin/half_test 2024-06-26T07:13:37.6386155Z inflating: build/bin/ivalue_test 2024-06-26T07:13:37.6434945Z inflating: build/bin/lazy_tensor_test 2024-06-26T07:13:37.6488064Z inflating: build/bin/math_kernel_test 2024-06-26T07:13:37.6541200Z inflating: build/bin/memory_format_test 2024-06-26T07:13:37.6592966Z inflating: build/bin/memory_overlapping_test 2024-06-26T07:13:37.6644796Z inflating: build/bin/mobile_memory_cleanup 2024-06-26T07:13:37.6699579Z inflating: build/bin/native_test 2024-06-26T07:13:37.6749145Z inflating: build/bin/operator_name_test 2024-06-26T07:13:37.6799597Z inflating: build/bin/operators_test 2024-06-26T07:13:37.6851452Z inflating: build/bin/packedtensoraccessor_test 2024-06-26T07:13:37.6917092Z inflating: build/bin/pow_test 2024-06-26T07:13:37.6972687Z inflating: build/bin/quantized_test 2024-06-26T07:13:37.7021766Z inflating: build/bin/reduce_ops_test 2024-06-26T07:13:37.7071873Z inflating: build/bin/reportMemoryUsage_test 2024-06-26T07:13:37.7126672Z inflating: build/bin/scalar_tensor_test 2024-06-26T07:13:37.7183590Z inflating: build/bin/scalar_test 2024-06-26T07:13:37.7234576Z inflating: build/bin/StorageUtils_test 2024-06-26T07:13:37.7285283Z inflating: build/bin/stride_properties_test 2024-06-26T07:13:37.7361780Z inflating: build/bin/tensor_iterator_test 2024-06-26T07:13:37.7414293Z inflating: build/bin/test_parallel 2024-06-26T07:13:37.7417226Z inflating: build/bin/thread_init_test 2024-06-26T07:13:37.7471666Z inflating: build/bin/type_ptr_test 2024-06-26T07:13:37.7529868Z inflating: build/bin/type_test 2024-06-26T07:13:37.7581070Z inflating: build/bin/undefined_tensor_test 2024-06-26T07:13:37.7582001Z inflating: build/bin/verify_api_visibility 2024-06-26T07:13:37.7649605Z inflating: build/bin/legacy_vmap_test 2024-06-26T07:13:37.7699854Z inflating: build/bin/weakref_test 2024-06-26T07:13:37.7749947Z inflating: build/bin/wrapdim_test 2024-06-26T07:13:37.7800453Z inflating: build/bin/xla_tensor_test 2024-06-26T07:13:37.7859956Z inflating: build/bin/IListRef_test 2024-06-26T07:13:37.7963006Z inflating: build/bin/List_test 2024-06-26T07:13:37.8027751Z inflating: build/bin/KernelFunction_test 2024-06-26T07:13:37.8146768Z inflating: build/bin/kernel_function_legacy_test 2024-06-26T07:13:37.8240842Z inflating: build/bin/kernel_function_test 2024-06-26T07:13:37.8364500Z inflating: build/bin/kernel_lambda_legacy_test 2024-06-26T07:13:37.8464762Z inflating: build/bin/kernel_lambda_test 2024-06-26T07:13:37.8524016Z inflating: build/bin/kernel_stackbased_test 2024-06-26T07:13:37.8617479Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-06-26T07:13:37.8667592Z inflating: build/bin/CppSignature_test 2024-06-26T07:13:37.8721273Z inflating: build/bin/backend_fallback_test 2024-06-26T07:13:37.8768759Z inflating: build/bin/op_allowlist_test 2024-06-26T07:13:37.9060935Z inflating: build/bin/op_registration_test 2024-06-26T07:13:37.9122170Z inflating: build/bin/inline_container_test 2024-06-26T07:13:37.9173517Z inflating: build/bin/cuda_apply_test 2024-06-26T07:13:37.9222912Z inflating: build/bin/cuda_allocator_test 2024-06-26T07:13:37.9280625Z inflating: build/bin/cuda_atomic_ops_test 2024-06-26T07:13:37.9334461Z inflating: build/bin/cuda_caching_host_allocator_test 2024-06-26T07:13:37.9402008Z inflating: build/bin/cuda_complex_math_test 2024-06-26T07:13:37.9459291Z inflating: build/bin/cuda_complex_test 2024-06-26T07:13:37.9507528Z inflating: build/bin/cuda_device_test 2024-06-26T07:13:37.9563457Z inflating: build/bin/cuda_cub_test 2024-06-26T07:13:37.9613118Z inflating: build/bin/cuda_dlconvertor_test 2024-06-26T07:13:37.9676364Z inflating: build/bin/cuda_distributions_test 2024-06-26T07:13:37.9731555Z inflating: build/bin/cuda_generator_test 2024-06-26T07:13:37.9779922Z inflating: build/bin/cuda_half_test 2024-06-26T07:13:37.9829744Z inflating: build/bin/cuda_integer_divider_test 2024-06-26T07:13:37.9877692Z inflating: build/bin/cuda_optional_test 2024-06-26T07:13:37.9928211Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-06-26T07:13:37.9979471Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-06-26T07:13:38.0028105Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-06-26T07:13:38.0087269Z inflating: build/bin/cuda_stream_test 2024-06-26T07:13:38.0135200Z inflating: build/bin/cuda_cudnn_test 2024-06-26T07:13:38.0186024Z inflating: build/bin/cuda_vectorized_test 2024-06-26T07:13:38.0200825Z inflating: build/bin/tutorial_tensorexpr 2024-06-26T07:13:38.0265366Z inflating: build/bin/ProcessGroupGlooTest 2024-06-26T07:13:38.0321338Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-06-26T07:13:38.0383224Z inflating: build/bin/ProcessGroupNCCLTest 2024-06-26T07:13:38.0443031Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-06-26T07:13:38.1248906Z inflating: build/bin/test_tensorexpr 2024-06-26T07:13:38.1807565Z inflating: build/bin/test_jit 2024-06-26T07:13:38.1808158Z creating: .additional_ci_files/ 2024-06-26T07:13:38.1861489Z inflating: .additional_ci_files/test-times.json 2024-06-26T07:13:38.2072536Z inflating: .additional_ci_files/test-class-times.json 2024-06-26T07:13:38.2101026Z ##[group]Run rm artifacts.zip 2024-06-26T07:13:38.2101437Z rm artifacts.zip 2024-06-26T07:13:38.2109070Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:13:38.2109576Z env: 2024-06-26T07:13:38.2109859Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:38.2110316Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:38.2110936Z ##[endgroup] 2024-06-26T07:13:38.2808939Z ##[group]Run df -H 2024-06-26T07:13:38.2809266Z df -H 2024-06-26T07:13:38.2816880Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:13:38.2817375Z env: 2024-06-26T07:13:38.2817654Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:38.2818112Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:38.2818597Z ##[endgroup] 2024-06-26T07:13:38.2846187Z Filesystem Size Used Avail Use% Mounted on 2024-06-26T07:13:38.2846703Z devtmpfs 34G 0 34G 0% /dev 2024-06-26T07:13:38.2847174Z tmpfs 34G 0 34G 0% /dev/shm 2024-06-26T07:13:38.2847869Z tmpfs 34G 455k 34G 1% /run 2024-06-26T07:13:38.2848351Z tmpfs 34G 0 34G 0% /sys/fs/cgroup 2024-06-26T07:13:38.2848849Z /dev/nvme0n1p1 162G 50G 112G 31% / 2024-06-26T07:13:38.2886953Z Prepare all required actions 2024-06-26T07:13:38.2887403Z Getting action download info 2024-06-26T07:13:38.4023367Z ##[group]Run ./.github/actions/download-td-artifacts 2024-06-26T07:13:38.4023833Z with: 2024-06-26T07:13:38.4024090Z env: 2024-06-26T07:13:38.4024368Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:38.4024824Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:38.4025302Z ##[endgroup] 2024-06-26T07:13:38.4055804Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:13:38.4056254Z with: 2024-06-26T07:13:38.4056524Z name: td_results 2024-06-26T07:13:38.4056847Z s3-bucket: gha-artifacts 2024-06-26T07:13:38.4057203Z region: us-east-1 2024-06-26T07:13:38.4057504Z env: 2024-06-26T07:13:38.4057767Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:38.4058221Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:38.4058709Z ##[endgroup] 2024-06-26T07:13:38.8143559Z (node:14227) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:13:38.8144949Z 2024-06-26T07:13:38.8145491Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:13:38.8146926Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:13:38.8148680Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:13:38.8879374Z Found 0 objects with prefix pytorch/pytorch/9673646806/td_results/ 2024-06-26T07:13:38.8886473Z Artifact download has finished successfully 2024-06-26T07:13:38.9013755Z ##[group]Run mkdir -p .additional_ci_files 2024-06-26T07:13:38.9014251Z mkdir -p .additional_ci_files 2024-06-26T07:13:38.9014824Z mv td_results.json .additional_ci_files/td_results.json 2024-06-26T07:13:38.9022616Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:13:38.9023111Z env: 2024-06-26T07:13:38.9023392Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:38.9023855Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:38.9024341Z ##[endgroup] 2024-06-26T07:13:38.9077470Z mv: cannot stat 'td_results.json': No such file or directory 2024-06-26T07:13:38.9088234Z ##[error]Process completed with exit code 1. 2024-06-26T07:13:38.9119650Z ##[group]Run .github/scripts/parse_ref.py 2024-06-26T07:13:38.9120131Z .github/scripts/parse_ref.py 2024-06-26T07:13:38.9127125Z shell: /usr/bin/bash -e {0} 2024-06-26T07:13:38.9127465Z env: 2024-06-26T07:13:38.9127741Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:38.9128193Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:38.9128672Z ##[endgroup] 2024-06-26T07:13:38.9356914Z Prepare all required actions 2024-06-26T07:13:38.9393698Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-06-26T07:13:38.9394158Z with: 2024-06-26T07:13:38.9394735Z github-token: *** 2024-06-26T07:13:38.9395043Z env: 2024-06-26T07:13:38.9395314Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:38.9395770Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:38.9396340Z ##[endgroup] 2024-06-26T07:13:38.9411357Z ##[group]Run set -eux 2024-06-26T07:13:38.9411826Z set -eux 2024-06-26T07:13:38.9412412Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-06-26T07:13:38.9420215Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:13:38.9420712Z env: 2024-06-26T07:13:38.9420999Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:38.9421456Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:38.9422094Z GITHUB_TOKEN: *** 2024-06-26T07:13:38.9422406Z ##[endgroup] 2024-06-26T07:13:38.9444526Z + python3 .github/scripts/get_workflow_job_id.py 9673646806 i-0c9d022330e7e8039 2024-06-26T07:13:39.5065512Z setting job-id=26688915732 2024-06-26T07:13:39.5066595Z setting job-name=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:13:39.5276165Z Prepare all required actions 2024-06-26T07:13:39.5276617Z Getting action download info 2024-06-26T07:13:39.6271658Z ##[group]Run ./.github/actions/filter-test-configs 2024-06-26T07:13:39.6272122Z with: 2024-06-26T07:13:39.6272579Z github-token: *** 2024-06-26T07:13:39.6279191Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:13:39.6286743Z job-name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:13:39.6287648Z env: 2024-06-26T07:13:39.6287916Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:39.6288374Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:39.6288857Z ##[endgroup] 2024-06-26T07:13:39.6329434Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:13:39.6329999Z with: 2024-06-26T07:13:39.6330270Z shell: bash 2024-06-26T07:13:39.6330637Z timeout_minutes: 10 2024-06-26T07:13:39.6331016Z max_attempts: 5 2024-06-26T07:13:39.6331337Z retry_wait_seconds: 30 2024-06-26T07:13:39.6332454Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:13:39.6333619Z polling_interval_seconds: 1 2024-06-26T07:13:39.6333998Z warning_on_retry: true 2024-06-26T07:13:39.6334341Z continue_on_error: false 2024-06-26T07:13:39.6334684Z env: 2024-06-26T07:13:39.6334956Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:39.6335402Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:39.6336199Z GITHUB_TOKEN: *** 2024-06-26T07:13:39.6336511Z ##[endgroup] 2024-06-26T07:13:39.6837481Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:13:39.8984994Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:13:39.9129275Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1) 2024-06-26T07:13:39.9269988Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-06-26T07:13:39.9277872Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.6.2) 2024-06-26T07:13:39.9286247Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7) 2024-06-26T07:13:39.9298649Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12) 2024-06-26T07:13:39.9318541Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19) 2024-06-26T07:13:40.6841933Z Command completed after 1 attempt(s). 2024-06-26T07:13:40.6883268Z ##[group]Run set -x 2024-06-26T07:13:40.6883601Z set -x 2024-06-26T07:13:40.6883895Z  2024-06-26T07:13:40.6884447Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:13:40.6885118Z # in runner workspace 2024-06-26T07:13:40.6885646Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-06-26T07:13:40.6893355Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:13:40.6893855Z env: 2024-06-26T07:13:40.6894149Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:40.6894604Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:40.6895090Z ##[endgroup] 2024-06-26T07:13:40.6915566Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-06-26T07:13:40.7114400Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:13:40.7114961Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:13:40.7115434Z echo "Job name: ${JOB_NAME}" 2024-06-26T07:13:40.7115840Z  2024-06-26T07:13:40.7116513Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:13:40.7117204Z # in runner workspace 2024-06-26T07:13:40.7117785Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-06-26T07:13:40.7118434Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-06-26T07:13:40.7118905Z  --job-name "${JOB_NAME}" \ 2024-06-26T07:13:40.7125723Z  --test-matrix "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-06-26T07:13:40.7132856Z  --selected-test-configs "" \ 2024-06-26T07:13:40.7133328Z  --pr-number "${PR_NUMBER}" \ 2024-06-26T07:13:40.7133754Z  --tag "${TAG}" \ 2024-06-26T07:13:40.7134157Z  --event-name "${EVENT_NAME}" \ 2024-06-26T07:13:40.7134604Z  --schedule "${SCHEDULE}" \ 2024-06-26T07:13:40.7135132Z  --branch "${HEAD_BRANCH}" 2024-06-26T07:13:40.7142375Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:13:40.7142886Z env: 2024-06-26T07:13:40.7143163Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:40.7143623Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:40.7144300Z GITHUB_TOKEN: *** 2024-06-26T07:13:40.7145149Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:13:40.7158683Z PR_NUMBER: 2024-06-26T07:13:40.7159108Z TAG: ciflow/inductor/129470 2024-06-26T07:13:40.7159607Z EVENT_NAME: push 2024-06-26T07:13:40.7160011Z SCHEDULE: 2024-06-26T07:13:40.7160349Z HEAD_BRANCH: 2024-06-26T07:13:40.7160652Z ##[endgroup] 2024-06-26T07:13:40.7179728Z Workflow: inductor-periodic 2024-06-26T07:13:40.7180764Z Job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:13:40.9491831Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-06-26T07:13:41.0810191Z ##[group]Run echo "Filtered matrix:" 2024-06-26T07:13:41.0810667Z echo "Filtered matrix:" 2024-06-26T07:13:41.0817331Z echo "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-06-26T07:13:41.0824199Z  2024-06-26T07:13:41.0824479Z echo 2024-06-26T07:13:41.0824870Z echo "Is the current job unstable? False" 2024-06-26T07:13:41.0825331Z  2024-06-26T07:13:41.0825600Z echo 2024-06-26T07:13:41.0825964Z echo "Is keep-going label set? False" 2024-06-26T07:13:41.0826525Z  2024-06-26T07:13:41.0826796Z echo 2024-06-26T07:13:41.0827115Z echo "Renabled issues? " 2024-06-26T07:13:41.0834773Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:13:41.0835269Z env: 2024-06-26T07:13:41.0835540Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:41.0835998Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:41.0836578Z ##[endgroup] 2024-06-26T07:13:41.0856186Z Filtered matrix: 2024-06-26T07:13:41.0864772Z {include: [{config: dynamo_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-06-26T07:13:41.0871682Z 2024-06-26T07:13:41.0871835Z Is the current job unstable? False 2024-06-26T07:13:41.0872119Z 2024-06-26T07:13:41.0872395Z Is keep-going label set? False 2024-06-26T07:13:41.0872646Z 2024-06-26T07:13:41.0872774Z Renabled issues? 2024-06-26T07:13:41.0911679Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:13:41.0912390Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:13:41.0920173Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:13:41.0920671Z env: 2024-06-26T07:13:41.0920951Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:41.0921404Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:41.0921889Z JOB_TIMEOUT: 240 2024-06-26T07:13:41.0922188Z ##[endgroup] 2024-06-26T07:13:41.0994530Z ##[group]Run set -x 2024-06-26T07:13:41.0994923Z set -x 2024-06-26T07:13:41.0995225Z  2024-06-26T07:13:41.0995582Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-06-26T07:13:41.0996234Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-06-26T07:13:41.0996801Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-06-26T07:13:41.0997314Z  TEST_COMMAND=.ci/onnx/test.sh 2024-06-26T07:13:41.0997731Z else 2024-06-26T07:13:41.0998069Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:13:41.0998489Z fi 2024-06-26T07:13:41.0998769Z  2024-06-26T07:13:41.0999240Z # detached container should get cleaned up by teardown_ec2_linux 2024-06-26T07:13:41.0999989Z # TODO: Stop building test binaries as part of the build phase 2024-06-26T07:13:41.1000662Z # Used for GPU_FLAG since that doesn't play nice 2024-06-26T07:13:41.1001237Z # shellcheck disable=SC2086,SC2090 2024-06-26T07:13:41.1001736Z container_name=$(docker run \ 2024-06-26T07:13:41.1002169Z  ${GPU_FLAG:-} \ 2024-06-26T07:13:41.1002540Z  -e BUILD_ENVIRONMENT \ 2024-06-26T07:13:41.1002938Z  -e PR_NUMBER \ 2024-06-26T07:13:41.1003390Z  -e GITHUB_ACTIONS \ 2024-06-26T07:13:41.1003786Z  -e GITHUB_REPOSITORY \ 2024-06-26T07:13:41.1004193Z  -e GITHUB_WORKFLOW \ 2024-06-26T07:13:41.1004578Z  -e GITHUB_JOB \ 2024-06-26T07:13:41.1004934Z  -e GITHUB_RUN_ID \ 2024-06-26T07:13:41.1005320Z  -e GITHUB_RUN_NUMBER \ 2024-06-26T07:13:41.1005728Z  -e GITHUB_RUN_ATTEMPT \ 2024-06-26T07:13:41.1006120Z  -e JOB_ID \ 2024-06-26T07:13:41.1006463Z  -e JOB_NAME \ 2024-06-26T07:13:41.1006809Z  -e BASE_SHA \ 2024-06-26T07:13:41.1007153Z  -e BRANCH \ 2024-06-26T07:13:41.1007481Z  -e SHA1 \ 2024-06-26T07:13:41.1007827Z  -e AWS_DEFAULT_REGION \ 2024-06-26T07:13:41.1008237Z  -e IN_WHEEL_TEST \ 2024-06-26T07:13:41.1008623Z  -e SHARD_NUMBER \ 2024-06-26T07:13:41.1008989Z  -e TEST_CONFIG \ 2024-06-26T07:13:41.1009360Z  -e NUM_TEST_SHARDS \ 2024-06-26T07:13:41.1009760Z  -e REENABLED_ISSUES \ 2024-06-26T07:13:41.1010170Z  -e CONTINUE_THROUGH_ERROR \ 2024-06-26T07:13:41.1010591Z  -e VERBOSE_TEST_LOGS \ 2024-06-26T07:13:41.1010990Z  -e NO_TEST_TIMEOUT \ 2024-06-26T07:13:41.1011368Z  -e NO_TD \ 2024-06-26T07:13:41.1011703Z  -e TD_DISTRIBUTED \ 2024-06-26T07:13:41.1012084Z  -e PR_LABELS \ 2024-06-26T07:13:41.1012491Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-06-26T07:13:41.1012945Z  -e SCCACHE_BUCKET \ 2024-06-26T07:13:41.1013334Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-06-26T07:13:41.1013744Z  -e XLA_CUDA \ 2024-06-26T07:13:41.1014150Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-06-26T07:13:41.1014662Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-06-26T07:13:41.1015182Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-06-26T07:13:41.1015696Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-06-26T07:13:41.1016168Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-06-26T07:13:41.1016595Z  -e DASHBOARD_TAG \ 2024-06-26T07:13:41.1017067Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-06-26T07:13:41.1017626Z  --security-opt seccomp=unconfined \ 2024-06-26T07:13:41.1018092Z  --cap-add=SYS_PTRACE \ 2024-06-26T07:13:41.1018485Z  --ipc=host \ 2024-06-26T07:13:41.1018841Z  --shm-size="${SHM_SIZE}" \ 2024-06-26T07:13:41.1019243Z  --tty \ 2024-06-26T07:13:41.1019564Z  --detach \ 2024-06-26T07:13:41.1019923Z  --name="${container_name}" \ 2024-06-26T07:13:41.1020334Z  --user jenkins \ 2024-06-26T07:13:41.1020889Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-06-26T07:13:41.1021457Z  -w /var/lib/jenkins/workspace \ 2024-06-26T07:13:41.1021896Z  "${DOCKER_IMAGE}" 2024-06-26T07:13:41.1022243Z ) 2024-06-26T07:13:41.1022654Z # Propagate download.pytorch.org IP to container 2024-06-26T07:13:41.1023578Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-06-26T07:13:41.1024548Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-06-26T07:13:41.1025455Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-06-26T07:13:41.1032905Z shell: /usr/bin/bash -e {0} 2024-06-26T07:13:41.1033255Z env: 2024-06-26T07:13:41.1033529Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:13:41.1033975Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:13:41.1034585Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:13:41.1035086Z PR_NUMBER: 2024-06-26T07:13:41.1035408Z GITHUB_REPOSITORY: pytorch/pytorch 2024-06-26T07:13:41.1035831Z GITHUB_WORKFLOW: inductor-periodic 2024-06-26T07:13:41.1036322Z GITHUB_JOB: test 2024-06-26T07:13:41.1036708Z GITHUB_RUN_ID: 9673646806 2024-06-26T07:13:41.1037068Z GITHUB_RUN_NUMBER: 45567 2024-06-26T07:13:41.1037417Z GITHUB_RUN_ATTEMPT: 1 2024-06-26T07:13:41.1037759Z JOB_ID: 26688915732 2024-06-26T07:13:41.1038612Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:13:41.1039514Z BRANCH: 2024-06-26T07:13:41.1039846Z SHA1: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:41.1040364Z BASE_SHA: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:13:41.1040853Z TEST_CONFIG: dynamo_eager_huggingface 2024-06-26T07:13:41.1041263Z SHARD_NUMBER: 1 2024-06-26T07:13:41.1041563Z NUM_TEST_SHARDS: 1 2024-06-26T07:13:41.1041892Z REENABLED_ISSUES: 2024-06-26T07:13:41.1042233Z CONTINUE_THROUGH_ERROR: False 2024-06-26T07:13:41.1042612Z VERBOSE_TEST_LOGS: False 2024-06-26T07:13:41.1042970Z NO_TEST_TIMEOUT: False 2024-06-26T07:13:41.1043303Z NO_TD: False 2024-06-26T07:13:41.1043609Z TD_DISTRIBUTED: False 2024-06-26T07:13:41.1044023Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-06-26T07:13:41.1044539Z SCCACHE_S3_KEY_PREFIX: inductor-periodic 2024-06-26T07:13:41.1044969Z SHM_SIZE: 2g 2024-06-26T07:13:41.1046043Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:13:41.1047191Z XLA_CUDA: 2024-06-26T07:13:41.1047674Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:13:41.1048286Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-06-26T07:13:41.1048720Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-06-26T07:13:41.1049120Z DASHBOARD_TAG: 2024-06-26T07:13:41.1049635Z HUGGING_FACE_HUB_TOKEN: *** 2024-06-26T07:13:41.1049995Z ##[endgroup] 2024-06-26T07:13:41.1069711Z + [[ dynamo_eager_huggingface == \m\u\l\t\i\g\p\u ]] 2024-06-26T07:13:41.1071290Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-06-26T07:13:41.1072203Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:13:41.1076354Z +++ nproc --ignore=2 2024-06-26T07:13:41.1089042Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9673646806 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:13:52.3204941Z + container_name=fe4fb56dd485413b71c9d62a9524407caf78d7d40e572f4e0319fe63d37a8796 2024-06-26T07:13:52.3206392Z + grep download.pytorch.org /etc/hosts 2024-06-26T07:13:52.3207832Z + docker exec -i fe4fb56dd485413b71c9d62a9524407caf78d7d40e572f4e0319fe63d37a8796 sudo bash -c '/bin/cat >> /etc/hosts' 2024-06-26T07:13:52.3829466Z + echo DOCKER_CONTAINER_ID=fe4fb56dd485413b71c9d62a9524407caf78d7d40e572f4e0319fe63d37a8796 2024-06-26T07:13:52.3832086Z ++ echo dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:13:52.3834504Z + docker exec -t fe4fb56dd485413b71c9d62a9524407caf78d7d40e572f4e0319fe63d37a8796 sh -c 'pip install dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-06-26T07:13:52.7610313Z Processing ./dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+gitb8c4c54) 2024-06-26T07:13:53.0746505Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.13.1) 2024-06-26T07:13:53.0748788Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (4.12.2) 2024-06-26T07:13:53.0751476Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.12.1) 2024-06-26T07:13:53.0755319Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.8.8) 2024-06-26T07:13:53.0757877Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.1.4) 2024-06-26T07:13:53.0761382Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2024.2.0) 2024-06-26T07:13:53.0786694Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.3.0) 2024-06-26T07:13:53.0844120Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.21.2) 2024-06-26T07:13:53.1268044Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.1.5) 2024-06-26T07:13:53.1426140Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.3.0) 2024-06-26T07:13:54.4270884Z Installing collected packages: torch 2024-06-26T07:14:03.0968550Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-06-26T07:14:03.0969919Z timm 0.9.7 requires torchvision, which is not installed. 2024-06-26T07:14:03.0970597Z Successfully installed torch-2.5.0a0+gitb8c4c54 2024-06-26T07:14:03.1771602Z ++ dirname .ci/pytorch/test.sh 2024-06-26T07:14:03.1775877Z + source .ci/pytorch/common.sh 2024-06-26T07:14:03.1777765Z +++ dirname .ci/pytorch/common.sh 2024-06-26T07:14:03.1782911Z ++ source .ci/pytorch/common_utils.sh 2024-06-26T07:14:03.1784614Z +++ declare -f -t trap_add 2024-06-26T07:14:03.1789785Z ++ set -ex 2024-06-26T07:14:03.1790251Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:14:03.1790770Z ++ BUILD_TEST_LIBTORCH=0 2024-06-26T07:14:03.1792411Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-06-26T07:14:03.1794136Z ++ stat -c %u /var/lib/jenkins/workspace 2024-06-26T07:14:03.1802106Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-06-26T07:14:03.1802537Z + trap_add cleanup_workspace EXIT 2024-06-26T07:14:03.1802942Z + trap_add_cmd=cleanup_workspace 2024-06-26T07:14:03.1805003Z + shift 2024-06-26T07:14:03.1805361Z + for trap_add_name in "$@" 2024-06-26T07:14:03.1808404Z +++ trap -p EXIT 2024-06-26T07:14:03.1810439Z ++ eval 'extract_trap_cmd ' 2024-06-26T07:14:03.1810939Z +++ extract_trap_cmd 2024-06-26T07:14:03.1811417Z +++ printf '%s\n' '' 2024-06-26T07:14:03.1811794Z ++ printf '%s\n' cleanup_workspace 2024-06-26T07:14:03.1812308Z + trap -- ' 2024-06-26T07:14:03.1812761Z cleanup_workspace' EXIT 2024-06-26T07:14:03.1813272Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-06-26T07:14:03.7113474Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-06-26T07:14:03.7125439Z + echo 'Environment variables:' 2024-06-26T07:14:03.7125934Z Environment variables: 2024-06-26T07:14:03.7126256Z + env 2024-06-26T07:14:03.7131185Z INSTALLED_DB=yes 2024-06-26T07:14:03.7131690Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:14:03.7132084Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:14:03.7132461Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:14:03.7133061Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:14:03.7133639Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:14:03.7134133Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:14:03.7134846Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:14:03.7135631Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:14:03.7136428Z HOSTNAME=fe4fb56dd485 2024-06-26T07:14:03.7137621Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_57ff668d-9954-4d16-9956-fb6cc5ca0e0d 2024-06-26T07:14:03.7138778Z GITHUB_ACTION=__self 2024-06-26T07:14:03.7139299Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:14:03.7144775Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:14:03.7151075Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:14:03.7152144Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:14:03.7152655Z GITHUB_RUN_NUMBER=45567 2024-06-26T07:14:03.7153176Z TEST_CONFIG=dynamo_eager_huggingface 2024-06-26T07:14:03.7153771Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:14:03.7154448Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:14:03.7155121Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:14:03.7155754Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:14:03.7156573Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:14:03.7157216Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:14:03.7157938Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:14:03.7158637Z GITHUB_REF_TYPE=tag 2024-06-26T07:14:03.7159111Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:14:03.7159645Z NCCL_VERSION=2.17.1-1 2024-06-26T07:14:03.7160530Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:14:03.7161200Z XLA_CUDA= 2024-06-26T07:14:03.7161850Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:14:03.7164043Z *** 2024-06-26T07:14:03.7164464Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:14:03.7165041Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:14:03.7165583Z GITHUB_ACTIONS=true 2024-06-26T07:14:03.7166084Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:14:03.7166834Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:14:03.7167622Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:14:03.7168297Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:14:03.7169007Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:14:03.7169751Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:14:03.7171151Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:14:03.7172366Z UCC_HOME=/usr 2024-06-26T07:14:03.7172836Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:14:03.7173360Z VERBOSE_TEST_LOGS=False 2024-06-26T07:14:03.7173852Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:14:03.7174519Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:14:03.7175256Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:14:03.7176196Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:14:03.7176755Z SHARD_NUMBER=1 2024-06-26T07:14:03.7177206Z GITHUB_REF_PROTECTED=false 2024-06-26T07:14:03.7177710Z HOME=/var/lib/jenkins 2024-06-26T07:14:03.7178207Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:14:03.7178842Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:14:03.7179568Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:14:03.7180439Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-06-26T07:14:03.7181072Z CUDA_VERSION=12.1.1 2024-06-26T07:14:03.7181692Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:14:03.7182309Z NUM_TEST_SHARDS=1 2024-06-26T07:14:03.7182763Z UCX_HOME=/usr 2024-06-26T07:14:03.7183538Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:14:03.7185023Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_57ff668d-9954-4d16-9956-fb6cc5ca0e0d 2024-06-26T07:14:03.7187143Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:14:03.7189312Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_57ff668d-9954-4d16-9956-fb6cc5ca0e0d 2024-06-26T07:14:03.7190949Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:14:03.7192192Z GITHUB_EVENT_NAME=push 2024-06-26T07:14:03.7192674Z DASHBOARD_TAG= 2024-06-26T07:14:03.7193115Z GITHUB_RUN_ID=9673646806 2024-06-26T07:14:03.7193822Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:14:03.7194608Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:14:03.7196222Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_57ff668d-9954-4d16-9956-fb6cc5ca0e0d 2024-06-26T07:14:03.7197554Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:14:03.7198197Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:14:03.7198709Z PR_NUMBER= 2024-06-26T07:14:03.7199134Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:14:03.7199641Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:14:03.7200260Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:14:03.7200956Z TERM=xterm 2024-06-26T07:14:03.7201451Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:14:03.7202104Z INSTALLED_VISION=yes 2024-06-26T07:14:03.7202560Z BRANCH= 2024-06-26T07:14:03.7202976Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:14:03.7203527Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:14:03.7204137Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:14:03.7205314Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:14:03.7206563Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:14:03.7207500Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:14:03.7208218Z REENABLED_ISSUES= 2024-06-26T07:14:03.7208649Z SHLVL=1 2024-06-26T07:14:03.7209000Z MAX_JOBS=14 2024-06-26T07:14:03.7209500Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:14:03.7210028Z NVARCH=x86_64 2024-06-26T07:14:03.7210464Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:14:03.7211131Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:14:03.7211933Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:14:03.7212717Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:14:03.7213727Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:14:03.7214638Z GITHUB_JOB=test 2024-06-26T07:14:03.7215216Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:14:03.7216035Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:14:03.7216819Z NO_TEST_TIMEOUT=False 2024-06-26T07:14:03.7217333Z TD_DISTRIBUTED=False 2024-06-26T07:14:03.7217891Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:14:03.7218550Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:14:03.7219180Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:14:03.7219697Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:14:03.7220198Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:14:03.7220883Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:14:03.7222470Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:14:03.7224014Z GITHUB_BASE_REF= 2024-06-26T07:14:03.7224509Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:14:03.7225078Z CI=true 2024-06-26T07:14:03.7225570Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:14:03.7226177Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:14:03.7226790Z JOB_ID=26688915732 2024-06-26T07:14:03.7227254Z INSTALLED_PROTOBUF=yes 2024-06-26T07:14:03.7227716Z GITHUB_HEAD_REF= 2024-06-26T07:14:03.7228153Z GITHUB_ACTION_REF= 2024-06-26T07:14:03.7228815Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:14:03.7229590Z GITHUB_WORKFLOW=inductor-periodic 2024-06-26T07:14:03.7230224Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:14:03.7231702Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_57ff668d-9954-4d16-9956-fb6cc5ca0e0d 2024-06-26T07:14:03.7232919Z NO_TD=False 2024-06-26T07:14:03.7233356Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:14:03.7233894Z _=/usr/bin/env 2024-06-26T07:14:03.7234627Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-06-26T07:14:03.7335242Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-06-26T07:14:03.7336397Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T07:14:03.7337288Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-06-26T07:14:03.7338289Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-06-26T07:14:03.7338896Z + BUILD_DIR=build 2024-06-26T07:14:03.7339235Z + BUILD_RENAMED_DIR=build_renamed 2024-06-26T07:14:03.7339689Z + BUILD_BIN_DIR=build/bin 2024-06-26T07:14:03.7340025Z + SHARD_NUMBER=1 2024-06-26T07:14:03.7340325Z + NUM_TEST_SHARDS=1 2024-06-26T07:14:03.7340638Z + export VALGRIND=ON 2024-06-26T07:14:03.7340957Z + VALGRIND=ON 2024-06-26T07:14:03.7341415Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-06-26T07:14:03.7341901Z + [[ 0 == \1 ]] 2024-06-26T07:14:03.7342188Z + [[ False == \1 ]] 2024-06-26T07:14:03.7342648Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-06-26T07:14:03.7343172Z ++ realpath build/custom_test_artifacts 2024-06-26T07:14:03.7348472Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-06-26T07:14:03.7349233Z + [[ -n '' ]] 2024-06-26T07:14:03.7349672Z + echo 'Environment variables' 2024-06-26T07:14:03.7350100Z Environment variables 2024-06-26T07:14:03.7350411Z + env 2024-06-26T07:14:03.7355114Z INSTALLED_DB=yes 2024-06-26T07:14:03.7355515Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:14:03.7356215Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:14:03.7356622Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:14:03.7357219Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:14:03.7357913Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:14:03.7358421Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:14:03.7358995Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:14:03.7359559Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:14:03.7360052Z HOSTNAME=fe4fb56dd485 2024-06-26T07:14:03.7360869Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_57ff668d-9954-4d16-9956-fb6cc5ca0e0d 2024-06-26T07:14:03.7361663Z GITHUB_ACTION=__self 2024-06-26T07:14:03.7362005Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:14:03.7365768Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:14:03.7369703Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:14:03.7370208Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:14:03.7370551Z GITHUB_RUN_NUMBER=45567 2024-06-26T07:14:03.7370911Z TEST_CONFIG=dynamo_eager_huggingface 2024-06-26T07:14:03.7371333Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:14:03.7371798Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:14:03.7372264Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:14:03.7372703Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:14:03.7373126Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:14:03.7373561Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:14:03.7374062Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:14:03.7374529Z GITHUB_REF_TYPE=tag 2024-06-26T07:14:03.7374855Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:14:03.7375223Z NCCL_VERSION=2.17.1-1 2024-06-26T07:14:03.7375608Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:14:03.7376046Z XLA_CUDA= 2024-06-26T07:14:03.7376528Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:14:03.7376918Z *** 2024-06-26T07:14:03.7377197Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:14:03.7377591Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:14:03.7377954Z GITHUB_ACTIONS=true 2024-06-26T07:14:03.7378278Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:14:03.7378763Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:14:03.7379297Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:14:03.7379763Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:14:03.7380255Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:14:03.7380734Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:14:03.7381659Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:14:03.7382427Z UCC_HOME=/usr 2024-06-26T07:14:03.7382756Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:14:03.7383151Z VERBOSE_TEST_LOGS=False 2024-06-26T07:14:03.7383490Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:14:03.7383935Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:14:03.7384429Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:14:03.7384891Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:14:03.7385271Z SHARD_NUMBER=1 2024-06-26T07:14:03.7385572Z GITHUB_REF_PROTECTED=false 2024-06-26T07:14:03.7385924Z HOME=/var/lib/jenkins 2024-06-26T07:14:03.7386358Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:14:03.7386793Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:14:03.7387273Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:14:03.7387814Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-06-26T07:14:03.7388230Z CUDA_VERSION=12.1.1 2024-06-26T07:14:03.7388634Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:14:03.7389068Z NUM_TEST_SHARDS=1 2024-06-26T07:14:03.7389365Z UCX_HOME=/usr 2024-06-26T07:14:03.7389869Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:14:03.7390928Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_57ff668d-9954-4d16-9956-fb6cc5ca0e0d 2024-06-26T07:14:03.7392673Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:14:03.7394062Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_57ff668d-9954-4d16-9956-fb6cc5ca0e0d 2024-06-26T07:14:03.7395177Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:14:03.7395832Z GITHUB_EVENT_NAME=push 2024-06-26T07:14:03.7396263Z DASHBOARD_TAG= 2024-06-26T07:14:03.7396561Z GITHUB_RUN_ID=9673646806 2024-06-26T07:14:03.7397118Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:14:03.7397643Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:14:03.7398624Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_57ff668d-9954-4d16-9956-fb6cc5ca0e0d 2024-06-26T07:14:03.7399523Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:14:03.7399921Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:14:03.7400289Z PR_NUMBER= 2024-06-26T07:14:03.7400569Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:14:03.7400880Z VALGRIND=ON 2024-06-26T07:14:03.7401174Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:14:03.7401605Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:14:03.7402062Z TERM=xterm 2024-06-26T07:14:03.7402399Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:14:03.7402799Z INSTALLED_VISION=yes 2024-06-26T07:14:03.7403107Z BRANCH= 2024-06-26T07:14:03.7403383Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:14:03.7403776Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:14:03.7404194Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:14:03.7404931Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:14:03.7405672Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:14:03.7406153Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:14:03.7406634Z REENABLED_ISSUES= 2024-06-26T07:14:03.7406951Z SHLVL=1 2024-06-26T07:14:03.7407206Z MAX_JOBS=14 2024-06-26T07:14:03.7407505Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:14:03.7420762Z NVARCH=x86_64 2024-06-26T07:14:03.7421074Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:14:03.7421540Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:14:03.7422076Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:14:03.7422580Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:14:03.7423239Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:14:03.7423807Z GITHUB_JOB=test 2024-06-26T07:14:03.7424203Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:14:03.7424757Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:14:03.7425272Z NO_TEST_TIMEOUT=False 2024-06-26T07:14:03.7425603Z TD_DISTRIBUTED=False 2024-06-26T07:14:03.7425980Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:14:03.7426412Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:14:03.7426830Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:14:03.7427198Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:14:03.7427543Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:14:03.7427890Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:14:03.7428932Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:14:03.7430098Z GITHUB_BASE_REF= 2024-06-26T07:14:03.7430430Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:14:03.7430795Z CI=true 2024-06-26T07:14:03.7431305Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:14:03.7431776Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:14:03.7432150Z JOB_ID=26688915732 2024-06-26T07:14:03.7432458Z INSTALLED_PROTOBUF=yes 2024-06-26T07:14:03.7432786Z GITHUB_HEAD_REF= 2024-06-26T07:14:03.7433086Z GITHUB_ACTION_REF= 2024-06-26T07:14:03.7433515Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:14:03.7434037Z GITHUB_WORKFLOW=inductor-periodic 2024-06-26T07:14:03.7434443Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:14:03.7435337Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_57ff668d-9954-4d16-9956-fb6cc5ca0e0d 2024-06-26T07:14:03.7436237Z NO_TD=False 2024-06-26T07:14:03.7436540Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:14:03.7436948Z _=/usr/bin/env 2024-06-26T07:14:03.7437280Z + echo 'Testing pytorch' 2024-06-26T07:14:03.7437628Z Testing pytorch 2024-06-26T07:14:03.7437947Z + export LANG=C.UTF-8 2024-06-26T07:14:03.7438285Z + LANG=C.UTF-8 2024-06-26T07:14:03.7438584Z + PR_NUMBER= 2024-06-26T07:14:03.7438938Z + [[ dynamo_eager_huggingface == \d\e\f\a\u\l\t ]] 2024-06-26T07:14:03.7439597Z + [[ dynamo_eager_huggingface == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-06-26T07:14:03.7440135Z + [[ dynamo_eager_huggingface == \s\l\o\w ]] 2024-06-26T07:14:03.7440782Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-06-26T07:14:03.7441477Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-06-26T07:14:03.7442027Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:14:03.7442509Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:14:03.7442965Z + [[ dynamo_eager_huggingface == *crossref* ]] 2024-06-26T07:14:03.7443550Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:14:03.7444178Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-06-26T07:14:03.7444827Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:14:03.7445377Z + pip_install --user ninja==1.10.2 2024-06-26T07:14:03.7445901Z + pip install --progress-bar off --user ninja==1.10.2 2024-06-26T07:14:04.0987847Z Collecting ninja==1.10.2 2024-06-26T07:14:04.1150573Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-06-26T07:14:04.1288191Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-06-26T07:14:05.4076208Z Installing collected packages: ninja 2024-06-26T07:14:05.4142493Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-06-26T07:14:05.4143753Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-06-26T07:14:05.4181809Z Successfully installed ninja-1.10.2 2024-06-26T07:14:05.5027224Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:14:05.5029359Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:14:05.5030888Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:14:05.5031541Z + install_tlparse 2024-06-26T07:14:05.5031930Z + pip_install --user tlparse==0.3.7 2024-06-26T07:14:05.5032478Z + pip install --progress-bar off --user tlparse==0.3.7 2024-06-26T07:14:05.8511819Z Collecting tlparse==0.3.7 2024-06-26T07:14:05.8659959Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-06-26T07:14:05.8777505Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-06-26T07:14:07.2031016Z Installing collected packages: tlparse 2024-06-26T07:14:07.2410868Z Successfully installed tlparse-0.3.7 2024-06-26T07:14:07.3242671Z ++ python -m site --user-base 2024-06-26T07:14:07.3394608Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:14:07.3396540Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-06-26T07:14:07.3397370Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-06-26T07:14:07.3398035Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:14:07.3398975Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-06-26T07:14:07.3400118Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-06-26T07:14:07.3400808Z + cd test 2024-06-26T07:14:07.3401330Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-06-26T07:14:08.8168430Z + [[ dynamo_eager_huggingface == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-06-26T07:14:08.8169233Z + [[ dynamo_eager_huggingface == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-06-26T07:14:08.8170025Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:14:08.8170846Z + pushd test 2024-06-26T07:14:08.8171167Z ~/workspace/test ~/workspace 2024-06-26T07:14:08.8171682Z ++ python -c 'import torch; print(torch.version.cuda)' 2024-06-26T07:14:10.2991666Z + CUDA_VERSION=12.1 2024-06-26T07:14:10.2992229Z + '[' 12.1 == 12.4 ']' 2024-06-26T07:14:10.2992607Z + ISCUDA124= 2024-06-26T07:14:10.2992904Z + popd 2024-06-26T07:14:10.2993223Z ~/workspace 2024-06-26T07:14:10.2997330Z + DYNAMO_BENCHMARK_FLAGS=() 2024-06-26T07:14:10.2997957Z + [[ dynamo_eager_huggingface == *dynamo_eager* ]] 2024-06-26T07:14:10.2998708Z + DYNAMO_BENCHMARK_FLAGS+=(--backend eager) 2024-06-26T07:14:10.2999204Z + [[ dynamo_eager_huggingface == *dynamic* ]] 2024-06-26T07:14:10.2999724Z + [[ dynamo_eager_huggingface == *cpu_inductor* ]] 2024-06-26T07:14:10.3000258Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-06-26T07:14:10.3025160Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:14:10.3026099Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-06-26T07:14:10.3027943Z + cd test 2024-06-26T07:14:10.3028490Z + python -c 'import torch; print(torch.__config__.show())' 2024-06-26T07:14:11.6458662Z PyTorch built with: 2024-06-26T07:14:11.6459395Z - GCC 9.4 2024-06-26T07:14:11.6459847Z - C++ Version: 201703 2024-06-26T07:14:11.6461046Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:14:11.6462146Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:14:11.6462982Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:14:11.6463604Z - LAPACK is enabled (usually provided by MKL) 2024-06-26T07:14:11.6464293Z - NNPACK is enabled 2024-06-26T07:14:11.6464755Z - CPU capability usage: AVX2 2024-06-26T07:14:11.6465160Z - CUDA Runtime 12.1 2024-06-26T07:14:11.6465682Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-06-26T07:14:11.6466284Z - CuDNN 90.1 (built against CUDA 12.4) 2024-06-26T07:14:11.6466745Z - Magma 2.6.1 2024-06-26T07:14:11.6474244Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-06-26T07:14:11.6480935Z 2024-06-26T07:14:11.9253488Z + cd test 2024-06-26T07:14:11.9254279Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-06-26T07:14:13.1410992Z ATen/Parallel: 2024-06-26T07:14:13.1411473Z at::get_num_threads() : 8 2024-06-26T07:14:13.1411933Z at::get_num_interop_threads() : 16 2024-06-26T07:14:13.1412379Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:14:13.1412844Z omp_get_max_threads() : 8 2024-06-26T07:14:13.1413900Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:14:13.1414782Z mkl_get_max_threads() : 8 2024-06-26T07:14:13.1415480Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:14:13.1416520Z std::thread::hardware_concurrency() : 16 2024-06-26T07:14:13.1417015Z Environment variables: 2024-06-26T07:14:13.1417365Z OMP_NUM_THREADS : [not set] 2024-06-26T07:14:13.1417800Z MKL_NUM_THREADS : [not set] 2024-06-26T07:14:13.1418173Z ATen parallel backend: OpenMP 2024-06-26T07:14:13.1418440Z 2024-06-26T07:14:13.3869511Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:14:13.3870125Z + [[ dynamo_eager_huggingface == *backward* ]] 2024-06-26T07:14:13.3870625Z + [[ dynamo_eager_huggingface == *xla* ]] 2024-06-26T07:14:13.3871396Z + [[ dynamo_eager_huggingface == *executorch* ]] 2024-06-26T07:14:13.3872130Z + [[ dynamo_eager_huggingface == \j\i\t\_\l\e\g\a\c\y ]] 2024-06-26T07:14:13.3872889Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:14:13.3873559Z + [[ dynamo_eager_huggingface == distributed ]] 2024-06-26T07:14:13.3874099Z + [[ dynamo_eager_huggingface == *inductor_distributed* ]] 2024-06-26T07:14:13.3874734Z + [[ dynamo_eager_huggingface == *inductor-halide* ]] 2024-06-26T07:14:13.3875378Z + [[ dynamo_eager_huggingface == *inductor-micro-benchmark* ]] 2024-06-26T07:14:13.3876073Z + [[ dynamo_eager_huggingface == *huggingface* ]] 2024-06-26T07:14:13.3876523Z + install_torchvision 2024-06-26T07:14:13.3876850Z + local orig_preload 2024-06-26T07:14:13.3877170Z + local commit 2024-06-26T07:14:13.3877485Z ++ get_pinned_commit vision 2024-06-26T07:14:13.3877871Z ++ cat .github/ci_commit_pins/vision.txt 2024-06-26T07:14:13.3884184Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:14:13.3884721Z + orig_preload= 2024-06-26T07:14:13.3885086Z + '[' -n '' ']' 2024-06-26T07:14:13.3885930Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:14:13.3887354Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:14:13.6980346Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:14:13.6983046Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-ogd9c9dv 2024-06-26T07:14:13.7001667Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-ogd9c9dv 2024-06-26T07:14:15.1003152Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-06-26T07:14:15.1021949Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:14:16.3011154Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:14:16.5377293Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:14:18.8449494Z Preparing metadata (setup.py) ... [?25l- \ done 2024-06-26T07:14:18.8498105Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2) 2024-06-26T07:14:18.8500975Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+gitb8c4c54) 2024-06-26T07:14:18.8507507Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-06-26T07:14:18.8711434Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-06-26T07:14:18.8717546Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-06-26T07:14:18.8721320Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12.1) 2024-06-26T07:14:18.8723924Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-06-26T07:14:18.8726097Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-06-26T07:14:18.8729441Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.2.0) 2024-06-26T07:14:18.9251680Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-06-26T07:14:18.9408235Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-06-26T07:14:18.9482715Z Building wheels for collected packages: torchvision 2024-06-26T07:15:32.4146318Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-06-26T07:15:32.4172881Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2112598 sha256=ac47cbeac66e216090cb49483f303f478038b2059e1ea42102b30229c8c75e5b 2024-06-26T07:15:32.4175097Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-06-26T07:15:32.4209457Z Successfully built torchvision 2024-06-26T07:15:33.6236537Z Installing collected packages: torchvision 2024-06-26T07:15:34.0056284Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-06-26T07:15:34.1149259Z + '[' -n '' ']' 2024-06-26T07:15:34.1149668Z + id=0 2024-06-26T07:15:34.1150083Z + test_dynamo_benchmark huggingface 0 2024-06-26T07:15:34.1152781Z ++ pwd 2024-06-26T07:15:34.1154751Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:15:34.1155333Z + local suite=huggingface 2024-06-26T07:15:34.1155761Z + shift 2024-06-26T07:15:34.1156034Z + local shard_id=0 2024-06-26T07:15:34.1156338Z + shift 2024-06-26T07:15:34.1156709Z + [[ dynamo_eager_huggingface == *perf_compare* ]] 2024-06-26T07:15:34.1161214Z + [[ dynamo_eager_huggingface == *perf* ]] 2024-06-26T07:15:34.1161797Z + [[ dynamo_eager_huggingface == *cpu_inductor* ]] 2024-06-26T07:15:34.1162333Z + [[ dynamo_eager_huggingface == *aot_inductor* ]] 2024-06-26T07:15:34.1163172Z + test_single_dynamo_benchmark inference huggingface 0 --inference --bfloat16 2024-06-26T07:15:34.1163772Z ++ pwd 2024-06-26T07:15:34.1164536Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:15:34.1165208Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:15:34.1173023Z + local name=inference 2024-06-26T07:15:34.1173417Z + shift 2024-06-26T07:15:34.1174051Z + local suite=huggingface 2024-06-26T07:15:34.1174579Z + shift 2024-06-26T07:15:34.1174933Z + local shard_id=0 2024-06-26T07:15:34.1175358Z + shift 2024-06-26T07:15:34.1175797Z + partition_flags=() 2024-06-26T07:15:34.1176249Z + local partition_flags 2024-06-26T07:15:34.1176767Z + [[ -n 1 ]] 2024-06-26T07:15:34.1177227Z + [[ -n 0 ]] 2024-06-26T07:15:34.1178099Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-06-26T07:15:34.1179129Z + [[ dynamo_eager_huggingface == *perf_compare* ]] 2024-06-26T07:15:34.1179867Z + [[ dynamo_eager_huggingface == *perf* ]] 2024-06-26T07:15:34.1180610Z + [[ dynamo_eager_huggingface == *aot_inductor* ]] 2024-06-26T07:15:34.1183128Z + python benchmarks/dynamo/huggingface.py --ci --accuracy --timing --explain --backend eager --device cuda --inference --bfloat16 --total-partitions 1 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv 2024-06-26T07:15:36.7818311Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:15:36.7820299Z warnings.warn( 2024-06-26T07:15:36.8264438Z 2024-06-26T07:15:36.8264722Z config.json: 0% 0.00/694 [00:00 will be ignored 2024-06-26T07:27:26.6386906Z pass 2024-06-26T07:27:26.7207408Z TIMING: entire_frame_compile:3.42628 backend_compile:0.00024 2024-06-26T07:27:26.7208809Z STATS: call_* op count: 534 | FakeTensorMode.__torch_dispatch__:9028 | FakeTensor.__torch_dispatch__:1705 2024-06-26T07:27:26.7209759Z Dynamo produced 2 graphs covering 534 ops with 5 graph breaks (4 unique) 2024-06-26T07:27:30.1080536Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:27:30.1082218Z warnings.warn( 2024-06-26T07:27:30.3279476Z 2024-06-26T07:27:33.0468532Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:27:33.0469599Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:27:33.0470487Z cuda train AllenaiLongformerBase 2024-06-26T07:27:40.5027958Z W0626 07:27:40.502000 140675854062208 torch/_dynamo/variables/tensor.py:714] [3/0] Graph break from `Tensor.item()`, consider setting: 2024-06-26T07:27:40.5029524Z W0626 07:27:40.502000 140675854062208 torch/_dynamo/variables/tensor.py:714] [3/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-26T07:27:40.5030942Z W0626 07:27:40.502000 140675854062208 torch/_dynamo/variables/tensor.py:714] [3/0] or: 2024-06-26T07:27:40.5032674Z W0626 07:27:40.502000 140675854062208 torch/_dynamo/variables/tensor.py:714] [3/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-26T07:27:40.5034327Z W0626 07:27:40.502000 140675854062208 torch/_dynamo/variables/tensor.py:714] [3/0] to include these operations in the captured graph. 2024-06-26T07:27:40.5035393Z W0626 07:27:40.502000 140675854062208 torch/_dynamo/variables/tensor.py:714] [3/0] 2024-06-26T07:27:47.9006402Z W0626 07:27:47.899000 140675854062208 torch/_logging/_internal.py:1040] [16/0] Profiler function will be ignored 2024-06-26T07:27:56.3320819Z pass 2024-06-26T07:27:56.3993242Z TIMING: entire_frame_compile:16.61831 backend_compile:0.00128 2024-06-26T07:27:56.3994130Z STATS: call_* op count: 2367 | FakeTensorMode.__torch_dispatch__:9581 | FakeTensor.__torch_dispatch__:4593 2024-06-26T07:27:56.3995087Z Dynamo produced 7 graphs covering 2367 ops with 9 graph breaks (5 unique) 2024-06-26T07:28:00.0869417Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:28:00.0871225Z warnings.warn( 2024-06-26T07:28:00.2765537Z 2024-06-26T07:28:03.8584745Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:28:03.8585622Z loading model: 0it [00:03, ?it/s] 2024-06-26T07:28:03.8588497Z cuda train BartForCausalLM 2024-06-26T07:28:13.2427111Z pass 2024-06-26T07:28:13.3867331Z TIMING: entire_frame_compile:1.70508 backend_compile:0.0007 2024-06-26T07:28:13.3869070Z STATS: call_* op count: 325 | FakeTensorMode.__torch_dispatch__:4002 | FakeTensor.__torch_dispatch__:736 2024-06-26T07:28:13.3870537Z Dynamo produced 16 graphs covering 325 ops with 6 graph breaks (5 unique) 2024-06-26T07:28:16.7136739Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:28:16.7138477Z warnings.warn( 2024-06-26T07:28:16.9098295Z 2024-06-26T07:28:23.4303669Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:28:23.4304385Z loading model: 0it [00:06, ?it/s] 2024-06-26T07:28:23.4305056Z cuda train BartForConditionalGeneration 2024-06-26T07:28:43.3855229Z pass 2024-06-26T07:28:43.6310425Z TIMING: entire_frame_compile:3.95722 backend_compile:0.0014 2024-06-26T07:28:43.6311733Z STATS: call_* op count: 878 | FakeTensorMode.__torch_dispatch__:10425 | FakeTensor.__torch_dispatch__:1927 2024-06-26T07:28:43.6313723Z Dynamo produced 30 graphs covering 878 ops with 8 graph breaks (5 unique) 2024-06-26T07:28:47.0278018Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:28:47.0279930Z warnings.warn( 2024-06-26T07:28:47.2545387Z 2024-06-26T07:28:49.0560362Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:28:49.0560999Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:28:49.0561459Z cuda train BertForMaskedLM 2024-06-26T07:28:56.3970520Z W0626 07:28:56.396000 140708000547456 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:29:04.0435320Z pass 2024-06-26T07:29:04.1309279Z TIMING: entire_frame_compile:12.08983 backend_compile:0.0007 2024-06-26T07:29:04.1310342Z STATS: call_* op count: 996 | FakeTensorMode.__torch_dispatch__:8795 | FakeTensor.__torch_dispatch__:5076 2024-06-26T07:29:04.1312269Z Dynamo produced 2 graphs covering 996 ops with 5 graph breaks (4 unique) 2024-06-26T07:29:07.7253901Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:29:07.7255884Z warnings.warn( 2024-06-26T07:29:07.9144265Z 2024-06-26T07:29:09.4970862Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:29:09.4971438Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:29:09.4972428Z cuda train BertForQuestionAnswering 2024-06-26T07:29:16.4581111Z W0626 07:29:16.457000 140234640925312 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:29:24.0040750Z pass 2024-06-26T07:29:24.0893506Z TIMING: entire_frame_compile:11.95213 backend_compile:0.00068 2024-06-26T07:29:24.0894763Z STATS: call_* op count: 994 | FakeTensorMode.__torch_dispatch__:8800 | FakeTensor.__torch_dispatch__:4996 2024-06-26T07:29:24.0896104Z Dynamo produced 2 graphs covering 994 ops with 5 graph breaks (4 unique) 2024-06-26T07:29:27.6887478Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:29:27.6889293Z warnings.warn( 2024-06-26T07:29:27.9430715Z 2024-06-26T07:29:55.0644024Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:29:55.0644736Z loading model: 0it [00:27, ?it/s] 2024-06-26T07:29:55.0645413Z cuda train BlenderbotForCausalLM 2024-06-26T07:29:55.0651420Z Traceback (most recent call last): 2024-06-26T07:29:55.0652553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T07:29:55.0653740Z self.model_iter_fn(model, example_inputs) 2024-06-26T07:29:55.0655038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in forward_and_backward_pass 2024-06-26T07:29:55.0656265Z pred = mod(**cloned_inputs) 2024-06-26T07:29:55.0657758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:29:55.0659110Z return self._call_impl(*args, **kwargs) 2024-06-26T07:29:55.0660443Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:29:55.0661695Z return forward_call(*args, **kwargs) 2024-06-26T07:29:55.0663259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 1531, in forward 2024-06-26T07:29:55.0664704Z outputs = self.model.decoder( 2024-06-26T07:29:55.0666086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:29:55.0667643Z return self._call_impl(*args, **kwargs) 2024-06-26T07:29:55.0669006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:29:55.0670206Z return forward_call(*args, **kwargs) 2024-06-26T07:29:55.0672023Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 997, in forward 2024-06-26T07:29:55.0673474Z layer_outputs = decoder_layer( 2024-06-26T07:29:55.0674853Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:29:55.0676286Z return self._call_impl(*args, **kwargs) 2024-06-26T07:29:55.0677631Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:29:55.0678841Z return forward_call(*args, **kwargs) 2024-06-26T07:29:55.0680394Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 397, in forward 2024-06-26T07:29:55.0682014Z hidden_states, self_attn_weights, present_key_value = self.self_attn( 2024-06-26T07:29:55.0683682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:29:55.0685176Z return self._call_impl(*args, **kwargs) 2024-06-26T07:29:55.0686542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:29:55.0687763Z return forward_call(*args, **kwargs) 2024-06-26T07:29:55.0689326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 152, in forward 2024-06-26T07:29:55.0690777Z query_states = self.q_proj(hidden_states) * self.scaling 2024-06-26T07:29:55.0692325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:29:55.0693617Z return self._call_impl(*args, **kwargs) 2024-06-26T07:29:55.0694909Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:29:55.0696121Z return forward_call(*args, **kwargs) 2024-06-26T07:29:55.0697449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 125, in forward 2024-06-26T07:29:55.0698711Z return F.linear(input, self.weight, self.bias) 2024-06-26T07:29:55.0703352Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 14.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 6.44 MiB is free. Process 18858 has 21.96 GiB memory in use. Of the allocated memory 21.61 GiB is allocated by PyTorch, and 27.77 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-06-26T07:29:55.0707105Z 2024-06-26T07:29:55.0707544Z The above exception was the direct cause of the following exception: 2024-06-26T07:29:55.0708209Z 2024-06-26T07:29:55.0708437Z Traceback (most recent call last): 2024-06-26T07:29:55.0709372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T07:29:55.0710326Z ) = runner.load_model( 2024-06-26T07:29:55.0711497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 522, in load_model 2024-06-26T07:29:55.0712632Z self.validate_model(model, example_inputs) 2024-06-26T07:29:55.0713767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T07:29:55.0714887Z raise RuntimeError("Eager run failed") from e 2024-06-26T07:29:55.0715564Z RuntimeError: Eager run failed 2024-06-26T07:29:55.0716027Z 2024-06-26T07:29:55.0716208Z eager_fail_to_run 2024-06-26T07:29:58.3442973Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:29:58.3444928Z warnings.warn( 2024-06-26T07:29:58.5232460Z 2024-06-26T07:29:59.7887934Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:29:59.7888410Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:29:59.7888876Z cuda train BlenderbotSmallForCausalLM 2024-06-26T07:29:59.8108354Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForCausalLM. Setting accuracy check to cosine 2024-06-26T07:30:02.0593767Z pass 2024-06-26T07:30:02.0946295Z TIMING: entire_frame_compile:1.52846 backend_compile:0.00061 2024-06-26T07:30:02.0947550Z STATS: call_* op count: 324 | FakeTensorMode.__torch_dispatch__:3088 | FakeTensor.__torch_dispatch__:508 2024-06-26T07:30:02.0948873Z Dynamo produced 13 graphs covering 324 ops with 6 graph breaks (5 unique) 2024-06-26T07:30:05.4362269Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:30:05.4364135Z warnings.warn( 2024-06-26T07:30:05.6388864Z 2024-06-26T07:30:07.2700502Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:30:07.2700971Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:30:07.2701468Z cuda train BlenderbotSmallForConditionalGeneration 2024-06-26T07:30:07.4108455Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T07:30:11.8336474Z pass 2024-06-26T07:30:11.8702296Z TIMING: entire_frame_compile:3.2923 backend_compile:0.00107 2024-06-26T07:30:11.8703278Z STATS: call_* op count: 830 | FakeTensorMode.__torch_dispatch__:7670 | FakeTensor.__torch_dispatch__:1283 2024-06-26T07:30:11.8704206Z Dynamo produced 22 graphs covering 830 ops with 8 graph breaks (5 unique) 2024-06-26T07:30:15.2693356Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:30:15.2695062Z warnings.warn( 2024-06-26T07:30:15.4826160Z 2024-06-26T07:30:17.3200501Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:30:17.3201084Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:30:17.3201594Z cuda train CamemBert 2024-06-26T07:30:24.6531920Z W0626 07:30:24.652000 139930066358912 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:30:32.2928330Z pass 2024-06-26T07:30:32.3789672Z TIMING: entire_frame_compile:12.04877 backend_compile:0.0007 2024-06-26T07:30:32.3790578Z STATS: call_* op count: 1004 | FakeTensorMode.__torch_dispatch__:8822 | FakeTensor.__torch_dispatch__:5076 2024-06-26T07:30:32.3791706Z Dynamo produced 2 graphs covering 1004 ops with 5 graph breaks (4 unique) 2024-06-26T07:30:35.9518774Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:30:35.9520450Z warnings.warn( 2024-06-26T07:30:36.1716181Z 2024-06-26T07:30:38.3266610Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:30:38.3267185Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:30:38.3268368Z cuda train DebertaForMaskedLM 2024-06-26T07:30:46.2129254Z W0626 07:30:46.212000 140273582400128 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:30:52.4925144Z pass 2024-06-26T07:30:52.5527916Z TIMING: entire_frame_compile:11.08161 backend_compile:0.00069 2024-06-26T07:30:52.5528859Z STATS: call_* op count: 1319 | FakeTensorMode.__torch_dispatch__:9153 | FakeTensor.__torch_dispatch__:3707 2024-06-26T07:30:52.5529799Z Dynamo produced 2 graphs covering 1319 ops with 5 graph breaks (4 unique) 2024-06-26T07:30:56.1077592Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:30:56.1079284Z warnings.warn( 2024-06-26T07:30:56.3150474Z 2024-06-26T07:30:58.1043450Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:30:58.1043919Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:30:58.1044471Z cuda train DebertaForQuestionAnswering 2024-06-26T07:31:05.3987673Z W0626 07:31:05.398000 139857325466240 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:31:11.5242043Z pass 2024-06-26T07:31:11.5933146Z TIMING: entire_frame_compile:10.86849 backend_compile:0.00069 2024-06-26T07:31:11.5934581Z STATS: call_* op count: 1317 | FakeTensorMode.__torch_dispatch__:9158 | FakeTensor.__torch_dispatch__:3628 2024-06-26T07:31:11.5935737Z Dynamo produced 2 graphs covering 1317 ops with 5 graph breaks (4 unique) 2024-06-26T07:31:15.1740065Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:31:15.1741710Z warnings.warn( 2024-06-26T07:31:15.3789563Z 2024-06-26T07:31:28.6923715Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:31:28.6924371Z loading model: 0it [00:13, ?it/s] 2024-06-26T07:31:28.6924911Z cuda train DebertaV2ForMaskedLM 2024-06-26T07:31:28.6925394Z pass_due_to_skip 2024-06-26T07:31:28.9140735Z TIMING: 2024-06-26T07:31:28.9141515Z STATS: call_* op count: 0 2024-06-26T07:31:28.9142778Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T07:31:32.2082372Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:31:32.2084040Z warnings.warn( 2024-06-26T07:31:32.4751852Z 2024-06-26T07:31:42.4104667Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:31:42.4105753Z loading model: 0it [00:09, ?it/s] 2024-06-26T07:31:42.4106720Z cuda train DebertaV2ForQuestionAnswering 2024-06-26T07:31:53.0389941Z ERROR:common: 2024-06-26T07:31:53.0390411Z Traceback (most recent call last): 2024-06-26T07:31:53.0391415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2543, in check_accuracy 2024-06-26T07:31:53.0392183Z correct_result = self.run_n_iterations( 2024-06-26T07:31:53.0392948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2359, in run_n_iterations 2024-06-26T07:31:53.0393792Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-06-26T07:31:53.0394715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in forward_and_backward_pass 2024-06-26T07:31:53.0395533Z pred = mod(**cloned_inputs) 2024-06-26T07:31:53.0396700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:31:53.0397598Z return self._call_impl(*args, **kwargs) 2024-06-26T07:31:53.0403739Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:31:53.0405203Z return forward_call(*args, **kwargs) 2024-06-26T07:31:53.0406991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1486, in forward 2024-06-26T07:31:53.0408313Z outputs = self.deberta( 2024-06-26T07:31:53.0409256Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:31:53.0410163Z return self._call_impl(*args, **kwargs) 2024-06-26T07:31:53.0411089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:31:53.0411921Z return forward_call(*args, **kwargs) 2024-06-26T07:31:53.0412967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1070, in forward 2024-06-26T07:31:53.0413914Z encoder_outputs = self.encoder( 2024-06-26T07:31:53.0414871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:31:53.0415759Z return self._call_impl(*args, **kwargs) 2024-06-26T07:31:53.0416688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:31:53.0417664Z return forward_call(*args, **kwargs) 2024-06-26T07:31:53.0418703Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 514, in forward 2024-06-26T07:31:53.0419648Z output_states = layer_module( 2024-06-26T07:31:53.0420611Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:31:53.0421719Z return self._call_impl(*args, **kwargs) 2024-06-26T07:31:53.0422642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:31:53.0423475Z return forward_call(*args, **kwargs) 2024-06-26T07:31:53.0424515Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 362, in forward 2024-06-26T07:31:53.0425464Z attention_output = self.attention( 2024-06-26T07:31:53.0426425Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:31:53.0427329Z return self._call_impl(*args, **kwargs) 2024-06-26T07:31:53.0428255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:31:53.0429083Z return forward_call(*args, **kwargs) 2024-06-26T07:31:53.0430125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 293, in forward 2024-06-26T07:31:53.0431054Z self_output = self.self( 2024-06-26T07:31:53.0432239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:31:53.0433126Z return self._call_impl(*args, **kwargs) 2024-06-26T07:31:53.0434053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:31:53.0434880Z return forward_call(*args, **kwargs) 2024-06-26T07:31:53.0436041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 733, in forward 2024-06-26T07:31:53.0437194Z attention_probs = XSoftmax.apply(attention_scores, attention_mask, -1) 2024-06-26T07:31:53.0438231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/function.py", line 574, in apply 2024-06-26T07:31:53.0439127Z return super().apply(*args, **kwargs) # type: ignore[misc] 2024-06-26T07:31:53.0440262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 113, in forward 2024-06-26T07:31:53.0441228Z output = torch.softmax(output, self.dim) 2024-06-26T07:31:53.0444554Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 24.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 10.44 MiB is free. Process 19226 has 21.96 GiB memory in use. Of the allocated memory 21.29 GiB is allocated by PyTorch, and 318.25 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-06-26T07:31:53.0708014Z eager_1st_run_OOM 2024-06-26T07:31:53.0708331Z TIMING: 2024-06-26T07:31:53.0708622Z STATS: call_* op count: 0 2024-06-26T07:31:53.0709165Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T07:31:56.3562738Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:31:56.3564408Z warnings.warn( 2024-06-26T07:31:56.5692133Z 2024-06-26T07:31:57.8786701Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:31:57.8787342Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:31:57.8788144Z cuda train DistilBertForMaskedLM 2024-06-26T07:32:00.1063295Z W0626 07:32:00.105000 139702928896640 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:32:04.2460949Z pass 2024-06-26T07:32:04.2946510Z TIMING: entire_frame_compile:5.36611 backend_compile:0.00039 2024-06-26T07:32:04.2947613Z STATS: call_* op count: 541 | FakeTensor.__torch_dispatch__:2650 | FakeTensorMode.__torch_dispatch__:4595 2024-06-26T07:32:04.2948533Z Dynamo produced 2 graphs covering 541 ops with 5 graph breaks (4 unique) 2024-06-26T07:32:07.7244735Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:32:07.7246502Z warnings.warn( 2024-06-26T07:32:07.9263269Z 2024-06-26T07:32:08.9898593Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:32:08.9899067Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:32:08.9899536Z cuda train DistilBertForQuestionAnswering 2024-06-26T07:32:11.1074887Z W0626 07:32:11.106000 140302736073344 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:32:15.1274726Z pass 2024-06-26T07:32:15.1761448Z TIMING: entire_frame_compile:5.23309 backend_compile:0.00039 2024-06-26T07:32:15.1762397Z STATS: call_* op count: 540 | FakeTensor.__torch_dispatch__:2571 | FakeTensorMode.__torch_dispatch__:4612 2024-06-26T07:32:15.1763314Z Dynamo produced 2 graphs covering 540 ops with 5 graph breaks (4 unique) 2024-06-26T07:32:18.6097265Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:32:18.6098919Z warnings.warn( 2024-06-26T07:32:18.7910526Z 2024-06-26T07:32:20.6194777Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:32:20.6195271Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:32:20.6195790Z cuda train DistillGPT2 2024-06-26T07:32:24.5180289Z W0626 07:32:24.517000 140486216393344 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:32:27.3412474Z pass 2024-06-26T07:32:27.3798825Z TIMING: entire_frame_compile:4.74832 backend_compile:0.00035 2024-06-26T07:32:27.3799810Z STATS: call_* op count: 572 | FakeTensorMode.__torch_dispatch__:3735 | FakeTensor.__torch_dispatch__:1629 2024-06-26T07:32:27.3800926Z Dynamo produced 2 graphs covering 572 ops with 5 graph breaks (4 unique) 2024-06-26T07:32:30.8009256Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:32:30.8010929Z warnings.warn( 2024-06-26T07:32:31.0025000Z 2024-06-26T07:32:31.0034165Z loading model: 0it [00:00, ?it/s]If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2024-06-26T07:32:31.5938708Z 2024-06-26T07:32:31.5939466Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:32:31.5939993Z cuda train ElectraForCausalLM 2024-06-26T07:32:37.4443824Z pass 2024-06-26T07:32:37.4653288Z TIMING: entire_frame_compile:4.73785 backend_compile:0.00012 2024-06-26T07:32:37.4655013Z STATS: call_* op count: 377 | FakeTensorMode.__torch_dispatch__:8919 | FakeTensor.__torch_dispatch__:1253 2024-06-26T07:32:37.4656829Z Dynamo produced 1 graphs covering 377 ops with 4 graph breaks (4 unique) 2024-06-26T07:32:40.9240734Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:32:40.9242752Z warnings.warn( 2024-06-26T07:32:41.2610798Z 2024-06-26T07:32:41.8435110Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:32:41.8435588Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:32:41.8436198Z cuda train ElectraForQuestionAnswering 2024-06-26T07:32:47.4923918Z W0626 07:32:47.491000 139831336465024 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:32:55.0739258Z pass 2024-06-26T07:32:55.0871644Z TIMING: entire_frame_compile:12.07067 backend_compile:0.00067 2024-06-26T07:32:55.0873633Z STATS: call_* op count: 1001 | FakeTensorMode.__torch_dispatch__:8878 | FakeTensor.__torch_dispatch__:5049 2024-06-26T07:32:55.0874617Z Dynamo produced 2 graphs covering 1001 ops with 5 graph breaks (4 unique) 2024-06-26T07:32:58.6692369Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:32:58.6694041Z warnings.warn( 2024-06-26T07:32:59.0324200Z 2024-06-26T07:33:01.1114189Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:33:01.1114668Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:33:01.1115902Z cuda train GPT2ForSequenceClassification 2024-06-26T07:33:10.7223925Z W0626 07:33:10.721000 139799765631616 torch/_logging/_internal.py:1040] [7/0] Profiler function will be ignored 2024-06-26T07:33:16.4802567Z pass 2024-06-26T07:33:16.5795618Z TIMING: entire_frame_compile:11.02572 backend_compile:0.00074 2024-06-26T07:33:16.5796787Z STATS: call_* op count: 1102 | FakeTensorMode.__torch_dispatch__:10294 | FakeTensor.__torch_dispatch__:3347 2024-06-26T07:33:16.5797727Z Dynamo produced 5 graphs covering 1102 ops with 7 graph breaks (5 unique) 2024-06-26T07:33:20.0651123Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:33:20.0652787Z warnings.warn( 2024-06-26T07:33:20.2586137Z 2024-06-26T07:33:21.6341332Z loading model: 0it [00:00, ?it/s]WARNING:common:Model GoogleFnet supports float32 only 2024-06-26T07:33:21.7862613Z 2024-06-26T07:33:21.7863323Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:33:21.7865043Z WARNING:common:Model GoogleFnet supports float32 only 2024-06-26T07:33:21.7872866Z cuda train GoogleFnet 2024-06-26T07:33:23.1153079Z WARNING:common:Model GoogleFnet supports float32 only 2024-06-26T07:33:25.6618789Z W0626 07:33:25.661000 139794981429888 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:33:29.6311399Z pass 2024-06-26T07:33:29.6722838Z TIMING: entire_frame_compile:5.89574 backend_compile:0.0004 2024-06-26T07:33:29.6723689Z STATS: call_* op count: 574 | FakeTensorMode.__torch_dispatch__:3085 | FakeTensor.__torch_dispatch__:2738 2024-06-26T07:33:29.6724601Z Dynamo produced 2 graphs covering 574 ops with 5 graph breaks (4 unique) 2024-06-26T07:33:33.1037475Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:33:33.1039145Z warnings.warn( 2024-06-26T07:33:33.2912798Z 2024-06-26T07:33:35.1382730Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:33:35.1383604Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:33:35.1384452Z cuda train LayoutLMForMaskedLM 2024-06-26T07:33:42.6383713Z W0626 07:33:42.637000 140483592192640 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:33:50.5027492Z pass 2024-06-26T07:33:50.5949669Z TIMING: entire_frame_compile:12.43805 backend_compile:0.0007 2024-06-26T07:33:50.5950552Z STATS: call_* op count: 1034 | FakeTensorMode.__torch_dispatch__:9145 | FakeTensor.__torch_dispatch__:5201 2024-06-26T07:33:50.5951688Z Dynamo produced 2 graphs covering 1034 ops with 5 graph breaks (4 unique) 2024-06-26T07:33:54.1378330Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:33:54.1379981Z warnings.warn( 2024-06-26T07:33:54.3192822Z 2024-06-26T07:33:55.9196089Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:33:55.9196777Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:33:55.9197506Z cuda train LayoutLMForSequenceClassification 2024-06-26T07:34:05.2207269Z W0626 07:34:05.219000 139856552764032 torch/_logging/_internal.py:1040] [7/0] Profiler function will be ignored 2024-06-26T07:34:13.0630355Z pass 2024-06-26T07:34:13.1509315Z TIMING: entire_frame_compile:14.5494 backend_compile:0.00083 2024-06-26T07:34:13.1510210Z STATS: call_* op count: 1031 | FakeTensorMode.__torch_dispatch__:13386 | FakeTensor.__torch_dispatch__:5804 2024-06-26T07:34:13.1511404Z Dynamo produced 5 graphs covering 1031 ops with 7 graph breaks (5 unique) 2024-06-26T07:34:16.7592334Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:34:16.7594203Z warnings.warn( 2024-06-26T07:34:16.9797741Z 2024-06-26T07:34:26.9022121Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:34:26.9023042Z loading model: 0it [00:09, ?it/s] 2024-06-26T07:34:26.9024012Z cuda train M2M100ForConditionalGeneration 2024-06-26T07:34:26.9804201Z WARNING:common:fp64 golden ref were not generated for M2M100ForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T07:34:34.6743735Z pass 2024-06-26T07:34:34.8823457Z TIMING: entire_frame_compile:5.73567 backend_compile:0.00029 2024-06-26T07:34:34.8824429Z STATS: call_* op count: 1296 | FakeTensorMode.__torch_dispatch__:21053 | FakeTensor.__torch_dispatch__:3084 2024-06-26T07:34:34.8825832Z Dynamo produced 1 graphs covering 1296 ops with 4 graph breaks (4 unique) 2024-06-26T07:34:38.3027526Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:34:38.3029459Z warnings.warn( 2024-06-26T07:34:38.7112584Z 2024-06-26T07:34:42.3329138Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:34:42.3329634Z loading model: 0it [00:03, ?it/s] 2024-06-26T07:34:42.3330081Z cuda train MBartForCausalLM 2024-06-26T07:34:42.3643474Z WARNING:common:fp64 golden ref were not generated for MBartForCausalLM. Setting accuracy check to cosine 2024-06-26T07:34:45.5797150Z pass 2024-06-26T07:34:45.6843798Z TIMING: entire_frame_compile:1.98813 backend_compile:0.00091 2024-06-26T07:34:45.6844753Z STATS: call_* op count: 478 | FakeTensorMode.__torch_dispatch__:4444 | FakeTensor.__torch_dispatch__:724 2024-06-26T07:34:45.6845692Z Dynamo produced 17 graphs covering 478 ops with 6 graph breaks (5 unique) 2024-06-26T07:34:49.0486907Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:34:49.0489140Z warnings.warn( 2024-06-26T07:34:49.2490225Z 2024-06-26T07:34:56.0649469Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:34:56.0649975Z loading model: 0it [00:06, ?it/s] 2024-06-26T07:34:56.0650445Z cuda train MBartForConditionalGeneration 2024-06-26T07:34:56.1656779Z WARNING:common:fp64 golden ref were not generated for MBartForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T07:35:03.7738329Z pass 2024-06-26T07:35:03.9797996Z TIMING: entire_frame_compile:4.41852 backend_compile:0.00152 2024-06-26T07:35:03.9798930Z STATS: call_* op count: 1251 | FakeTensorMode.__torch_dispatch__:11312 | FakeTensor.__torch_dispatch__:1843 2024-06-26T07:35:03.9799855Z Dynamo produced 31 graphs covering 1251 ops with 8 graph breaks (5 unique) 2024-06-26T07:35:07.4274306Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:35:07.4276075Z warnings.warn( 2024-06-26T07:35:07.6151898Z 2024-06-26T07:35:11.4385571Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:35:11.4386039Z loading model: 0it [00:03, ?it/s] 2024-06-26T07:35:11.4388794Z cuda train MT5ForConditionalGeneration 2024-06-26T07:35:20.8962315Z W0626 07:35:20.895000 139778899878528 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:35:29.4550880Z pass 2024-06-26T07:35:29.6388539Z TIMING: entire_frame_compile:15.29827 backend_compile:0.00081 2024-06-26T07:35:29.6389551Z STATS: call_* op count: 1764 | FakeTensorMode.__torch_dispatch__:15374 | FakeTensor.__torch_dispatch__:4522 2024-06-26T07:35:29.6390484Z Dynamo produced 2 graphs covering 1764 ops with 5 graph breaks (4 unique) 2024-06-26T07:35:33.2594175Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:35:33.2595994Z warnings.warn( 2024-06-26T07:35:33.4830426Z 2024-06-26T07:35:33.4877680Z loading model: 0it [00:00, ?it/s]If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2024-06-26T07:35:37.9143694Z 2024-06-26T07:35:37.9144581Z loading model: 0it [00:04, ?it/s] 2024-06-26T07:35:37.9145127Z cuda train MegatronBertForCausalLM 2024-06-26T07:35:54.2661476Z W0626 07:35:54.265000 140473787028096 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:36:09.3432672Z pass 2024-06-26T07:36:09.5922816Z TIMING: entire_frame_compile:23.91487 backend_compile:0.00132 2024-06-26T07:36:09.5923764Z STATS: call_* op count: 1923 | FakeTensorMode.__torch_dispatch__:17059 | FakeTensor.__torch_dispatch__:9876 2024-06-26T07:36:09.5924691Z Dynamo produced 2 graphs covering 1923 ops with 5 graph breaks (4 unique) 2024-06-26T07:36:13.4896200Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:36:13.4897841Z warnings.warn( 2024-06-26T07:36:13.7546717Z 2024-06-26T07:36:17.9048359Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:36:17.9048963Z loading model: 0it [00:04, ?it/s] 2024-06-26T07:36:17.9049458Z cuda train MegatronBertForQuestionAnswering 2024-06-26T07:36:33.7913726Z W0626 07:36:33.790000 139833588339328 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:36:48.8958838Z pass 2024-06-26T07:36:49.1383893Z TIMING: entire_frame_compile:23.92521 backend_compile:0.00126 2024-06-26T07:36:49.1384987Z STATS: call_* op count: 1917 | FakeTensorMode.__torch_dispatch__:17022 | FakeTensor.__torch_dispatch__:9796 2024-06-26T07:36:49.1385984Z Dynamo produced 2 graphs covering 1917 ops with 5 graph breaks (4 unique) 2024-06-26T07:36:53.0064774Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:36:53.0066443Z warnings.warn( 2024-06-26T07:36:53.6749505Z 2024-06-26T07:36:54.9761081Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:36:54.9761548Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:36:54.9762006Z cuda train MobileBertForMaskedLM 2024-06-26T07:37:22.2561474Z pass 2024-06-26T07:37:22.2886424Z TIMING: entire_frame_compile:24.05099 backend_compile:0.00038 2024-06-26T07:37:22.2887317Z STATS: call_* op count: 1449 | FakeTensorMode.__torch_dispatch__:32658 | FakeTensor.__torch_dispatch__:4391 2024-06-26T07:37:22.2888258Z Dynamo produced 1 graphs covering 1449 ops with 3 graph breaks (3 unique) 2024-06-26T07:37:26.0490687Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:37:26.0492348Z warnings.warn( 2024-06-26T07:37:26.2497587Z 2024-06-26T07:37:27.4207284Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:37:27.4207790Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:37:27.4210851Z cuda train MobileBertForQuestionAnswering 2024-06-26T07:37:54.8132248Z pass 2024-06-26T07:37:54.8353544Z TIMING: entire_frame_compile:24.14951 backend_compile:0.00038 2024-06-26T07:37:54.8354454Z STATS: call_* op count: 1453 | FakeTensorMode.__torch_dispatch__:32672 | FakeTensor.__torch_dispatch__:4384 2024-06-26T07:37:54.8355391Z Dynamo produced 1 graphs covering 1453 ops with 3 graph breaks (3 unique) 2024-06-26T07:37:58.5923746Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:37:58.5925425Z warnings.warn( 2024-06-26T07:37:58.7835921Z 2024-06-26T07:38:01.0852125Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:38:01.0852597Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:38:01.0853398Z cuda train OPTForCausalLM 2024-06-26T07:38:01.1054984Z WARNING:common:fp64 golden ref were not generated for OPTForCausalLM. Setting accuracy check to cosine 2024-06-26T07:38:05.0717239Z pass 2024-06-26T07:38:05.1298870Z TIMING: entire_frame_compile:1.95681 backend_compile:0.00082 2024-06-26T07:38:05.1299751Z STATS: call_* op count: 533 | FakeTensorMode.__torch_dispatch__:4449 | FakeTensor.__torch_dispatch__:715 2024-06-26T07:38:05.1300673Z Dynamo produced 17 graphs covering 533 ops with 6 graph breaks (5 unique) 2024-06-26T07:38:08.5005750Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:38:08.5007415Z warnings.warn( 2024-06-26T07:38:08.6961241Z 2024-06-26T07:38:10.4383526Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:38:10.4384624Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:38:10.4385438Z cuda train PLBartForCausalLM 2024-06-26T07:38:10.4557384Z WARNING:common:fp64 golden ref were not generated for PLBartForCausalLM. Setting accuracy check to cosine 2024-06-26T07:38:12.3797619Z pass 2024-06-26T07:38:12.4227408Z TIMING: entire_frame_compile:1.19974 backend_compile:0.00062 2024-06-26T07:38:12.4228351Z STATS: call_* op count: 250 | FakeTensorMode.__torch_dispatch__:2470 | FakeTensor.__torch_dispatch__:400 2024-06-26T07:38:12.4229335Z Dynamo produced 11 graphs covering 250 ops with 6 graph breaks (5 unique) 2024-06-26T07:38:15.7640594Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:38:15.7642470Z warnings.warn( 2024-06-26T07:38:15.9742036Z 2024-06-26T07:38:19.0685006Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:38:19.0685509Z loading model: 0it [00:03, ?it/s] 2024-06-26T07:38:19.0685990Z cuda train PLBartForConditionalGeneration 2024-06-26T07:38:19.1215925Z WARNING:common:fp64 golden ref were not generated for PLBartForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T07:38:23.2826904Z pass 2024-06-26T07:38:23.3570407Z TIMING: entire_frame_compile:2.68791 backend_compile:0.00094 2024-06-26T07:38:23.3571453Z STATS: call_* op count: 646 | FakeTensorMode.__torch_dispatch__:6076 | FakeTensor.__torch_dispatch__:1003 2024-06-26T07:38:23.3572394Z Dynamo produced 19 graphs covering 646 ops with 8 graph breaks (5 unique) 2024-06-26T07:38:26.7415741Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:38:26.7417496Z warnings.warn( 2024-06-26T07:38:27.0072057Z 2024-06-26T07:38:33.0290362Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:38:33.0290953Z loading model: 0it [00:06, ?it/s] 2024-06-26T07:38:33.0292344Z cuda train PegasusForCausalLM 2024-06-26T07:38:33.1526785Z WARNING:common:fp64 golden ref were not generated for PegasusForCausalLM. Setting accuracy check to cosine 2024-06-26T07:38:36.0184117Z pass 2024-06-26T07:38:36.0998923Z TIMING: entire_frame_compile:1.96138 backend_compile:0.00093 2024-06-26T07:38:36.1000041Z STATS: call_* op count: 476 | FakeTensorMode.__torch_dispatch__:4367 | FakeTensor.__torch_dispatch__:706 2024-06-26T07:38:36.1000977Z Dynamo produced 17 graphs covering 476 ops with 6 graph breaks (5 unique) 2024-06-26T07:38:39.4894136Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:38:39.4895797Z warnings.warn( 2024-06-26T07:38:39.6587997Z 2024-06-26T07:38:49.8928884Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:38:49.8929506Z loading model: 0it [00:10, ?it/s] 2024-06-26T07:38:49.8930026Z cuda train PegasusForConditionalGeneration 2024-06-26T07:38:49.9696526Z WARNING:common:fp64 golden ref were not generated for PegasusForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T07:38:56.3089341Z pass 2024-06-26T07:38:56.4199311Z TIMING: entire_frame_compile:4.38902 backend_compile:0.00151 2024-06-26T07:38:56.4200361Z STATS: call_* op count: 1234 | FakeTensorMode.__torch_dispatch__:11104 | FakeTensor.__torch_dispatch__:1807 2024-06-26T07:38:56.4201346Z Dynamo produced 30 graphs covering 1234 ops with 7 graph breaks (4 unique) 2024-06-26T07:38:59.8475513Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:38:59.8477799Z warnings.warn( 2024-06-26T07:39:00.0318977Z 2024-06-26T07:39:00.0355373Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2024-06-26T07:39:01.9341415Z We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2024-06-26T07:39:01.9343360Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2024-06-26T07:39:02.1584907Z 2024-06-26T07:39:02.1585604Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:39:02.1586796Z cuda train RobertaForCausalLM 2024-06-26T07:39:09.7846199Z W0626 07:39:09.783000 139787169149568 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:39:17.4973513Z pass 2024-06-26T07:39:17.5696241Z TIMING: entire_frame_compile:12.16111 backend_compile:0.00073 2024-06-26T07:39:17.5697978Z STATS: call_* op count: 1008 | FakeTensorMode.__torch_dispatch__:8859 | FakeTensor.__torch_dispatch__:5076 2024-06-26T07:39:17.5699850Z Dynamo produced 2 graphs covering 1008 ops with 5 graph breaks (4 unique) 2024-06-26T07:39:21.1503627Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:39:21.1505315Z warnings.warn( 2024-06-26T07:39:21.3506706Z 2024-06-26T07:39:22.8598129Z loading model: 0it [00:00, ?it/s]We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2024-06-26T07:39:22.8600249Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2024-06-26T07:39:23.0983629Z 2024-06-26T07:39:23.0984428Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:39:23.0985046Z cuda train RobertaForQuestionAnswering 2024-06-26T07:39:30.1266119Z W0626 07:39:30.125000 139848928313984 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:39:37.7598645Z pass 2024-06-26T07:39:37.8505335Z TIMING: entire_frame_compile:12.06156 backend_compile:0.00069 2024-06-26T07:39:37.8506226Z STATS: call_* op count: 1001 | FakeTensorMode.__torch_dispatch__:8818 | FakeTensor.__torch_dispatch__:4996 2024-06-26T07:39:37.8507154Z Dynamo produced 2 graphs covering 1001 ops with 5 graph breaks (4 unique) 2024-06-26T07:39:41.4096186Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:39:41.4097875Z warnings.warn( 2024-06-26T07:39:41.6302065Z 2024-06-26T07:39:42.1463036Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:39:42.1463652Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:39:42.1464195Z cuda train Speech2Text2ForCausalLM 2024-06-26T07:39:42.1627831Z WARNING:common:fp64 golden ref were not generated for Speech2Text2ForCausalLM. Setting accuracy check to cosine 2024-06-26T07:39:43.8908159Z pass 2024-06-26T07:39:43.8961062Z TIMING: entire_frame_compile:1.17529 backend_compile:0.00048 2024-06-26T07:39:43.8962081Z STATS: call_* op count: 258 | FakeTensorMode.__torch_dispatch__:2446 | FakeTensor.__torch_dispatch__:384 2024-06-26T07:39:43.8963010Z Dynamo produced 11 graphs covering 258 ops with 6 graph breaks (5 unique) 2024-06-26T07:39:47.2179722Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:39:47.2181871Z warnings.warn( 2024-06-26T07:39:47.4196485Z 2024-06-26T07:39:48.6691382Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:39:48.6691864Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:39:48.6692340Z cuda train T5ForConditionalGeneration 2024-06-26T07:39:57.4497542Z W0626 07:39:57.448000 140471063782016 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:40:03.1765930Z pass 2024-06-26T07:40:03.2347509Z TIMING: entire_frame_compile:10.72224 backend_compile:0.00058 2024-06-26T07:40:03.2348562Z STATS: call_* op count: 1224 | FakeTensorMode.__torch_dispatch__:10732 | FakeTensor.__torch_dispatch__:3112 2024-06-26T07:40:03.2349641Z Dynamo produced 2 graphs covering 1224 ops with 5 graph breaks (4 unique) 2024-06-26T07:40:06.7913566Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:40:06.7915329Z warnings.warn( 2024-06-26T07:40:07.0707194Z 2024-06-26T07:40:08.2970143Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:40:08.2970633Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:40:08.2971074Z cuda train T5Small 2024-06-26T07:40:17.1194363Z W0626 07:40:17.118000 140497468174976 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:40:22.7885001Z pass 2024-06-26T07:40:22.8467248Z TIMING: entire_frame_compile:10.68123 backend_compile:0.00058 2024-06-26T07:40:22.8468315Z STATS: call_* op count: 1224 | FakeTensorMode.__torch_dispatch__:10732 | FakeTensor.__torch_dispatch__:3112 2024-06-26T07:40:22.8469238Z Dynamo produced 2 graphs covering 1224 ops with 5 graph breaks (4 unique) 2024-06-26T07:40:26.3650435Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:40:26.3652327Z warnings.warn( 2024-06-26T07:40:26.6108533Z 2024-06-26T07:40:30.1926066Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:40:30.1926556Z loading model: 0it [00:03, ?it/s] 2024-06-26T07:40:30.1927011Z cuda train TrOCRForCausalLM 2024-06-26T07:40:30.2239747Z WARNING:common:fp64 golden ref were not generated for TrOCRForCausalLM. Setting accuracy check to cosine 2024-06-26T07:40:33.2144585Z pass 2024-06-26T07:40:33.3440021Z TIMING: entire_frame_compile:1.9797 backend_compile:0.0008 2024-06-26T07:40:33.3440938Z STATS: call_* op count: 477 | FakeTensorMode.__torch_dispatch__:4436 | FakeTensor.__torch_dispatch__:724 2024-06-26T07:40:33.3442441Z Dynamo produced 17 graphs covering 477 ops with 6 graph breaks (5 unique) 2024-06-26T07:40:36.7032452Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:40:36.7034131Z warnings.warn( 2024-06-26T07:40:36.9153734Z 2024-06-26T07:40:46.0806214Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:40:46.0806670Z loading model: 0it [00:09, ?it/s] 2024-06-26T07:40:46.0807109Z cuda train XGLMForCausalLM 2024-06-26T07:40:46.2251669Z WARNING:common:fp64 golden ref were not generated for XGLMForCausalLM. Setting accuracy check to cosine 2024-06-26T07:40:51.5422593Z pass 2024-06-26T07:40:51.7145355Z TIMING: entire_frame_compile:3.50621 backend_compile:0.00143 2024-06-26T07:40:51.7146216Z STATS: call_* op count: 987 | FakeTensorMode.__torch_dispatch__:8572 | FakeTensor.__torch_dispatch__:1356 2024-06-26T07:40:51.7147393Z Dynamo produced 29 graphs covering 987 ops with 6 graph breaks (5 unique) 2024-06-26T07:40:55.1480203Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:40:55.1481937Z warnings.warn( 2024-06-26T07:40:55.3342685Z 2024-06-26T07:40:59.4530045Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:40:59.4530636Z loading model: 0it [00:04, ?it/s] 2024-06-26T07:40:59.4531172Z cuda train XLNetLMHeadModel 2024-06-26T07:41:18.0705615Z W0626 07:41:18.069000 140313829958272 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:41:35.5677968Z pass 2024-06-26T07:41:35.8279684Z TIMING: entire_frame_compile:26.90765 backend_compile:0.00122 2024-06-26T07:41:35.8280599Z STATS: call_* op count: 1874 | FakeTensorMode.__torch_dispatch__:26627 | FakeTensor.__torch_dispatch__:8065 2024-06-26T07:41:35.8281525Z Dynamo produced 2 graphs covering 1874 ops with 5 graph breaks (4 unique) 2024-06-26T07:41:39.7752504Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:41:39.7754174Z warnings.warn( 2024-06-26T07:41:39.9590961Z 2024-06-26T07:41:41.9984881Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:41:41.9985444Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:41:41.9985932Z cuda train YituTechConvBert 2024-06-26T07:41:51.4782197Z W0626 07:41:51.477000 140635676058240 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:42:02.9426429Z pass 2024-06-26T07:42:03.0131934Z TIMING: entire_frame_compile:17.95243 backend_compile:0.00096 2024-06-26T07:42:03.0133994Z STATS: call_* op count: 1512 | FakeTensorMode.__torch_dispatch__:13046 | FakeTensor.__torch_dispatch__:7110 2024-06-26T07:42:03.0136339Z Dynamo produced 2 graphs covering 1512 ops with 5 graph breaks (4 unique) 2024-06-26T07:42:04.1347935Z accuracy pass_rate=93.48% 2024-06-26T07:42:04.1360765Z calls_captured gmean=0.00x mean=920.348x 2024-06-26T07:42:04.1364339Z unique_graphs gmean=0.00x mean=7.478x 2024-06-26T07:42:04.1368074Z graph_breaks gmean=0.00x mean=5.196x 2024-06-26T07:42:04.1371606Z unique_graph_breaks gmean=0.00x mean=4.043x 2024-06-26T07:42:04.1375906Z autograd_captures gmean=0.00x mean=0.000x 2024-06-26T07:42:04.1379356Z autograd_compiles gmean=0.00x mean=0.000x 2024-06-26T07:42:04.1383199Z cudagraph_skips gmean=0.00x mean=0.000x 2024-06-26T07:42:04.6995208Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy//dynamo_eager_huggingface_training.csv 2024-06-26T07:42:04.9532922Z AlbertForMaskedLM PASS 2024-06-26T07:42:04.9536828Z AlbertForQuestionAnswering PASS 2024-06-26T07:42:04.9541637Z AllenaiLongformerBase PASS 2024-06-26T07:42:04.9546326Z BartForCausalLM PASS 2024-06-26T07:42:04.9551573Z BartForConditionalGeneration PASS 2024-06-26T07:42:04.9556607Z BertForMaskedLM PASS 2024-06-26T07:42:04.9561928Z BertForQuestionAnswering PASS 2024-06-26T07:42:04.9567153Z BlenderbotForCausalLM XFAIL 2024-06-26T07:42:04.9572369Z BlenderbotSmallForCausalLM PASS 2024-06-26T07:42:04.9577616Z BlenderbotSmallForConditionalGeneration PASS 2024-06-26T07:42:04.9582679Z CamemBert PASS 2024-06-26T07:42:04.9587927Z DebertaForMaskedLM PASS 2024-06-26T07:42:04.9593065Z DebertaForQuestionAnswering PASS 2024-06-26T07:42:04.9598499Z DebertaV2ForMaskedLM XFAIL 2024-06-26T07:42:04.9603411Z DebertaV2ForQuestionAnswering XFAIL 2024-06-26T07:42:04.9608372Z DistilBertForMaskedLM PASS 2024-06-26T07:42:04.9613325Z DistilBertForQuestionAnswering PASS 2024-06-26T07:42:04.9618262Z DistillGPT2 PASS 2024-06-26T07:42:04.9623237Z ElectraForCausalLM PASS 2024-06-26T07:42:04.9628200Z ElectraForQuestionAnswering PASS 2024-06-26T07:42:04.9633153Z GPT2ForSequenceClassification PASS 2024-06-26T07:42:04.9638469Z GoogleFnet PASS 2024-06-26T07:42:04.9643441Z LayoutLMForMaskedLM PASS 2024-06-26T07:42:04.9648449Z LayoutLMForSequenceClassification PASS 2024-06-26T07:42:04.9653478Z M2M100ForConditionalGeneration PASS 2024-06-26T07:42:04.9658460Z MBartForCausalLM PASS 2024-06-26T07:42:04.9663394Z MBartForConditionalGeneration PASS 2024-06-26T07:42:04.9668407Z MT5ForConditionalGeneration PASS 2024-06-26T07:42:04.9673401Z MegatronBertForCausalLM PASS 2024-06-26T07:42:04.9678770Z MegatronBertForQuestionAnswering PASS 2024-06-26T07:42:04.9683650Z MobileBertForMaskedLM PASS 2024-06-26T07:42:04.9688672Z MobileBertForQuestionAnswering PASS 2024-06-26T07:42:04.9693596Z OPTForCausalLM PASS 2024-06-26T07:42:04.9698590Z PLBartForCausalLM PASS 2024-06-26T07:42:04.9703549Z PLBartForConditionalGeneration PASS 2024-06-26T07:42:04.9708541Z PegasusForCausalLM PASS 2024-06-26T07:42:04.9713706Z PegasusForConditionalGeneration PASS 2024-06-26T07:42:04.9718946Z RobertaForCausalLM PASS 2024-06-26T07:42:04.9723915Z RobertaForQuestionAnswering PASS 2024-06-26T07:42:04.9728863Z Speech2Text2ForCausalLM PASS 2024-06-26T07:42:04.9733846Z T5ForConditionalGeneration PASS 2024-06-26T07:42:04.9738769Z T5Small PASS 2024-06-26T07:42:04.9743724Z TrOCRForCausalLM PASS 2024-06-26T07:42:04.9748654Z XGLMForCausalLM PASS 2024-06-26T07:42:04.9753824Z XLNetLMHeadModel PASS 2024-06-26T07:42:04.9759164Z YituTechConvBert PASS 2024-06-26T07:42:05.0133848Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy//dynamo_eager_huggingface_training.csv 2024-06-26T07:42:05.2984999Z AlbertForMaskedLM PASS 2024-06-26T07:42:05.2986655Z AlbertForQuestionAnswering PASS 2024-06-26T07:42:05.2991678Z AllenaiLongformerBase PASS 2024-06-26T07:42:05.2996751Z BartForCausalLM PASS 2024-06-26T07:42:05.3002328Z BartForConditionalGeneration PASS 2024-06-26T07:42:05.3007699Z BertForMaskedLM PASS 2024-06-26T07:42:05.3013063Z BertForQuestionAnswering PASS 2024-06-26T07:42:05.3018414Z BlenderbotForCausalLM PASS 2024-06-26T07:42:05.3023820Z BlenderbotSmallForCausalLM PASS 2024-06-26T07:42:05.3029202Z BlenderbotSmallForConditionalGeneration PASS 2024-06-26T07:42:05.3034417Z CamemBert PASS 2024-06-26T07:42:05.3039736Z DebertaForMaskedLM PASS 2024-06-26T07:42:05.3044192Z DebertaForQuestionAnswering PASS 2024-06-26T07:42:05.3048899Z DebertaV2ForMaskedLM PASS 2024-06-26T07:42:05.3053533Z DebertaV2ForQuestionAnswering PASS 2024-06-26T07:42:05.3058172Z DistilBertForMaskedLM PASS 2024-06-26T07:42:05.3063025Z DistilBertForQuestionAnswering PASS 2024-06-26T07:42:05.3067954Z DistillGPT2 PASS 2024-06-26T07:42:05.3072810Z ElectraForCausalLM PASS 2024-06-26T07:42:05.3077731Z ElectraForQuestionAnswering PASS 2024-06-26T07:42:05.3082554Z GPT2ForSequenceClassification PASS 2024-06-26T07:42:05.3087418Z GoogleFnet PASS 2024-06-26T07:42:05.3095645Z LayoutLMForMaskedLM PASS 2024-06-26T07:42:05.3096956Z LayoutLMForSequenceClassification PASS 2024-06-26T07:42:05.3101658Z M2M100ForConditionalGeneration PASS 2024-06-26T07:42:05.3106400Z MBartForCausalLM PASS 2024-06-26T07:42:05.3111366Z MBartForConditionalGeneration PASS 2024-06-26T07:42:05.3116425Z MT5ForConditionalGeneration PASS 2024-06-26T07:42:05.3121290Z MegatronBertForCausalLM PASS 2024-06-26T07:42:05.3125854Z MegatronBertForQuestionAnswering PASS 2024-06-26T07:42:05.3130651Z MobileBertForMaskedLM PASS 2024-06-26T07:42:05.3135348Z MobileBertForQuestionAnswering PASS 2024-06-26T07:42:05.3140165Z OPTForCausalLM PASS 2024-06-26T07:42:05.3144937Z PLBartForCausalLM PASS 2024-06-26T07:42:05.3149690Z PLBartForConditionalGeneration PASS 2024-06-26T07:42:05.3154651Z PegasusForCausalLM PASS 2024-06-26T07:42:05.3159844Z PegasusForConditionalGeneration PASS 2024-06-26T07:42:05.3164453Z RobertaForCausalLM PASS 2024-06-26T07:42:05.3169284Z RobertaForQuestionAnswering PASS 2024-06-26T07:42:05.3173951Z Speech2Text2ForCausalLM PASS 2024-06-26T07:42:05.3178812Z T5ForConditionalGeneration PASS 2024-06-26T07:42:05.3183492Z T5Small PASS 2024-06-26T07:42:05.3188297Z TrOCRForCausalLM PASS 2024-06-26T07:42:05.3193111Z XGLMForCausalLM PASS 2024-06-26T07:42:05.3198221Z XLNetLMHeadModel PASS 2024-06-26T07:42:05.3202886Z YituTechConvBert PASS 2024-06-26T07:42:05.3597117Z + cleanup_workspace 2024-06-26T07:42:05.3598033Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-06-26T07:42:05.3599336Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-06-26T07:42:05.3600318Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-06-26T07:42:05.3601036Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-06-26T07:42:05.3601844Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-06-26T07:42:05.3602713Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-06-26T07:42:05.3603399Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-06-26T07:42:06.0893055Z ##[group]Run cat test/**/*_toprint.log || true 2024-06-26T07:42:06.0893583Z cat test/**/*_toprint.log || true 2024-06-26T07:42:06.0901793Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:42:06.0902299Z env: 2024-06-26T07:42:06.0902571Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:42:06.0903031Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:42:06.0903769Z DOCKER_CONTAINER_ID: fe4fb56dd485413b71c9d62a9524407caf78d7d40e572f4e0319fe63d37a8796 2024-06-26T07:42:06.0904427Z ##[endgroup] 2024-06-26T07:42:06.0967011Z cat: test/**/*_toprint.log: No such file or directory 2024-06-26T07:42:06.0995600Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-06-26T07:42:06.0996256Z kill "$MONITOR_SCRIPT_PID" 2024-06-26T07:42:06.1003777Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:42:06.1004281Z env: 2024-06-26T07:42:06.1004563Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:42:06.1005022Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:42:06.1005762Z DOCKER_CONTAINER_ID: fe4fb56dd485413b71c9d62a9524407caf78d7d40e572f4e0319fe63d37a8796 2024-06-26T07:42:06.1006420Z MONITOR_SCRIPT_PID: 14172 2024-06-26T07:42:06.1006779Z ##[endgroup] 2024-06-26T07:42:06.1190075Z Prepare all required actions 2024-06-26T07:42:06.1190528Z Getting action download info 2024-06-26T07:42:06.2795556Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-06-26T07:42:06.4403617Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-06-26T07:42:06.4404086Z with: 2024-06-26T07:42:06.4404645Z file-suffix: test-dynamo_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688915732 2024-06-26T07:42:06.4405336Z s3-bucket: gha-artifacts 2024-06-26T07:42:06.4405674Z env: 2024-06-26T07:42:06.4405939Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:42:06.4406388Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:42:06.4407128Z DOCKER_CONTAINER_ID: fe4fb56dd485413b71c9d62a9524407caf78d7d40e572f4e0319fe63d37a8796 2024-06-26T07:42:06.4407775Z ##[endgroup] 2024-06-26T07:42:06.4432888Z ##[group]Run # Remove any previous test jsons if they exist 2024-06-26T07:42:06.4433536Z # Remove any previous test jsons if they exist 2024-06-26T07:42:06.4434059Z rm -f test-jsons-*.zip 2024-06-26T07:42:06.4434580Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-06-26T07:42:06.4442531Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:42:06.4443042Z env: 2024-06-26T07:42:06.4443317Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:42:06.4443761Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:42:06.4444491Z DOCKER_CONTAINER_ID: fe4fb56dd485413b71c9d62a9524407caf78d7d40e572f4e0319fe63d37a8796 2024-06-26T07:42:06.4445408Z FILE_SUFFIX: test-dynamo_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688915732 2024-06-26T07:42:06.4446072Z ##[endgroup] 2024-06-26T07:42:06.4618495Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-06-26T07:42:06.4647756Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-06-26T07:42:06.4648448Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-06-26T07:42:06.4654304Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-06-26T07:42:06.4679971Z ##[group]Run # Remove any previous test reports if they exist 2024-06-26T07:42:06.4680625Z # Remove any previous test reports if they exist 2024-06-26T07:42:06.4681164Z rm -f test-reports-*.zip 2024-06-26T07:42:06.4681755Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-06-26T07:42:06.4689239Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:42:06.4689730Z env: 2024-06-26T07:42:06.4690008Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:42:06.4690465Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:42:06.4691194Z DOCKER_CONTAINER_ID: fe4fb56dd485413b71c9d62a9524407caf78d7d40e572f4e0319fe63d37a8796 2024-06-26T07:42:06.4692101Z FILE_SUFFIX: test-dynamo_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688915732 2024-06-26T07:42:06.4692770Z ##[endgroup] 2024-06-26T07:42:06.4844071Z adding: test/test-reports/inference_huggingface.csv (deflated 76%) 2024-06-26T07:42:06.4845275Z adding: test/test-reports/inference_huggingface_graph_breaks.csv (deflated 89%) 2024-06-26T07:42:06.4846266Z adding: test/test-reports/inference_huggingface_graph_break_deduped.csv (deflated 78%) 2024-06-26T07:42:06.4847148Z adding: test/test-reports/training_huggingface.csv (deflated 73%) 2024-06-26T07:42:06.4853100Z adding: test/test-reports/training_huggingface_graph_breaks.csv (deflated 97%) 2024-06-26T07:42:06.4854313Z adding: test/test-reports/training_huggingface_graph_break_deduped.csv (deflated 77%) 2024-06-26T07:42:06.4879731Z ##[group]Run # Remove any previous usage logs if they exist 2024-06-26T07:42:06.4880364Z # Remove any previous usage logs if they exist 2024-06-26T07:42:06.4880871Z rm -f logs-*.zip 2024-06-26T07:42:06.4881542Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-06-26T07:42:06.4882458Z # so check to see if the file exists first 2024-06-26T07:42:06.4882969Z if [ -f 'usage_log.txt' ]; then 2024-06-26T07:42:06.4883508Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-06-26T07:42:06.4883996Z fi 2024-06-26T07:42:06.4884363Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-06-26T07:42:06.4885061Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-06-26T07:42:06.4885558Z fi 2024-06-26T07:42:06.4895834Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:42:06.4896340Z env: 2024-06-26T07:42:06.4896609Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:42:06.4897066Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:42:06.4897806Z DOCKER_CONTAINER_ID: fe4fb56dd485413b71c9d62a9524407caf78d7d40e572f4e0319fe63d37a8796 2024-06-26T07:42:06.4898718Z FILE_SUFFIX: test-dynamo_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688915732 2024-06-26T07:42:06.4899391Z ##[endgroup] 2024-06-26T07:42:06.4942188Z adding: usage_log.txt (deflated 92%) 2024-06-26T07:42:06.5007142Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-06-26T07:42:06.5007974Z # Remove any previous debugging artifacts if they exist 2024-06-26T07:42:06.5008526Z rm -f debug-*.zip 2024-06-26T07:42:06.5008920Z if [ -d 'test/debug' ]; then 2024-06-26T07:42:06.5009415Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-06-26T07:42:06.5009893Z fi 2024-06-26T07:42:06.5016598Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:42:06.5017093Z env: 2024-06-26T07:42:06.5017361Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:42:06.5017810Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:42:06.5018548Z DOCKER_CONTAINER_ID: fe4fb56dd485413b71c9d62a9524407caf78d7d40e572f4e0319fe63d37a8796 2024-06-26T07:42:06.5019510Z FILE_SUFFIX: test-dynamo_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688915732 2024-06-26T07:42:06.5020173Z ##[endgroup] 2024-06-26T07:42:06.5094739Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T07:42:06.5095178Z with: 2024-06-26T07:42:06.5095480Z s3-bucket: gha-artifacts 2024-06-26T07:42:06.5095927Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T07:42:06.5096417Z retention-days: 14 2024-06-26T07:42:06.5096757Z if-no-files-found: warn 2024-06-26T07:42:06.5097122Z path: test-jsons-*.zip 2024-06-26T07:42:06.5097473Z name: artifact 2024-06-26T07:42:06.5097775Z region: us-east-1 2024-06-26T07:42:06.5098082Z env: 2024-06-26T07:42:06.5098359Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:42:06.5098808Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:42:06.5099567Z DOCKER_CONTAINER_ID: fe4fb56dd485413b71c9d62a9524407caf78d7d40e572f4e0319fe63d37a8796 2024-06-26T07:42:06.5100228Z ##[endgroup] 2024-06-26T07:42:06.8189958Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T07:42:06.8190565Z With the provided path, there will be 1 file uploaded 2024-06-26T07:42:06.8191537Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T07:42:06.8217105Z Starting upload of test-jsons-test-dynamo_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688915732.zip 2024-06-26T07:42:06.9557988Z Finished upload of test-jsons-test-dynamo_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688915732.zip 2024-06-26T07:42:06.9678077Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T07:42:06.9678525Z with: 2024-06-26T07:42:06.9678817Z s3-bucket: gha-artifacts 2024-06-26T07:42:06.9679284Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T07:42:06.9679776Z retention-days: 14 2024-06-26T07:42:06.9680109Z if-no-files-found: error 2024-06-26T07:42:06.9680482Z path: test-reports-*.zip 2024-06-26T07:42:06.9680835Z name: artifact 2024-06-26T07:42:06.9681134Z region: us-east-1 2024-06-26T07:42:06.9681440Z env: 2024-06-26T07:42:06.9681715Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:42:06.9682168Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:42:06.9683021Z DOCKER_CONTAINER_ID: fe4fb56dd485413b71c9d62a9524407caf78d7d40e572f4e0319fe63d37a8796 2024-06-26T07:42:06.9683668Z ##[endgroup] 2024-06-26T07:42:07.2784911Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T07:42:07.2785599Z With the provided path, there will be 1 file uploaded 2024-06-26T07:42:07.2786521Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T07:42:07.2811580Z Starting upload of test-reports-test-dynamo_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688915732.zip 2024-06-26T07:42:07.4007008Z Finished upload of test-reports-test-dynamo_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688915732.zip 2024-06-26T07:42:07.4145520Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T07:42:07.4145968Z with: 2024-06-26T07:42:07.4146258Z s3-bucket: gha-artifacts 2024-06-26T07:42:07.4146689Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T07:42:07.4147151Z retention-days: 14 2024-06-26T07:42:07.4147503Z if-no-files-found: ignore 2024-06-26T07:42:07.4147861Z path: logs-*.zip 2024-06-26T07:42:07.4148175Z name: artifact 2024-06-26T07:42:07.4148474Z region: us-east-1 2024-06-26T07:42:07.4148770Z env: 2024-06-26T07:42:07.4149042Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:42:07.4149503Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:42:07.4150290Z DOCKER_CONTAINER_ID: fe4fb56dd485413b71c9d62a9524407caf78d7d40e572f4e0319fe63d37a8796 2024-06-26T07:42:07.4150930Z ##[endgroup] 2024-06-26T07:42:07.7195181Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T07:42:07.7195973Z With the provided path, there will be 1 file uploaded 2024-06-26T07:42:07.7196607Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T07:42:07.7221668Z Starting upload of logs-test-dynamo_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688915732.zip 2024-06-26T07:42:07.8124624Z Finished upload of logs-test-dynamo_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688915732.zip 2024-06-26T07:42:07.8247470Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T07:42:07.8247910Z with: 2024-06-26T07:42:07.8248194Z s3-bucket: gha-artifacts 2024-06-26T07:42:07.8248620Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T07:42:07.8249087Z retention-days: 14 2024-06-26T07:42:07.8249436Z if-no-files-found: ignore 2024-06-26T07:42:07.8249797Z path: debug-*.zip 2024-06-26T07:42:07.8250106Z name: artifact 2024-06-26T07:42:07.8250403Z region: us-east-1 2024-06-26T07:42:07.8250697Z env: 2024-06-26T07:42:07.8250970Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:42:07.8251419Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:42:07.8252145Z DOCKER_CONTAINER_ID: fe4fb56dd485413b71c9d62a9524407caf78d7d40e572f4e0319fe63d37a8796 2024-06-26T07:42:07.8252784Z ##[endgroup] 2024-06-26T07:42:08.1234954Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-06-26T07:42:08.1357037Z ##[group]Run # shellcheck disable=SC2156 2024-06-26T07:42:08.1357519Z # shellcheck disable=SC2156 2024-06-26T07:42:08.1358317Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-06-26T07:42:08.1366138Z shell: /usr/bin/bash -e {0} 2024-06-26T07:42:08.1366506Z env: 2024-06-26T07:42:08.1366777Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:42:08.1367232Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:42:08.1367953Z DOCKER_CONTAINER_ID: fe4fb56dd485413b71c9d62a9524407caf78d7d40e572f4e0319fe63d37a8796 2024-06-26T07:42:08.1368585Z ##[endgroup] 2024-06-26T07:42:08.2976350Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-06-26T07:42:08.2976912Z with: 2024-06-26T07:42:08.2977177Z env: 2024-06-26T07:42:08.2977457Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:42:08.2977909Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:42:08.2978645Z DOCKER_CONTAINER_ID: fe4fb56dd485413b71c9d62a9524407caf78d7d40e572f4e0319fe63d37a8796 2024-06-26T07:42:08.2979291Z ##[endgroup] 2024-06-26T07:42:08.2996810Z ##[group]Run set -eou pipefail 2024-06-26T07:42:08.2997225Z set -eou pipefail 2024-06-26T07:42:08.2997578Z  2024-06-26T07:42:08.2998087Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-06-26T07:42:08.2998852Z for _ in $(seq 1440); do 2024-06-26T07:42:08.2999330Z  # Break if no ssh session exists anymore 2024-06-26T07:42:08.2999835Z  if [ "$(who)" = "" ]; then 2024-06-26T07:42:08.3000248Z  break 2024-06-26T07:42:08.3000592Z  fi 2024-06-26T07:42:08.3000888Z  echo "." 2024-06-26T07:42:08.3001217Z  sleep 5 2024-06-26T07:42:08.3001533Z done 2024-06-26T07:42:08.3008895Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:42:08.3009386Z env: 2024-06-26T07:42:08.3009659Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:42:08.3010108Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:42:08.3010840Z DOCKER_CONTAINER_ID: fe4fb56dd485413b71c9d62a9524407caf78d7d40e572f4e0319fe63d37a8796 2024-06-26T07:42:08.3011473Z ##[endgroup] 2024-06-26T07:42:08.3033088Z Holding runner for 2 hours until all ssh sessions have logged out 2024-06-26T07:42:08.3068739Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:42:08.3069547Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:42:08.3070145Z # shellcheck disable=SC2046 2024-06-26T07:42:08.3070625Z docker stop $(docker ps -q) || true 2024-06-26T07:42:08.3071318Z # Prune all of the docker images 2024-06-26T07:42:08.3071807Z docker system prune -af 2024-06-26T07:42:08.3079800Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:42:08.3080337Z env: 2024-06-26T07:42:08.3080615Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:42:08.3081063Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:42:08.3081791Z DOCKER_CONTAINER_ID: fe4fb56dd485413b71c9d62a9524407caf78d7d40e572f4e0319fe63d37a8796 2024-06-26T07:42:08.3082420Z ##[endgroup] 2024-06-26T07:42:08.6072946Z fe4fb56dd485 2024-06-26T07:42:09.0047476Z Deleted Containers: 2024-06-26T07:42:09.0048061Z fe4fb56dd485413b71c9d62a9524407caf78d7d40e572f4e0319fe63d37a8796 2024-06-26T07:42:09.0048496Z 2024-06-26T07:42:13.9969123Z Deleted Images: 2024-06-26T07:42:13.9971085Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:42:13.9973700Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T07:42:13.9975453Z deleted: sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd 2024-06-26T07:42:13.9976294Z deleted: sha256:754ead7012f0b7691885a9660f02e80be6cee8c5665486fb995c1fb06adb6e92 2024-06-26T07:42:13.9977130Z deleted: sha256:fbe4496d8830e11be86d8f222522e7956f32f85bfc06b05ea53961f34a58af74 2024-06-26T07:42:13.9977971Z deleted: sha256:1c83814aa1a1a7e0cc998e3479bdab8e8766b975560cbf00690acf39ed899a80 2024-06-26T07:42:13.9978826Z deleted: sha256:e9ea8e61942c7f8ae4aab97b8d102b01d6bb4b6cf2a605808cd34130a17d8a0f 2024-06-26T07:42:13.9979671Z deleted: sha256:acc5fc32588cd95e082a6ce35e48d0f303ae8f96e3ab65c4e375b85e1f3c4a92 2024-06-26T07:42:13.9980502Z deleted: sha256:a93e53400904d29e4185d3be942ea34f19572ea438962788ae4e9b3d9bf4c498 2024-06-26T07:42:13.9981605Z deleted: sha256:c97092b88b5e4c77c68bbc4b51afab64e38719af1ff58013ec49a696cdd40a6e 2024-06-26T07:42:13.9982439Z deleted: sha256:b132889c90d67ffa53655846603e3fcc2372c23f04dbdbc54690a39d304eb89b 2024-06-26T07:42:13.9983272Z deleted: sha256:915d54081ed607710624abe3f2490114c745858831f4fc13396bd3b4b9a121bc 2024-06-26T07:42:13.9984093Z deleted: sha256:bfa8d446c498c87f535aef102695564de1b22c755e6c63944ae2f5073a2b31ec 2024-06-26T07:42:13.9984916Z deleted: sha256:e71460fb363c29e70bbe357d680aaa7ae694f094ef2f2713609158ad90611744 2024-06-26T07:42:13.9985743Z deleted: sha256:84079bb1b4ee1d0119ae3d3e701de460a1324912297c675425c9a36d30fcdb0f 2024-06-26T07:42:13.9986572Z deleted: sha256:d86ac3ca148d4758cb03b493fdd49109d8f1a2c7c85c4b755493c3c4529553e5 2024-06-26T07:42:13.9987528Z deleted: sha256:532b3ab97c44a673a0d5f62ee46fa6a83248a84de8877754a612355be56ed0a3 2024-06-26T07:42:13.9988346Z deleted: sha256:617e2a35f25ae24500c88c0c9b8475f0687fbcbed3d200013b4c99c6d8941aff 2024-06-26T07:42:13.9989181Z deleted: sha256:b88120bcd5a26a660c09c5df13176e0ef098fb2c8cdb57d74ab0796a3d1194d6 2024-06-26T07:42:13.9990006Z deleted: sha256:c016b2e7764735243205fb76fa9a273bb3dd4bf05532288e3a613ce022131ac3 2024-06-26T07:42:13.9990840Z deleted: sha256:bcaca462c390deddd2995de2d245a7495e7d2e1e94f0edad30a573c83d0588fd 2024-06-26T07:42:13.9991951Z deleted: sha256:9e260eba003d5c25d7b410e5c4c3093845d9e46611896840901c689d7fbf8ee6 2024-06-26T07:42:13.9992767Z deleted: sha256:3e62b86f8b911a9b59c62d55f063e71683ae6a01c22418db5d27930712adaad6 2024-06-26T07:42:13.9993587Z deleted: sha256:905920ccb1d88449b8dde686c34a4b2a8b6f59e78ac44916b189ea3593b9fbbb 2024-06-26T07:42:13.9994402Z deleted: sha256:961201237e03acd5061406238dd7a3c1343ac888d7583212cd0ab5ae1b4f0540 2024-06-26T07:42:13.9995231Z deleted: sha256:70047a2aac62159fa28fb4dbd4567ec2ffbab68fe046967e09acb0eb3e5cb9cb 2024-06-26T07:42:13.9996159Z deleted: sha256:1ec680ad781436ae65fa987f5fa1ed032ecd5c82a6e35eea6707de501f62c0f4 2024-06-26T07:42:13.9997002Z deleted: sha256:ef859623a50e85abf6e6c1dbd6188aa765a66cc7b2bd618a1cbe4825b7b2bd91 2024-06-26T07:42:13.9997852Z deleted: sha256:404ade5cacff59773b93ab438bfa64cd3a171d07f6f168bdc4a5d4bfa0b43526 2024-06-26T07:42:13.9998686Z deleted: sha256:5b27de6ee3d9f2f18217bf18dec725f7b4954c0e48c85d35752634c22d62209b 2024-06-26T07:42:13.9999615Z deleted: sha256:498e39c575b3fa069acaac7e46b91b4e54ae3a05f2ada5c4347091f346f25d4c 2024-06-26T07:42:14.0000428Z deleted: sha256:1405919d2694d65669b1f4c842282021afe89208ce86570e96464903733da2d0 2024-06-26T07:42:14.0001233Z deleted: sha256:d5b41e5d70896c0472fc3e6da0516a17d89cd6635dc75010ecddc31ce9690fad 2024-06-26T07:42:14.0002058Z deleted: sha256:18e9379b62b9d0bb40293d28a652eafaee66a499f66f0776bf14e37a513101b7 2024-06-26T07:42:14.0002884Z deleted: sha256:a781f80ed00ff2926a699fd01a309bb0b4fb7d5655eb652c205bb9c7f811035f 2024-06-26T07:42:14.0003705Z deleted: sha256:c0430973b7cf759075330ee29c79bfd1312c91e6033d9dd883b4b1c29743a171 2024-06-26T07:42:14.0004535Z deleted: sha256:e733b3d66efb9342e5a283376c0e1ddffa47a42fecc26e61ecaf0a3472885d97 2024-06-26T07:42:14.0005361Z deleted: sha256:8929c0b2047f74b84c82c39eff9077a92ae1a828cde00c9ab78fb3e0ed5e0283 2024-06-26T07:42:14.0006176Z deleted: sha256:f5b56155fb58a0a60fe6391d3581cf91610ed69d4f0a321257c19944da296a52 2024-06-26T07:42:14.0006990Z deleted: sha256:b54d2cb355238d1aa91e58df93064447b80f53706d1ca62823acc834f5ac54fe 2024-06-26T07:42:14.0007806Z deleted: sha256:e7311285af9eeddb51aa5629aa98386e89f1456ff32843b6aee6443247edf66c 2024-06-26T07:42:14.0008630Z deleted: sha256:67657ceec347da8dc8085d5731633efeac115ba677fa4b7c5966c4260f7abab0 2024-06-26T07:42:14.0009446Z deleted: sha256:b496e50409dbf414930fc9b4437aa0aa4ce27bc7840e050ae459aaffcf9aca86 2024-06-26T07:42:14.0010262Z deleted: sha256:8f5276170cc4a60d36ad7b964a502306b29546254cedcbcb101ea9123497d2c0 2024-06-26T07:42:14.0011118Z deleted: sha256:53d524d1564745e96a7e515462171b73184d9f1ff5454b32fcfbbca39d361467 2024-06-26T07:42:14.0011946Z deleted: sha256:2acbadb00d69b6aca575f04baba4ea26ae2b5c2d85f09945004ad93a78a8eb75 2024-06-26T07:42:14.0012909Z deleted: sha256:02665efeff878aeb603c1edb2e21cb700c171c025d64f5aae1b2b20b12f3964a 2024-06-26T07:42:14.0013755Z deleted: sha256:19bd23e9a51dfe883a8b8d2ff6d66ff1b3476b8912856406bb79b81762aa6d4b 2024-06-26T07:42:14.0014579Z deleted: sha256:73aa8637bf9299d6298dabd858480c274a62117388f2ca1d5e9eb15c6d1e4155 2024-06-26T07:42:14.0015397Z deleted: sha256:5f8968406305f348b1caebc0e035187d6e9965ace7b9d5f2f21114d37c9fe2e1 2024-06-26T07:42:14.0016219Z deleted: sha256:d827104914db0c503cf5c81f13e2456bf4efce42d4cf1f8d5d70838207278bee 2024-06-26T07:42:14.0017043Z deleted: sha256:867b219724f2f1aaf56f42e3e9f03a5d64bdca8a6a72407efd10328aeea5d510 2024-06-26T07:42:14.0017878Z deleted: sha256:b9ff2a7cceb9237b26bcb17728edf8a558eec5a76f63626e84f80541dfc0f1d4 2024-06-26T07:42:14.0018787Z deleted: sha256:be73f509106819007ce2b0f5a1301ac56ae583cb6c1fb69bee93efa0840368f1 2024-06-26T07:42:14.0019622Z deleted: sha256:2a11c07c915c3fdbb4e73096dea73ea950c970d50fe1a5764060bd62b7b7cf2c 2024-06-26T07:42:14.0020458Z deleted: sha256:5e9989622e2abc1c72f8184343507b45d75e4ff971edb68b96cbcd75ae484ff5 2024-06-26T07:42:14.0021347Z deleted: sha256:05ce642bd9cf5a9205bff107df28b1d6cef7d5f6ab3ea944d28c032ec61270df 2024-06-26T07:42:14.0022173Z deleted: sha256:017702617f6c07b73f2088678f4710b794cfe897a213a5cf122a0e2519eae0ac 2024-06-26T07:42:14.0022987Z deleted: sha256:44b3b37bb1ca39dc5cf356de41a23a5c12f605a1992813e408087fb2b0ad6d1c 2024-06-26T07:42:14.0023809Z deleted: sha256:8c387982e9f27db357a1d149dc149cabc0bd04668ad6956387a5072682dada67 2024-06-26T07:42:14.0024632Z deleted: sha256:10d415dca4e3fbb88a747b40317f3c6333c041ba8f73855f67fb9fe981fae5ee 2024-06-26T07:42:14.0025468Z deleted: sha256:6c65469bf0f53dcd625faefe363b8f3136c08388a6cf64d9158e8576e4ee89b1 2024-06-26T07:42:14.0026317Z deleted: sha256:027db5d8b76deeefbd8c34b485ab7ff14ed2a3e6800425c1ce0d77c2e9fdd4ab 2024-06-26T07:42:14.0027148Z deleted: sha256:24941234ad28cf5c8174d7cf18326c10b4e17f2c2137672ea0322df461597dd7 2024-06-26T07:42:14.0027981Z deleted: sha256:240ea2eebfa8519d6c2d88f4c99e66ce4e038d000f93ce8d03726c01d81180ef 2024-06-26T07:42:14.0028819Z deleted: sha256:09e823cb391a69f6a195e3ca8a41624b5f8b34eee14c6eb36277de814a91966e 2024-06-26T07:42:14.0029640Z deleted: sha256:730479d695169a6593bf0ac16b377491dce65175a6d158e012422367d04577fd 2024-06-26T07:42:14.0030452Z deleted: sha256:b0a30f9f7bd5459ff662453b9346a7ba8ea8db1191528345c1953eab4aa2f54e 2024-06-26T07:42:14.0031463Z deleted: sha256:75cefa87effaa3ad361c74642839923101f0f9982f814f3c9e2ed4ad52f2a008 2024-06-26T07:42:14.0032293Z deleted: sha256:6f97db87ba9512e615e871c6982000602982e1e5a5be334a56e7292251e84e00 2024-06-26T07:42:14.0033120Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-06-26T07:42:14.0033960Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-06-26T07:42:14.0034770Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-06-26T07:42:14.0035598Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-06-26T07:42:14.0036500Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-06-26T07:42:14.0037330Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-06-26T07:42:14.0038169Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-06-26T07:42:14.0039013Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-06-26T07:42:14.0039859Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-06-26T07:42:14.0040725Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-06-26T07:42:14.0041569Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-06-26T07:42:14.0042078Z 2024-06-26T07:42:14.0042231Z Total reclaimed space: 31.35GB 2024-06-26T07:42:14.0103908Z Post job cleanup. 2024-06-26T07:42:14.0164469Z Post job cleanup. 2024-06-26T07:42:14.0979215Z [command]/usr/bin/git version 2024-06-26T07:42:14.1021397Z git version 2.40.1 2024-06-26T07:42:14.1061171Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/9f0849ed-7e0e-4a86-8f9b-5aa9e95e408e' before making global git config changes 2024-06-26T07:42:14.1062428Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T07:42:14.1066806Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:42:14.1100620Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T07:42:14.1125169Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T07:42:14.1335293Z Entering 'android/libs/fbjni' 2024-06-26T07:42:14.1370043Z Entering 'third_party/FP16' 2024-06-26T07:42:14.1405317Z Entering 'third_party/FXdiv' 2024-06-26T07:42:14.1442164Z Entering 'third_party/NNPACK' 2024-06-26T07:42:14.1479265Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:42:14.1516375Z Entering 'third_party/XNNPACK' 2024-06-26T07:42:14.1565158Z Entering 'third_party/benchmark' 2024-06-26T07:42:14.1602886Z Entering 'third_party/cpp-httplib' 2024-06-26T07:42:14.1638819Z Entering 'third_party/cpuinfo' 2024-06-26T07:42:14.1675543Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:42:14.1713169Z Entering 'third_party/cutlass' 2024-06-26T07:42:14.1755342Z Entering 'third_party/eigen' 2024-06-26T07:42:14.1791018Z Entering 'third_party/fbgemm' 2024-06-26T07:42:14.1827487Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:42:14.1861810Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:42:14.1896714Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:42:14.1938045Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:42:14.1973101Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:42:14.2009391Z Entering 'third_party/flatbuffers' 2024-06-26T07:42:14.2048428Z Entering 'third_party/fmt' 2024-06-26T07:42:14.2083769Z Entering 'third_party/foxi' 2024-06-26T07:42:14.2117072Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:42:14.2152113Z Entering 'third_party/gloo' 2024-06-26T07:42:14.2185733Z Entering 'third_party/googletest' 2024-06-26T07:42:14.2223991Z Entering 'third_party/ideep' 2024-06-26T07:42:14.2260371Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:42:14.2301037Z Entering 'third_party/ittapi' 2024-06-26T07:42:14.2337383Z Entering 'third_party/kineto' 2024-06-26T07:42:14.2373440Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:42:14.2409003Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:42:14.2445753Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:42:14.2480138Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:42:14.2514523Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:42:14.2547474Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:42:14.2585272Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:42:14.2620284Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:42:14.2654729Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:42:14.2691147Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:42:14.2727354Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:42:14.2761265Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:42:14.2798397Z Entering 'third_party/mimalloc' 2024-06-26T07:42:14.2835445Z Entering 'third_party/nccl/nccl' 2024-06-26T07:42:14.2869921Z Entering 'third_party/nlohmann' 2024-06-26T07:42:14.2904621Z Entering 'third_party/onnx' 2024-06-26T07:42:14.2952489Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:42:14.2986673Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:42:14.3024122Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:42:14.3063272Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:42:14.3098639Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:42:14.3131783Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:42:14.3166357Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:42:14.3200590Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:42:14.3236203Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:42:14.3270148Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:42:14.3305207Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:42:14.3343079Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:42:14.3380958Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:42:14.3433596Z Entering 'third_party/pocketfft' 2024-06-26T07:42:14.3469655Z Entering 'third_party/protobuf' 2024-06-26T07:42:14.3507098Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:42:14.3541750Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:42:14.3580036Z Entering 'third_party/psimd' 2024-06-26T07:42:14.3616914Z Entering 'third_party/pthreadpool' 2024-06-26T07:42:14.3650714Z Entering 'third_party/pybind11' 2024-06-26T07:42:14.3683278Z Entering 'third_party/python-peachpy' 2024-06-26T07:42:14.3718633Z Entering 'third_party/sleef' 2024-06-26T07:42:14.3752908Z Entering 'third_party/tensorpipe' 2024-06-26T07:42:14.3786679Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:42:14.3821208Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:42:14.3857837Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:42:14.3891794Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:42:14.3926859Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:42:14.3977786Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T07:42:14.4000824Z http.https://github.com/.extraheader 2024-06-26T07:42:14.4009847Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-06-26T07:42:14.4038697Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T07:42:14.4246560Z Entering 'android/libs/fbjni' 2024-06-26T07:42:14.4268712Z http.https://github.com/.extraheader 2024-06-26T07:42:14.4293620Z Entering 'third_party/FP16' 2024-06-26T07:42:14.4314766Z http.https://github.com/.extraheader 2024-06-26T07:42:14.4337999Z Entering 'third_party/FXdiv' 2024-06-26T07:42:14.4360266Z http.https://github.com/.extraheader 2024-06-26T07:42:14.4383067Z Entering 'third_party/NNPACK' 2024-06-26T07:42:14.4405006Z http.https://github.com/.extraheader 2024-06-26T07:42:14.4433449Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:42:14.4455466Z http.https://github.com/.extraheader 2024-06-26T07:42:14.4479431Z Entering 'third_party/XNNPACK' 2024-06-26T07:42:14.4501313Z http.https://github.com/.extraheader 2024-06-26T07:42:14.4536066Z Entering 'third_party/benchmark' 2024-06-26T07:42:14.4558314Z http.https://github.com/.extraheader 2024-06-26T07:42:14.4582920Z Entering 'third_party/cpp-httplib' 2024-06-26T07:42:14.4604889Z http.https://github.com/.extraheader 2024-06-26T07:42:14.4627851Z Entering 'third_party/cpuinfo' 2024-06-26T07:42:14.4650486Z http.https://github.com/.extraheader 2024-06-26T07:42:14.4676713Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:42:14.4697677Z http.https://github.com/.extraheader 2024-06-26T07:42:14.4720264Z Entering 'third_party/cutlass' 2024-06-26T07:42:14.4742450Z http.https://github.com/.extraheader 2024-06-26T07:42:14.4773826Z Entering 'third_party/eigen' 2024-06-26T07:42:14.4796301Z http.https://github.com/.extraheader 2024-06-26T07:42:14.4821453Z Entering 'third_party/fbgemm' 2024-06-26T07:42:14.4844345Z http.https://github.com/.extraheader 2024-06-26T07:42:14.4869009Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:42:14.4891819Z http.https://github.com/.extraheader 2024-06-26T07:42:14.4915986Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:42:14.4938874Z http.https://github.com/.extraheader 2024-06-26T07:42:14.4961923Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:42:14.4982340Z http.https://github.com/.extraheader 2024-06-26T07:42:14.5009932Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:42:14.5030025Z http.https://github.com/.extraheader 2024-06-26T07:42:14.5052838Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:42:14.5074177Z http.https://github.com/.extraheader 2024-06-26T07:42:14.5097720Z Entering 'third_party/flatbuffers' 2024-06-26T07:42:14.5120461Z http.https://github.com/.extraheader 2024-06-26T07:42:14.5145668Z Entering 'third_party/fmt' 2024-06-26T07:42:14.5167725Z http.https://github.com/.extraheader 2024-06-26T07:42:14.5191711Z Entering 'third_party/foxi' 2024-06-26T07:42:14.5213549Z http.https://github.com/.extraheader 2024-06-26T07:42:14.5237009Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:42:14.5257771Z http.https://github.com/.extraheader 2024-06-26T07:42:14.5279085Z Entering 'third_party/gloo' 2024-06-26T07:42:14.5301200Z http.https://github.com/.extraheader 2024-06-26T07:42:14.5323255Z Entering 'third_party/googletest' 2024-06-26T07:42:14.5345830Z http.https://github.com/.extraheader 2024-06-26T07:42:14.5370017Z Entering 'third_party/ideep' 2024-06-26T07:42:14.5391953Z http.https://github.com/.extraheader 2024-06-26T07:42:14.5414298Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:42:14.5436236Z http.https://github.com/.extraheader 2024-06-26T07:42:14.5465519Z Entering 'third_party/ittapi' 2024-06-26T07:42:14.5488889Z http.https://github.com/.extraheader 2024-06-26T07:42:14.5512751Z Entering 'third_party/kineto' 2024-06-26T07:42:14.5534823Z http.https://github.com/.extraheader 2024-06-26T07:42:14.5558630Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:42:14.5579922Z http.https://github.com/.extraheader 2024-06-26T07:42:14.5604078Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:42:14.5626268Z http.https://github.com/.extraheader 2024-06-26T07:42:14.5649508Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:42:14.5669362Z http.https://github.com/.extraheader 2024-06-26T07:42:14.5692440Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:42:14.5716081Z http.https://github.com/.extraheader 2024-06-26T07:42:14.5739773Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:42:14.5762044Z http.https://github.com/.extraheader 2024-06-26T07:42:14.5784861Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:42:14.5806850Z http.https://github.com/.extraheader 2024-06-26T07:42:14.5830257Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:42:14.5852161Z http.https://github.com/.extraheader 2024-06-26T07:42:14.5876331Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:42:14.5897772Z http.https://github.com/.extraheader 2024-06-26T07:42:14.5920929Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:42:14.5941376Z http.https://github.com/.extraheader 2024-06-26T07:42:14.5966247Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:42:14.5986972Z http.https://github.com/.extraheader 2024-06-26T07:42:14.6010637Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:42:14.6031626Z http.https://github.com/.extraheader 2024-06-26T07:42:14.6054037Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:42:14.6075230Z http.https://github.com/.extraheader 2024-06-26T07:42:14.6100260Z Entering 'third_party/mimalloc' 2024-06-26T07:42:14.6122828Z http.https://github.com/.extraheader 2024-06-26T07:42:14.6146274Z Entering 'third_party/nccl/nccl' 2024-06-26T07:42:14.6168729Z http.https://github.com/.extraheader 2024-06-26T07:42:14.6191859Z Entering 'third_party/nlohmann' 2024-06-26T07:42:14.6212827Z http.https://github.com/.extraheader 2024-06-26T07:42:14.6238337Z Entering 'third_party/onnx' 2024-06-26T07:42:14.6260721Z http.https://github.com/.extraheader 2024-06-26T07:42:14.6296727Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:42:14.6318712Z http.https://github.com/.extraheader 2024-06-26T07:42:14.6342843Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:42:14.6364056Z http.https://github.com/.extraheader 2024-06-26T07:42:14.6386792Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:42:14.6410234Z http.https://github.com/.extraheader 2024-06-26T07:42:14.6437447Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:42:14.6458555Z http.https://github.com/.extraheader 2024-06-26T07:42:14.6481193Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:42:14.6502008Z http.https://github.com/.extraheader 2024-06-26T07:42:14.6524685Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:42:14.6544543Z http.https://github.com/.extraheader 2024-06-26T07:42:14.6566677Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:42:14.6587924Z http.https://github.com/.extraheader 2024-06-26T07:42:14.6612279Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:42:14.6633475Z http.https://github.com/.extraheader 2024-06-26T07:42:14.6656551Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:42:14.6677964Z http.https://github.com/.extraheader 2024-06-26T07:42:14.6702776Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:42:14.6724569Z http.https://github.com/.extraheader 2024-06-26T07:42:14.6746912Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:42:14.6768722Z http.https://github.com/.extraheader 2024-06-26T07:42:14.6794506Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:42:14.6816620Z http.https://github.com/.extraheader 2024-06-26T07:42:14.6840403Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:42:14.6861484Z http.https://github.com/.extraheader 2024-06-26T07:42:14.6900202Z Entering 'third_party/pocketfft' 2024-06-26T07:42:14.6922472Z http.https://github.com/.extraheader 2024-06-26T07:42:14.6945735Z Entering 'third_party/protobuf' 2024-06-26T07:42:14.6968413Z http.https://github.com/.extraheader 2024-06-26T07:42:14.6994810Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:42:14.7017342Z http.https://github.com/.extraheader 2024-06-26T07:42:14.7041181Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:42:14.7062559Z http.https://github.com/.extraheader 2024-06-26T07:42:14.7087154Z Entering 'third_party/psimd' 2024-06-26T07:42:14.7108251Z http.https://github.com/.extraheader 2024-06-26T07:42:14.7131236Z Entering 'third_party/pthreadpool' 2024-06-26T07:42:14.7152169Z http.https://github.com/.extraheader 2024-06-26T07:42:14.7174605Z Entering 'third_party/pybind11' 2024-06-26T07:42:14.7197609Z http.https://github.com/.extraheader 2024-06-26T07:42:14.7221934Z Entering 'third_party/python-peachpy' 2024-06-26T07:42:14.7244761Z http.https://github.com/.extraheader 2024-06-26T07:42:14.7268046Z Entering 'third_party/sleef' 2024-06-26T07:42:14.7290057Z http.https://github.com/.extraheader 2024-06-26T07:42:14.7314286Z Entering 'third_party/tensorpipe' 2024-06-26T07:42:14.7336385Z http.https://github.com/.extraheader 2024-06-26T07:42:14.7358454Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:42:14.7379694Z http.https://github.com/.extraheader 2024-06-26T07:42:14.7403472Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:42:14.7423436Z http.https://github.com/.extraheader 2024-06-26T07:42:14.7447249Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:42:14.7468437Z http.https://github.com/.extraheader 2024-06-26T07:42:14.7491223Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:42:14.7512447Z http.https://github.com/.extraheader 2024-06-26T07:42:14.7535390Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:42:14.7556230Z http.https://github.com/.extraheader 2024-06-26T07:42:14.7655876Z A job completed hook has been configured by the self-hosted runner administrator 2024-06-26T07:42:14.7673539Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T07:42:14.7680342Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:42:14.7680862Z ##[endgroup] 2024-06-26T07:42:15.5219406Z Cleaning up orphan processes