2024-06-26T07:01:46.6047361Z Current runner version: '2.317.0' 2024-06-26T07:01:46.6053340Z Runner name: 'i-0f77a10328249b1bd' 2024-06-26T07:01:46.6054141Z Runner group name: 'Default' 2024-06-26T07:01:46.6055075Z Machine name: 'ip-10-0-62-17' 2024-06-26T07:01:46.6060088Z ##[group]GITHUB_TOKEN Permissions 2024-06-26T07:01:46.6062026Z Actions: read 2024-06-26T07:01:46.6062552Z Attestations: read 2024-06-26T07:01:46.6063079Z Checks: read 2024-06-26T07:01:46.6063524Z Contents: read 2024-06-26T07:01:46.6064000Z Deployments: read 2024-06-26T07:01:46.6064481Z Discussions: read 2024-06-26T07:01:46.6064985Z Issues: read 2024-06-26T07:01:46.6065412Z Metadata: read 2024-06-26T07:01:46.6065907Z Packages: read 2024-06-26T07:01:46.6066394Z Pages: read 2024-06-26T07:01:46.6066831Z PullRequests: read 2024-06-26T07:01:46.6067355Z RepositoryProjects: read 2024-06-26T07:01:46.6067943Z SecurityEvents: read 2024-06-26T07:01:46.6068444Z Statuses: read 2024-06-26T07:01:46.6068931Z ##[endgroup] 2024-06-26T07:01:46.6072102Z Secret source: Actions 2024-06-26T07:01:46.6072816Z Prepare workflow directory 2024-06-26T07:01:46.9754163Z Prepare all required actions 2024-06-26T07:01:46.9933334Z Getting action download info 2024-06-26T07:01:47.1847828Z Download action repository 'pytorch/test-infra@main' (SHA:43a2ce341cc31288e9a38b65ce600a7f43021bd5) 2024-06-26T07:01:47.4959253Z Download action repository 'pytorch/pytorch@main' (SHA:c422a9549d2b320b47a9391f6cbcf592b2876c88) 2024-06-26T07:01:50.2476786Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-06-26T07:01:50.3897996Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-06-26T07:01:50.6390961Z Getting action download info 2024-06-26T07:01:50.7188064Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-06-26T07:01:50.9030984Z Getting action download info 2024-06-26T07:01:51.0033769Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-06-26T07:01:51.1557534Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/129470 (b8c4c54d347aa776934c60784e35936878ef18dc) 2024-06-26T07:01:51.1559690Z ##[group] Inputs 2024-06-26T07:01:51.1560168Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:01:51.1568680Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:01:51.1578312Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:01:51.1579500Z sync-tag: 2024-06-26T07:01:51.1580372Z timeout-minutes: 240 2024-06-26T07:01:51.1580728Z use-gha: 2024-06-26T07:01:51.1581016Z dashboard-tag: 2024-06-26T07:01:51.1581351Z s3-bucket: gha-artifacts 2024-06-26T07:01:51.1581725Z aws-role-to-assume: 2024-06-26T07:01:51.1582117Z ##[endgroup] 2024-06-26T07:01:51.1582977Z Complete job name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:01:51.2141121Z A job started hook has been configured by the self-hosted runner administrator 2024-06-26T07:01:51.2277636Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T07:01:51.2287962Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:01:51.2288497Z ##[endgroup] 2024-06-26T07:01:51.9868822Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-06-26T07:01:51.9869373Z with: 2024-06-26T07:01:51.9870061Z github-secret: *** 2024-06-26T07:01:51.9871022Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-06-26T07:01:51.9872133Z activate-with-label: false 2024-06-26T07:01:51.9872514Z label: with-ssh 2024-06-26T07:01:51.9872844Z remove-existing-keys: true 2024-06-26T07:01:51.9873227Z fail-silently: true 2024-06-26T07:01:51.9873544Z env: 2024-06-26T07:01:51.9873845Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:01:51.9874202Z ##[endgroup] 2024-06-26T07:01:52.0720775Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-06-26T07:01:52.0721810Z ciflow reference detected, attempting to extract PR number 2024-06-26T07:01:52.3537993Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-06-26T07:01:52.4152944Z No SSH keys found for user pytorch-bot[bot] 2024-06-26T07:01:52.4154530Z Grabbing public ssh keys from https://github.com/leslie-fang-intel.keys 2024-06-26T07:01:52.4789438Z No SSH keys found for user leslie-fang-intel 2024-06-26T07:01:52.4890385Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-06-26T07:01:52.4890939Z with: 2024-06-26T07:01:52.4891229Z submodules: recursive 2024-06-26T07:01:52.4891562Z fetch-depth: 0 2024-06-26T07:01:52.4891860Z env: 2024-06-26T07:01:52.4892136Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:01:52.4892476Z ##[endgroup] 2024-06-26T07:01:52.5080186Z ##[group]Run retry () { 2024-06-26T07:01:52.5080622Z retry () { 2024-06-26T07:01:52.5081237Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-06-26T07:01:52.5081841Z } 2024-06-26T07:01:52.5082227Z echo "${GITHUB_WORKSPACE}" 2024-06-26T07:01:52.5082709Z if [ -z "${NO_SUDO}" ]; then 2024-06-26T07:01:52.5083233Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:01:52.5083737Z else 2024-06-26T07:01:52.5084151Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:01:52.5084595Z fi 2024-06-26T07:01:52.5085084Z mkdir "${GITHUB_WORKSPACE}" 2024-06-26T07:01:52.5093374Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:01:52.5093951Z env: 2024-06-26T07:01:52.5094326Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:01:52.5094719Z NO_SUDO: 2024-06-26T07:01:52.5095060Z ##[endgroup] 2024-06-26T07:01:52.5117691Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:01:54.8343420Z ##[group]Run malfet/checkout@silent-checkout 2024-06-26T07:01:54.8343866Z with: 2024-06-26T07:01:54.8344192Z ref: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:01:54.8344622Z fetch-depth: 0 2024-06-26T07:01:54.8344938Z submodules: recursive 2024-06-26T07:01:54.8345278Z quiet-checkout: true 2024-06-26T07:01:54.8345625Z repository: pytorch/pytorch 2024-06-26T07:01:54.8346102Z token: *** 2024-06-26T07:01:54.8346392Z ssh-strict: true 2024-06-26T07:01:54.8346719Z persist-credentials: true 2024-06-26T07:01:54.8347079Z clean: true 2024-06-26T07:01:54.8347404Z sparse-checkout-cone-mode: true 2024-06-26T07:01:54.8347791Z lfs: false 2024-06-26T07:01:54.8348093Z set-safe-directory: true 2024-06-26T07:01:54.8348430Z env: 2024-06-26T07:01:54.8348696Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:01:54.8349040Z ##[endgroup] 2024-06-26T07:01:54.9284041Z Syncing repository: pytorch/pytorch 2024-06-26T07:01:54.9285631Z ##[group]Getting Git version info 2024-06-26T07:01:54.9286395Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:01:54.9287243Z [command]/usr/bin/git version 2024-06-26T07:01:54.9287631Z git version 2.40.1 2024-06-26T07:01:54.9297667Z ##[endgroup] 2024-06-26T07:01:54.9311808Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/a8349d53-69fc-46b1-aa76-d94417914c71/.gitconfig' 2024-06-26T07:01:54.9317512Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/a8349d53-69fc-46b1-aa76-d94417914c71' before making global git config changes 2024-06-26T07:01:54.9318767Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T07:01:54.9321920Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:01:54.9353305Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:01:54.9357064Z ##[group]Initializing the repository 2024-06-26T07:01:54.9359766Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:01:54.9379562Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-06-26T07:01:54.9380449Z hint: is subject to change. To configure the initial branch name to use in all 2024-06-26T07:01:54.9381394Z hint: of your new repositories, which will suppress this warning, call: 2024-06-26T07:01:54.9382248Z hint: 2024-06-26T07:01:54.9382699Z hint: git config --global init.defaultBranch 2024-06-26T07:01:54.9383178Z hint: 2024-06-26T07:01:54.9383869Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-06-26T07:01:54.9384874Z hint: 'development'. The just-created branch can be renamed via this command: 2024-06-26T07:01:54.9385462Z hint: 2024-06-26T07:01:54.9385788Z hint: git branch -m 2024-06-26T07:01:54.9386531Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-06-26T07:01:54.9388220Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-06-26T07:01:54.9409816Z ##[endgroup] 2024-06-26T07:01:54.9410416Z ##[group]Disabling automatic garbage collection 2024-06-26T07:01:54.9412596Z [command]/usr/bin/git config --local gc.auto 0 2024-06-26T07:01:54.9431915Z ##[endgroup] 2024-06-26T07:01:54.9432440Z ##[group]Setting up auth 2024-06-26T07:01:54.9437083Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T07:01:54.9459813Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T07:01:54.9662752Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T07:01:54.9685520Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T07:01:54.9884475Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:01:54.9925470Z ##[endgroup] 2024-06-26T07:01:54.9926032Z ##[group]Fetching the repository 2024-06-26T07:01:54.9931332Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-06-26T07:01:57.3236381Z remote: Enumerating objects: 987619 2024-06-26T07:01:57.3274190Z remote: Enumerating objects: 989180 2024-06-26T07:01:57.3275016Z remote: Enumerating objects: 990209, done. 2024-06-26T07:01:57.3276580Z remote: Counting objects: 0% (1/2590) 2024-06-26T07:01:57.3277236Z remote: Counting objects: 1% (26/2590) 2024-06-26T07:01:57.3277936Z remote: Counting objects: 2% (52/2590) 2024-06-26T07:01:57.3279152Z remote: Counting objects: 3% (78/2590) 2024-06-26T07:01:57.3280264Z remote: Counting objects: 4% (104/2590) 2024-06-26T07:01:57.3281022Z remote: Counting objects: 5% (130/2590) 2024-06-26T07:01:57.3281626Z remote: Counting objects: 6% (156/2590) 2024-06-26T07:01:57.3282442Z remote: Counting objects: 7% (182/2590) 2024-06-26T07:01:57.3283220Z remote: Counting objects: 8% (208/2590) 2024-06-26T07:01:57.3284443Z remote: Counting objects: 9% (234/2590) 2024-06-26T07:01:57.3285165Z remote: Counting objects: 10% (259/2590) 2024-06-26T07:01:57.3285884Z remote: Counting objects: 11% (285/2590) 2024-06-26T07:01:57.3286621Z remote: Counting objects: 12% (311/2590) 2024-06-26T07:01:57.3287278Z remote: Counting objects: 13% (337/2590) 2024-06-26T07:01:57.3287994Z remote: Counting objects: 14% (363/2590) 2024-06-26T07:01:57.3288659Z remote: Counting objects: 15% (389/2590) 2024-06-26T07:01:57.3289480Z remote: Counting objects: 16% (415/2590) 2024-06-26T07:01:57.3290221Z remote: Counting objects: 17% (441/2590) 2024-06-26T07:01:57.3290961Z remote: Counting objects: 18% (467/2590) 2024-06-26T07:01:57.3291669Z remote: Counting objects: 19% (493/2590) 2024-06-26T07:01:57.3292371Z remote: Counting objects: 20% (518/2590) 2024-06-26T07:01:57.3293074Z remote: Counting objects: 21% (544/2590) 2024-06-26T07:01:57.3293591Z remote: Counting objects: 22% (570/2590) 2024-06-26T07:01:57.3294199Z remote: Counting objects: 23% (596/2590) 2024-06-26T07:01:57.3294915Z remote: Counting objects: 24% (622/2590) 2024-06-26T07:01:57.3295941Z remote: Counting objects: 25% (648/2590) 2024-06-26T07:01:57.3296684Z remote: Counting objects: 26% (674/2590) 2024-06-26T07:01:57.3297393Z remote: Counting objects: 27% (700/2590) 2024-06-26T07:01:57.3298394Z remote: Counting objects: 28% (726/2590) 2024-06-26T07:01:57.3299088Z remote: Counting objects: 29% (752/2590) 2024-06-26T07:01:57.3299638Z remote: Counting objects: 30% (777/2590) 2024-06-26T07:01:57.3300319Z remote: Counting objects: 31% (803/2590) 2024-06-26T07:01:57.3300947Z remote: Counting objects: 32% (829/2590) 2024-06-26T07:01:57.3301661Z remote: Counting objects: 33% (855/2590) 2024-06-26T07:01:57.3302332Z remote: Counting objects: 34% (881/2590) 2024-06-26T07:01:57.3302959Z remote: Counting objects: 35% (907/2590) 2024-06-26T07:01:57.3303591Z remote: Counting objects: 36% (933/2590) 2024-06-26T07:01:57.3304303Z remote: Counting objects: 37% (959/2590) 2024-06-26T07:01:57.3304911Z remote: Counting objects: 38% (985/2590) 2024-06-26T07:01:57.3305636Z remote: Counting objects: 39% (1011/2590) 2024-06-26T07:01:57.3306352Z remote: Counting objects: 40% (1036/2590) 2024-06-26T07:01:57.3307091Z remote: Counting objects: 41% (1062/2590) 2024-06-26T07:01:57.3307856Z remote: Counting objects: 42% (1088/2590) 2024-06-26T07:01:57.3308723Z remote: Counting objects: 43% (1114/2590) 2024-06-26T07:01:57.3309479Z remote: Counting objects: 44% (1140/2590) 2024-06-26T07:01:57.3310106Z remote: Counting objects: 45% (1166/2590) 2024-06-26T07:01:57.3310802Z remote: Counting objects: 46% (1192/2590) 2024-06-26T07:01:57.3311465Z remote: Counting objects: 47% (1218/2590) 2024-06-26T07:01:57.3311983Z remote: Counting objects: 48% (1244/2590) 2024-06-26T07:01:57.3312689Z remote: Counting objects: 49% (1270/2590) 2024-06-26T07:01:57.3313400Z remote: Counting objects: 50% (1295/2590) 2024-06-26T07:01:57.3314083Z remote: Counting objects: 51% (1321/2590) 2024-06-26T07:01:57.3314779Z remote: Counting objects: 52% (1347/2590) 2024-06-26T07:01:57.3315486Z remote: Counting objects: 53% (1373/2590) 2024-06-26T07:01:57.3316175Z remote: Counting objects: 54% (1399/2590) 2024-06-26T07:01:57.3316739Z remote: Counting objects: 55% (1425/2590) 2024-06-26T07:01:57.3317451Z remote: Counting objects: 56% (1451/2590) 2024-06-26T07:01:57.3318084Z remote: Counting objects: 57% (1477/2590) 2024-06-26T07:01:57.3318784Z remote: Counting objects: 58% (1503/2590) 2024-06-26T07:01:57.3319495Z remote: Counting objects: 59% (1529/2590) 2024-06-26T07:01:57.3320185Z remote: Counting objects: 60% (1554/2590) 2024-06-26T07:01:57.3320883Z remote: Counting objects: 61% (1580/2590) 2024-06-26T07:01:57.3321582Z remote: Counting objects: 62% (1606/2590) 2024-06-26T07:01:57.3322159Z remote: Counting objects: 63% (1632/2590) 2024-06-26T07:01:57.3322894Z remote: Counting objects: 64% (1658/2590) 2024-06-26T07:01:57.3323424Z remote: Counting objects: 65% (1684/2590) 2024-06-26T07:01:57.3324117Z remote: Counting objects: 66% (1710/2590) 2024-06-26T07:01:57.3324830Z remote: Counting objects: 67% (1736/2590) 2024-06-26T07:01:57.3325540Z remote: Counting objects: 68% (1762/2590) 2024-06-26T07:01:57.3326217Z remote: Counting objects: 69% (1788/2590) 2024-06-26T07:01:57.3326803Z remote: Counting objects: 70% (1813/2590) 2024-06-26T07:01:57.3327318Z remote: Counting objects: 71% (1839/2590) 2024-06-26T07:01:57.3327871Z remote: Counting objects: 72% (1865/2590) 2024-06-26T07:01:57.3328544Z remote: Counting objects: 73% (1891/2590) 2024-06-26T07:01:57.3329355Z remote: Counting objects: 74% (1917/2590) 2024-06-26T07:01:57.3329898Z remote: Counting objects: 75% (1943/2590) 2024-06-26T07:01:57.3330531Z remote: Counting objects: 76% (1969/2590) 2024-06-26T07:01:57.3331043Z remote: Counting objects: 77% (1995/2590) 2024-06-26T07:01:57.3331550Z remote: Counting objects: 78% (2021/2590) 2024-06-26T07:01:57.3332077Z remote: Counting objects: 79% (2047/2590) 2024-06-26T07:01:57.3332585Z remote: Counting objects: 80% (2072/2590) 2024-06-26T07:01:57.3333090Z remote: Counting objects: 81% (2098/2590) 2024-06-26T07:01:57.3333607Z remote: Counting objects: 82% (2124/2590) 2024-06-26T07:01:57.3334114Z remote: Counting objects: 83% (2150/2590) 2024-06-26T07:01:57.3334622Z remote: Counting objects: 84% (2176/2590) 2024-06-26T07:01:57.3335124Z remote: Counting objects: 85% (2202/2590) 2024-06-26T07:01:57.3335635Z remote: Counting objects: 86% (2228/2590) 2024-06-26T07:01:57.3336142Z remote: Counting objects: 87% (2254/2590) 2024-06-26T07:01:57.3336655Z remote: Counting objects: 88% (2280/2590) 2024-06-26T07:01:57.3337162Z remote: Counting objects: 89% (2306/2590) 2024-06-26T07:01:57.3337831Z remote: Counting objects: 90% (2331/2590) 2024-06-26T07:01:57.3338343Z remote: Counting objects: 91% (2357/2590) 2024-06-26T07:01:57.3338853Z remote: Counting objects: 92% (2383/2590) 2024-06-26T07:01:57.3339360Z remote: Counting objects: 93% (2409/2590) 2024-06-26T07:01:57.3339958Z remote: Counting objects: 94% (2435/2590) 2024-06-26T07:01:57.3340468Z remote: Counting objects: 95% (2461/2590) 2024-06-26T07:01:57.3340982Z remote: Counting objects: 96% (2487/2590) 2024-06-26T07:01:57.3341485Z remote: Counting objects: 97% (2513/2590) 2024-06-26T07:01:57.3342000Z remote: Counting objects: 98% (2539/2590) 2024-06-26T07:01:57.3342513Z remote: Counting objects: 99% (2565/2590) 2024-06-26T07:01:57.3343028Z remote: Counting objects: 100% (2590/2590) 2024-06-26T07:01:57.3343569Z remote: Counting objects: 100% (2590/2590), done. 2024-06-26T07:01:57.3468070Z remote: Compressing objects: 0% (1/1068) 2024-06-26T07:01:57.3917633Z remote: Compressing objects: 1% (11/1068) 2024-06-26T07:01:57.4388690Z remote: Compressing objects: 2% (22/1068) 2024-06-26T07:01:57.4765453Z remote: Compressing objects: 3% (33/1068) 2024-06-26T07:01:57.4998190Z remote: Compressing objects: 4% (43/1068) 2024-06-26T07:01:57.5294292Z remote: Compressing objects: 5% (54/1068) 2024-06-26T07:01:57.5512564Z remote: Compressing objects: 6% (65/1068) 2024-06-26T07:01:57.5927305Z remote: Compressing objects: 7% (75/1068) 2024-06-26T07:01:57.6814445Z remote: Compressing objects: 8% (86/1068) 2024-06-26T07:01:57.7613117Z remote: Compressing objects: 9% (97/1068) 2024-06-26T07:01:57.7864496Z remote: Compressing objects: 10% (107/1068) 2024-06-26T07:01:57.8052646Z remote: Compressing objects: 11% (118/1068) 2024-06-26T07:01:57.8159778Z remote: Compressing objects: 12% (129/1068) 2024-06-26T07:01:57.8160914Z remote: Compressing objects: 13% (139/1068) 2024-06-26T07:01:57.8285930Z remote: Compressing objects: 14% (150/1068) 2024-06-26T07:01:57.8286639Z remote: Compressing objects: 15% (161/1068) 2024-06-26T07:01:57.8351220Z remote: Compressing objects: 16% (171/1068) 2024-06-26T07:01:57.8351977Z remote: Compressing objects: 17% (182/1068) 2024-06-26T07:01:57.8352527Z remote: Compressing objects: 18% (193/1068) 2024-06-26T07:01:57.8366120Z remote: Compressing objects: 19% (203/1068) 2024-06-26T07:01:57.8387266Z remote: Compressing objects: 20% (214/1068) 2024-06-26T07:01:57.8402386Z remote: Compressing objects: 21% (225/1068) 2024-06-26T07:01:57.8419616Z remote: Compressing objects: 22% (235/1068) 2024-06-26T07:01:57.8443161Z remote: Compressing objects: 23% (246/1068) 2024-06-26T07:01:57.8457073Z remote: Compressing objects: 24% (257/1068) 2024-06-26T07:01:57.8462443Z remote: Compressing objects: 25% (267/1068) 2024-06-26T07:01:57.8471474Z remote: Compressing objects: 26% (278/1068) 2024-06-26T07:01:57.8476178Z remote: Compressing objects: 27% (289/1068) 2024-06-26T07:01:57.8480221Z remote: Compressing objects: 28% (300/1068) 2024-06-26T07:01:57.8489071Z remote: Compressing objects: 29% (310/1068) 2024-06-26T07:01:57.8496333Z remote: Compressing objects: 30% (321/1068) 2024-06-26T07:01:57.8501753Z remote: Compressing objects: 31% (332/1068) 2024-06-26T07:01:57.8508408Z remote: Compressing objects: 32% (342/1068) 2024-06-26T07:01:57.8518221Z remote: Compressing objects: 33% (353/1068) 2024-06-26T07:01:57.8531496Z remote: Compressing objects: 34% (364/1068) 2024-06-26T07:01:57.8547837Z remote: Compressing objects: 35% (374/1068) 2024-06-26T07:01:57.8565787Z remote: Compressing objects: 36% (385/1068) 2024-06-26T07:01:57.8581173Z remote: Compressing objects: 37% (396/1068) 2024-06-26T07:01:57.8596273Z remote: Compressing objects: 38% (406/1068) 2024-06-26T07:01:57.8610883Z remote: Compressing objects: 39% (417/1068) 2024-06-26T07:01:57.8626828Z remote: Compressing objects: 40% (428/1068) 2024-06-26T07:01:57.8646227Z remote: Compressing objects: 41% (438/1068) 2024-06-26T07:01:57.8661004Z remote: Compressing objects: 42% (449/1068) 2024-06-26T07:01:57.8674386Z remote: Compressing objects: 43% (460/1068) 2024-06-26T07:01:57.8679025Z remote: Compressing objects: 44% (470/1068) 2024-06-26T07:01:57.8684046Z remote: Compressing objects: 45% (481/1068) 2024-06-26T07:01:57.8692299Z remote: Compressing objects: 46% (492/1068) 2024-06-26T07:01:57.8709322Z remote: Compressing objects: 47% (502/1068) 2024-06-26T07:01:57.8721901Z remote: Compressing objects: 48% (513/1068) 2024-06-26T07:01:57.8725751Z remote: Compressing objects: 49% (524/1068) 2024-06-26T07:01:57.8733215Z remote: Compressing objects: 50% (534/1068) 2024-06-26T07:01:57.8743324Z remote: Compressing objects: 51% (545/1068) 2024-06-26T07:01:57.8755680Z remote: Compressing objects: 52% (556/1068) 2024-06-26T07:01:57.8772162Z remote: Compressing objects: 53% (567/1068) 2024-06-26T07:01:57.8788327Z remote: Compressing objects: 54% (577/1068) 2024-06-26T07:01:57.8797981Z remote: Compressing objects: 55% (588/1068) 2024-06-26T07:01:57.8808432Z remote: Compressing objects: 56% (599/1068) 2024-06-26T07:01:57.8824275Z remote: Compressing objects: 57% (609/1068) 2024-06-26T07:01:57.8835395Z remote: Compressing objects: 58% (620/1068) 2024-06-26T07:01:57.8846141Z remote: Compressing objects: 59% (631/1068) 2024-06-26T07:01:57.8855526Z remote: Compressing objects: 60% (641/1068) 2024-06-26T07:01:57.8868605Z remote: Compressing objects: 61% (652/1068) 2024-06-26T07:01:57.8876080Z remote: Compressing objects: 62% (663/1068) 2024-06-26T07:01:57.8885442Z remote: Compressing objects: 63% (673/1068) 2024-06-26T07:01:57.8893611Z remote: Compressing objects: 64% (684/1068) 2024-06-26T07:01:57.8901234Z remote: Compressing objects: 65% (695/1068) 2024-06-26T07:01:57.8903252Z remote: Compressing objects: 66% (705/1068) 2024-06-26T07:01:57.8914367Z remote: Compressing objects: 67% (716/1068) 2024-06-26T07:01:57.8917339Z remote: Compressing objects: 68% (727/1068) 2024-06-26T07:01:57.8921577Z remote: Compressing objects: 69% (737/1068) 2024-06-26T07:01:57.8925444Z remote: Compressing objects: 70% (748/1068) 2024-06-26T07:01:57.8929602Z remote: Compressing objects: 71% (759/1068) 2024-06-26T07:01:57.8930572Z remote: Compressing objects: 72% (769/1068) 2024-06-26T07:01:57.8931230Z remote: Compressing objects: 73% (780/1068) 2024-06-26T07:01:57.8932429Z remote: Compressing objects: 74% (791/1068) 2024-06-26T07:01:57.8933434Z remote: Compressing objects: 75% (801/1068) 2024-06-26T07:01:57.8933988Z remote: Compressing objects: 76% (812/1068) 2024-06-26T07:01:57.8934514Z remote: Compressing objects: 77% (823/1068) 2024-06-26T07:01:57.8935054Z remote: Compressing objects: 78% (834/1068) 2024-06-26T07:01:57.8938735Z remote: Compressing objects: 79% (844/1068) 2024-06-26T07:01:57.8939293Z remote: Compressing objects: 80% (855/1068) 2024-06-26T07:01:57.8941361Z remote: Compressing objects: 81% (866/1068) 2024-06-26T07:01:57.8943917Z remote: Compressing objects: 82% (876/1068) 2024-06-26T07:01:57.8944876Z remote: Compressing objects: 83% (887/1068) 2024-06-26T07:01:57.8946854Z remote: Compressing objects: 84% (898/1068) 2024-06-26T07:01:57.8947621Z remote: Compressing objects: 85% (908/1068) 2024-06-26T07:01:57.8949969Z remote: Compressing objects: 86% (919/1068) 2024-06-26T07:01:57.8952006Z remote: Compressing objects: 87% (930/1068) 2024-06-26T07:01:57.8952883Z remote: Compressing objects: 88% (940/1068) 2024-06-26T07:01:57.8955102Z remote: Compressing objects: 89% (951/1068) 2024-06-26T07:01:57.8956022Z remote: Compressing objects: 90% (962/1068) 2024-06-26T07:01:57.8956681Z remote: Compressing objects: 91% (972/1068) 2024-06-26T07:01:57.8957993Z remote: Compressing objects: 92% (983/1068) 2024-06-26T07:01:57.8958994Z remote: Compressing objects: 93% (994/1068) 2024-06-26T07:01:57.8960419Z remote: Compressing objects: 94% (1004/1068) 2024-06-26T07:01:57.8961986Z remote: Compressing objects: 95% (1015/1068) 2024-06-26T07:01:57.8962791Z remote: Compressing objects: 96% (1026/1068) 2024-06-26T07:01:57.8964056Z remote: Compressing objects: 97% (1036/1068) 2024-06-26T07:01:57.8964783Z remote: Compressing objects: 98% (1047/1068) 2024-06-26T07:01:57.8966465Z remote: Compressing objects: 99% (1058/1068) 2024-06-26T07:01:57.8967473Z remote: Compressing objects: 100% (1068/1068) 2024-06-26T07:01:57.8968099Z remote: Compressing objects: 100% (1068/1068), done. 2024-06-26T07:02:18.0365488Z remote: Total 990209 (delta 2055), reused 1861 (delta 1521), pack-reused 987619 2024-06-26T07:02:39.7808814Z [command]/usr/bin/git rev-parse --verify --quiet b8c4c54d347aa776934c60784e35936878ef18dc^{object} 2024-06-26T07:02:39.7827224Z b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:02:39.7830760Z ##[endgroup] 2024-06-26T07:02:39.7831489Z ##[group]Determining the checkout info 2024-06-26T07:02:39.7833345Z ##[endgroup] 2024-06-26T07:02:39.7833906Z ##[group]Checking out the ref 2024-06-26T07:02:39.7835172Z [command]/usr/bin/git checkout --quiet --force b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:02:40.9984931Z ##[endgroup] 2024-06-26T07:02:40.9985534Z ##[group]Setting up auth for fetching submodules 2024-06-26T07:02:40.9989297Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:02:41.0034148Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-06-26T07:02:41.0053948Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-06-26T07:02:41.0074811Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-06-26T07:02:41.0091465Z ##[endgroup] 2024-06-26T07:02:41.0092011Z ##[group]Fetching submodules 2024-06-26T07:02:41.0095071Z [command]/usr/bin/git submodule sync --recursive 2024-06-26T07:02:41.0296358Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-06-26T07:02:41.0497032Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-06-26T07:02:41.0498840Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-06-26T07:02:41.0500402Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-06-26T07:02:41.0502547Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-06-26T07:02:41.0505400Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-06-26T07:02:41.0507849Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-06-26T07:02:41.0510582Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-06-26T07:02:41.0514346Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-06-26T07:02:41.0516815Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-06-26T07:02:41.0520402Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-06-26T07:02:41.0523221Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-06-26T07:02:41.0526818Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-06-26T07:02:41.0530315Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-06-26T07:02:41.0533969Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-06-26T07:02:41.0537472Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-06-26T07:02:41.0541563Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-06-26T07:02:41.0545303Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:02:41.0548957Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-06-26T07:02:41.0552912Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-06-26T07:02:41.0556722Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-06-26T07:02:41.0560727Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-06-26T07:02:41.0564775Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-06-26T07:02:41.0569329Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-06-26T07:02:41.0573453Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-06-26T07:02:41.0577967Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-06-26T07:02:41.0582359Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-06-26T07:02:41.0587063Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-06-26T07:02:41.0591395Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-06-26T07:02:41.0595978Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-06-26T07:02:41.0600735Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-06-26T07:02:41.0605548Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-06-26T07:02:41.0610260Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-06-26T07:02:41.0615187Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-06-26T07:02:41.0620681Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-06-26T07:02:41.0625580Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-06-26T07:02:41.0645993Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-06-26T07:02:41.3234494Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-06-26T07:02:41.5106969Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-06-26T07:02:41.7002507Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-06-26T07:02:41.9330629Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-06-26T07:02:43.9034537Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-06-26T07:02:54.2105248Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-06-26T07:02:54.5602948Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-06-26T07:02:55.0945002Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-06-26T07:02:55.6705146Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-06-26T07:02:56.8278005Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-06-26T07:02:58.6164682Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-06-26T07:03:04.9829571Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-06-26T07:03:06.3158437Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-06-26T07:03:08.0707498Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-06-26T07:03:09.2707744Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-06-26T07:03:09.4479899Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-06-26T07:03:09.8031303Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-06-26T07:03:10.0896522Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-06-26T07:03:11.0832336Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-06-26T07:03:11.4211674Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-06-26T07:03:11.7018506Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-06-26T07:03:13.2565847Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-06-26T07:03:13.9868597Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-06-26T07:03:14.6826932Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-06-26T07:03:20.3356700Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-06-26T07:03:22.8069544Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-06-26T07:03:28.3947265Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-06-26T07:03:28.6525984Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-06-26T07:03:37.4520157Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-06-26T07:03:37.6033880Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-06-26T07:03:37.7951259Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-06-26T07:03:39.0116413Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-06-26T07:03:39.2826992Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-06-26T07:03:39.8552784Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-06-26T07:03:40.2855376Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-06-26T07:03:40.2942366Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-06-26T07:03:40.3002323Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-06-26T07:03:40.3181995Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-06-26T07:03:40.3474498Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-06-26T07:03:41.1216314Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-06-26T07:03:41.1387614Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-06-26T07:03:41.1735455Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-06-26T07:03:41.2583752Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-06-26T07:03:41.2857013Z Submodule path 'third_party/cudnn_frontend': checked out 'aa3abd4bc689d6412979c7f55f9cd132848c9c6a' 2024-06-26T07:03:41.7105358Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-06-26T07:03:41.9239542Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-06-26T07:03:41.9873152Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-06-26T07:03:41.9884837Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:03:41.9886842Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:03:41.9889604Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:03:41.9892151Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:03:41.9894839Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:03:41.9910630Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-06-26T07:03:42.9722530Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-06-26T07:03:43.5475745Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-06-26T07:03:45.2810910Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-06-26T07:03:46.2248192Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-06-26T07:03:46.5649699Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-06-26T07:03:46.6486441Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-06-26T07:03:46.9911423Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-06-26T07:03:47.0451506Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-06-26T07:03:47.0539549Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-06-26T07:03:47.1472990Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-06-26T07:03:47.1761286Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-06-26T07:03:47.1822657Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-06-26T07:03:47.2125797Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-06-26T07:03:47.2320567Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-06-26T07:03:47.2683955Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:03:47.2780206Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-06-26T07:03:47.2791093Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-06-26T07:03:47.2806314Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-06-26T07:03:59.0270414Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-06-26T07:03:59.0409645Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-06-26T07:03:59.1189851Z Submodule path 'third_party/kineto': checked out '8681ff11e1fa54da39023076c5c43eddd87b7a8a' 2024-06-26T07:03:59.1200642Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:03:59.1204170Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:03:59.1205978Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:03:59.1220033Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-06-26T07:03:59.6660360Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-06-26T07:04:00.8663126Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-06-26T07:04:01.8841687Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-06-26T07:04:01.8851277Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:01.8853399Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:01.8855642Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:01.8858061Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:01.8860655Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:01.8863096Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:01.8865400Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:01.8867253Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:01.8884469Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-06-26T07:04:02.9312797Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-06-26T07:04:03.3322761Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-06-26T07:04:04.6427279Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-06-26T07:04:04.9056185Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-06-26T07:04:05.3901102Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-06-26T07:04:06.3566337Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-06-26T07:04:12.2504887Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-06-26T07:04:12.6443780Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-06-26T07:04:12.6579665Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-06-26T07:04:12.6868134Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-06-26T07:04:12.6960798Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-06-26T07:04:12.6970812Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:04:12.6986544Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-06-26T07:04:13.2204963Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-06-26T07:04:13.2340273Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-06-26T07:04:13.2670502Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-06-26T07:04:13.3468521Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-06-26T07:04:13.3584299Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-06-26T07:04:13.3876189Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-06-26T07:04:13.4374337Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-06-26T07:04:13.4665787Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-06-26T07:04:13.4857095Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-06-26T07:04:13.5705938Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-06-26T07:04:13.8432896Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-06-26T07:04:13.8460193Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-06-26T07:04:13.8488414Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-06-26T07:04:13.8489979Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-06-26T07:04:14.2670947Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-06-26T07:04:15.4498399Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-06-26T07:04:15.4777173Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-06-26T07:04:15.5301024Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-06-26T07:04:15.5313064Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:04:15.5319390Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:04:15.5321117Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:04:15.5322941Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:04:15.5325099Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:04:15.5327393Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:04:15.5329427Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:04:15.5331018Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:04:15.5344547Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-06-26T07:04:15.9323988Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-06-26T07:04:16.9765568Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-06-26T07:04:17.2609859Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-06-26T07:04:22.8442637Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-06-26T07:04:23.0976774Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-06-26T07:04:23.2867830Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-06-26T07:04:23.5502986Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-06-26T07:04:31.0076062Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-06-26T07:04:31.0398434Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-06-26T07:04:31.0517503Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-06-26T07:04:31.1358522Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-06-26T07:04:31.1447501Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-06-26T07:04:31.1551967Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-06-26T07:04:31.1660642Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-06-26T07:04:31.1670886Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:04:31.1673569Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:04:31.1688453Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-06-26T07:04:32.8981751Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-06-26T07:04:34.0533922Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-06-26T07:04:34.0898022Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:04:34.4509283Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-06-26T07:04:34.4591642Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-06-26T07:04:34.6739960Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-06-26T07:04:34.6753192Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:04:34.6755090Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-06-26T07:04:34.6776940Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-06-26T07:04:35.0666224Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-06-26T07:04:36.0549700Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-06-26T07:04:36.1167889Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-06-26T07:04:36.1227495Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-06-26T07:04:36.1310495Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-06-26T07:04:36.1588503Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-06-26T07:04:36.1808094Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-06-26T07:04:36.2134429Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-06-26T07:04:36.2335935Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-06-26T07:04:36.2347072Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:04:36.2348967Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:04:36.2350637Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:04:36.2352541Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:04:36.2370585Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-06-26T07:04:37.1955040Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-06-26T07:04:37.3990875Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-06-26T07:04:38.4014511Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-06-26T07:04:39.4552505Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-06-26T07:04:39.4668168Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-06-26T07:04:39.5231818Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-06-26T07:04:39.5462322Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-06-26T07:04:39.5473125Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:04:39.5488973Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-06-26T07:04:39.7294642Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-06-26T07:04:39.7316790Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-06-26T07:04:39.7522164Z Entering 'android/libs/fbjni' 2024-06-26T07:04:39.7550404Z Entering 'third_party/FP16' 2024-06-26T07:04:39.7579102Z Entering 'third_party/FXdiv' 2024-06-26T07:04:39.7607329Z Entering 'third_party/NNPACK' 2024-06-26T07:04:39.7634391Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:04:39.7660839Z Entering 'third_party/XNNPACK' 2024-06-26T07:04:39.7698968Z Entering 'third_party/benchmark' 2024-06-26T07:04:39.7727154Z Entering 'third_party/cpp-httplib' 2024-06-26T07:04:39.7754064Z Entering 'third_party/cpuinfo' 2024-06-26T07:04:39.7782673Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:04:39.7811797Z Entering 'third_party/cutlass' 2024-06-26T07:04:39.7844157Z Entering 'third_party/eigen' 2024-06-26T07:04:39.7873866Z Entering 'third_party/fbgemm' 2024-06-26T07:04:39.7900390Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:04:39.7926073Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:04:39.7953739Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:04:39.7983908Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:04:39.8010903Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:04:39.8039712Z Entering 'third_party/flatbuffers' 2024-06-26T07:04:39.8067676Z Entering 'third_party/fmt' 2024-06-26T07:04:39.8095158Z Entering 'third_party/foxi' 2024-06-26T07:04:39.8123635Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:04:39.8151297Z Entering 'third_party/gloo' 2024-06-26T07:04:39.8179411Z Entering 'third_party/googletest' 2024-06-26T07:04:39.8207133Z Entering 'third_party/ideep' 2024-06-26T07:04:39.8233049Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:04:39.8266266Z Entering 'third_party/ittapi' 2024-06-26T07:04:39.8292130Z Entering 'third_party/kineto' 2024-06-26T07:04:39.8320711Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:04:39.8345872Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:39.8373316Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:39.8401081Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:39.8427200Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:39.8452525Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:04:39.8481069Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:39.8508422Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:39.8535123Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:39.8563203Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:39.8589132Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:04:39.8613984Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:04:39.8642390Z Entering 'third_party/mimalloc' 2024-06-26T07:04:39.8668360Z Entering 'third_party/nccl/nccl' 2024-06-26T07:04:39.8696311Z Entering 'third_party/nlohmann' 2024-06-26T07:04:39.8725601Z Entering 'third_party/onnx' 2024-06-26T07:04:39.8765714Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:04:39.8796827Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:04:39.8824452Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:04:39.8856063Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:04:39.8883451Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:04:39.8910724Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:04:39.8937204Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:04:39.8966297Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:04:39.8992271Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:04:39.9020635Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:04:39.9046716Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:04:39.9075718Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:04:39.9103786Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:04:39.9147461Z Entering 'third_party/pocketfft' 2024-06-26T07:04:39.9174071Z Entering 'third_party/protobuf' 2024-06-26T07:04:39.9205168Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:04:39.9233261Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:04:39.9260738Z Entering 'third_party/psimd' 2024-06-26T07:04:39.9287962Z Entering 'third_party/pthreadpool' 2024-06-26T07:04:39.9317281Z Entering 'third_party/pybind11' 2024-06-26T07:04:39.9346981Z Entering 'third_party/python-peachpy' 2024-06-26T07:04:39.9372833Z Entering 'third_party/sleef' 2024-06-26T07:04:39.9401537Z Entering 'third_party/tensorpipe' 2024-06-26T07:04:39.9427194Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:04:39.9453396Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:04:39.9481036Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:04:39.9507310Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:04:39.9533518Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:04:39.9573136Z ##[endgroup] 2024-06-26T07:04:39.9575138Z ##[group]Persisting credentials for submodules 2024-06-26T07:04:39.9579280Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-06-26T07:04:39.9785950Z Entering 'android/libs/fbjni' 2024-06-26T07:04:39.9822737Z Entering 'third_party/FP16' 2024-06-26T07:04:39.9855197Z Entering 'third_party/FXdiv' 2024-06-26T07:04:39.9890605Z Entering 'third_party/NNPACK' 2024-06-26T07:04:39.9926251Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:04:39.9961419Z Entering 'third_party/XNNPACK' 2024-06-26T07:04:40.0004331Z Entering 'third_party/benchmark' 2024-06-26T07:04:40.0037766Z Entering 'third_party/cpp-httplib' 2024-06-26T07:04:40.0073000Z Entering 'third_party/cpuinfo' 2024-06-26T07:04:40.0109909Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:04:40.0146477Z Entering 'third_party/cutlass' 2024-06-26T07:04:40.0188293Z Entering 'third_party/eigen' 2024-06-26T07:04:40.0223362Z Entering 'third_party/fbgemm' 2024-06-26T07:04:40.0256574Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:04:40.0291850Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:04:40.0324829Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:04:40.0361949Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:04:40.0397330Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:04:40.0432103Z Entering 'third_party/flatbuffers' 2024-06-26T07:04:40.0469521Z Entering 'third_party/fmt' 2024-06-26T07:04:40.0503934Z Entering 'third_party/foxi' 2024-06-26T07:04:40.0538977Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:04:40.0572730Z Entering 'third_party/gloo' 2024-06-26T07:04:40.0609188Z Entering 'third_party/googletest' 2024-06-26T07:04:40.0644008Z Entering 'third_party/ideep' 2024-06-26T07:04:40.0676022Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:04:40.0715166Z Entering 'third_party/ittapi' 2024-06-26T07:04:40.0749509Z Entering 'third_party/kineto' 2024-06-26T07:04:40.0782891Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:04:40.0816674Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:40.0853104Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:40.0887233Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:40.0922195Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:40.0954094Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:04:40.0988620Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:40.1023089Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:40.1057219Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:40.1092967Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:40.1130253Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:04:40.1164678Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:04:40.1201404Z Entering 'third_party/mimalloc' 2024-06-26T07:04:40.1236609Z Entering 'third_party/nccl/nccl' 2024-06-26T07:04:40.1272154Z Entering 'third_party/nlohmann' 2024-06-26T07:04:40.1306225Z Entering 'third_party/onnx' 2024-06-26T07:04:40.1352472Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:04:40.1386980Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:04:40.1423378Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:04:40.1464876Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:04:40.1499004Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:04:40.1533016Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:04:40.1565907Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:04:40.1601675Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:04:40.1633429Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:04:40.1664275Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:04:40.1697743Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:04:40.1732199Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:04:40.1767822Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:04:40.1819259Z Entering 'third_party/pocketfft' 2024-06-26T07:04:40.1854730Z Entering 'third_party/protobuf' 2024-06-26T07:04:40.1892405Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:04:40.1926412Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:04:40.1962632Z Entering 'third_party/psimd' 2024-06-26T07:04:40.1999012Z Entering 'third_party/pthreadpool' 2024-06-26T07:04:40.2032715Z Entering 'third_party/pybind11' 2024-06-26T07:04:40.2067812Z Entering 'third_party/python-peachpy' 2024-06-26T07:04:40.2102525Z Entering 'third_party/sleef' 2024-06-26T07:04:40.2133699Z Entering 'third_party/tensorpipe' 2024-06-26T07:04:40.2167640Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:04:40.2201174Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:04:40.2233060Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:04:40.2267724Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:04:40.2300614Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:04:40.2345155Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-06-26T07:04:40.2553331Z Entering 'android/libs/fbjni' 2024-06-26T07:04:40.2585451Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-06-26T07:04:40.2599023Z Entering 'third_party/FP16' 2024-06-26T07:04:40.2631384Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-06-26T07:04:40.2644033Z Entering 'third_party/FXdiv' 2024-06-26T07:04:40.2674888Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-06-26T07:04:40.2688480Z Entering 'third_party/NNPACK' 2024-06-26T07:04:40.2720054Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-06-26T07:04:40.2733334Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:04:40.2765640Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-06-26T07:04:40.2779794Z Entering 'third_party/XNNPACK' 2024-06-26T07:04:40.2810129Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-06-26T07:04:40.2832788Z Entering 'third_party/benchmark' 2024-06-26T07:04:40.2864830Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:04:40.2877787Z Entering 'third_party/cpp-httplib' 2024-06-26T07:04:40.2911410Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-06-26T07:04:40.2923769Z Entering 'third_party/cpuinfo' 2024-06-26T07:04:40.2956541Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:04:40.2969167Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:04:40.3002057Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-06-26T07:04:40.3014121Z Entering 'third_party/cutlass' 2024-06-26T07:04:40.3046458Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:04:40.3065410Z Entering 'third_party/eigen' 2024-06-26T07:04:40.3096405Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-06-26T07:04:40.3112257Z Entering 'third_party/fbgemm' 2024-06-26T07:04:40.3144550Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-06-26T07:04:40.3157003Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:04:40.3189113Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-06-26T07:04:40.3201567Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:04:40.3233381Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:04:40.3245010Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:04:40.3275616Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:04:40.3294364Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:04:40.3326798Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:04:40.3340087Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:04:40.3371257Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-06-26T07:04:40.3384950Z Entering 'third_party/flatbuffers' 2024-06-26T07:04:40.3418813Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-06-26T07:04:40.3433438Z Entering 'third_party/fmt' 2024-06-26T07:04:40.3464261Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:04:40.3477237Z Entering 'third_party/foxi' 2024-06-26T07:04:40.3508773Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-06-26T07:04:40.3522131Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:04:40.3554145Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-06-26T07:04:40.3567008Z Entering 'third_party/gloo' 2024-06-26T07:04:40.3598548Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-06-26T07:04:40.3612258Z Entering 'third_party/googletest' 2024-06-26T07:04:40.3644655Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:04:40.3658808Z Entering 'third_party/ideep' 2024-06-26T07:04:40.3689721Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-06-26T07:04:40.3702175Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:04:40.3732959Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-06-26T07:04:40.3751338Z Entering 'third_party/ittapi' 2024-06-26T07:04:40.3783098Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-06-26T07:04:40.3795789Z Entering 'third_party/kineto' 2024-06-26T07:04:40.3827543Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-06-26T07:04:40.3839807Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:04:40.3877434Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-06-26T07:04:40.3890589Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:40.3923180Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-06-26T07:04:40.3936644Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:40.3968228Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-06-26T07:04:40.3980758Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:40.4011236Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:04:40.4024607Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:40.4055727Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-06-26T07:04:40.4068539Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:04:40.4100677Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-06-26T07:04:40.4115585Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:40.4147363Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-06-26T07:04:40.4160366Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:40.4191899Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:04:40.4204623Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:40.4237325Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-06-26T07:04:40.4249236Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:40.4280133Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-06-26T07:04:40.4293149Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:04:40.4326367Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-06-26T07:04:40.4340219Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:04:40.4372318Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-06-26T07:04:40.4385477Z Entering 'third_party/mimalloc' 2024-06-26T07:04:40.4418694Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-06-26T07:04:40.4431736Z Entering 'third_party/nccl/nccl' 2024-06-26T07:04:40.4463729Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-06-26T07:04:40.4476976Z Entering 'third_party/nlohmann' 2024-06-26T07:04:40.4509372Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-06-26T07:04:40.4522578Z Entering 'third_party/onnx' 2024-06-26T07:04:40.4553054Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-06-26T07:04:40.4578346Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:04:40.4608486Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:04:40.4622842Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:04:40.4652943Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:04:40.4668986Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:04:40.4702007Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-06-26T07:04:40.4715455Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:04:40.4745702Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:04:40.4758331Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:04:40.4789659Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:04:40.4802838Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:04:40.4834371Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-06-26T07:04:40.4847019Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:04:40.4879721Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-06-26T07:04:40.4893615Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:04:40.4923810Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-06-26T07:04:40.4935126Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:04:40.4967214Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-06-26T07:04:40.4979407Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:04:40.5009092Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-06-26T07:04:40.5021465Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:04:40.5053934Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-06-26T07:04:40.5068636Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:04:40.5101741Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-06-26T07:04:40.5114595Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:04:40.5145284Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-06-26T07:04:40.5173392Z Entering 'third_party/pocketfft' 2024-06-26T07:04:40.5206726Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-06-26T07:04:40.5221564Z Entering 'third_party/protobuf' 2024-06-26T07:04:40.5252137Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-06-26T07:04:40.5266658Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:04:40.5296761Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:04:40.5308925Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:04:40.5339879Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:04:40.5353744Z Entering 'third_party/psimd' 2024-06-26T07:04:40.5385305Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-06-26T07:04:40.5397077Z Entering 'third_party/pthreadpool' 2024-06-26T07:04:40.5428445Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-06-26T07:04:40.5439084Z Entering 'third_party/pybind11' 2024-06-26T07:04:40.5469620Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:04:40.5483240Z Entering 'third_party/python-peachpy' 2024-06-26T07:04:40.5515323Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-06-26T07:04:40.5528033Z Entering 'third_party/sleef' 2024-06-26T07:04:40.5557584Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-06-26T07:04:40.5568503Z Entering 'third_party/tensorpipe' 2024-06-26T07:04:40.5600935Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-06-26T07:04:40.5613947Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:04:40.5644154Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:04:40.5656590Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:04:40.5687700Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-06-26T07:04:40.5700708Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:04:40.5730726Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-06-26T07:04:40.5744080Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:04:40.5774681Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:04:40.5786584Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:04:40.5817700Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-06-26T07:04:40.6997294Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-06-26T07:04:40.7204765Z Entering 'android/libs/fbjni' 2024-06-26T07:04:40.7235091Z Entering 'third_party/FP16' 2024-06-26T07:04:40.7264299Z Entering 'third_party/FXdiv' 2024-06-26T07:04:40.7292569Z Entering 'third_party/NNPACK' 2024-06-26T07:04:40.7319409Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:04:40.7347328Z Entering 'third_party/XNNPACK' 2024-06-26T07:04:40.7386817Z Entering 'third_party/benchmark' 2024-06-26T07:04:40.7413065Z Entering 'third_party/cpp-httplib' 2024-06-26T07:04:40.7440238Z Entering 'third_party/cpuinfo' 2024-06-26T07:04:40.7468108Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:04:40.7501375Z Entering 'third_party/cutlass' 2024-06-26T07:04:40.7535457Z Entering 'third_party/eigen' 2024-06-26T07:04:40.7565316Z Entering 'third_party/fbgemm' 2024-06-26T07:04:40.7596714Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:04:40.7623549Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:04:40.7650160Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:04:40.7683485Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:04:40.7710006Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:04:40.7738292Z Entering 'third_party/flatbuffers' 2024-06-26T07:04:40.7770925Z Entering 'third_party/fmt' 2024-06-26T07:04:40.7797070Z Entering 'third_party/foxi' 2024-06-26T07:04:40.7825803Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:04:40.7854758Z Entering 'third_party/gloo' 2024-06-26T07:04:40.7881836Z Entering 'third_party/googletest' 2024-06-26T07:04:40.7909986Z Entering 'third_party/ideep' 2024-06-26T07:04:40.7937045Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:04:40.7969034Z Entering 'third_party/ittapi' 2024-06-26T07:04:40.7996502Z Entering 'third_party/kineto' 2024-06-26T07:04:40.8026241Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:04:40.8054870Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:40.8085074Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:40.8113158Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:40.8142470Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:40.8169278Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:04:40.8199448Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:40.8226349Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:40.8253607Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:40.8282703Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:40.8309948Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:04:40.8336291Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:04:40.8365003Z Entering 'third_party/mimalloc' 2024-06-26T07:04:40.8391487Z Entering 'third_party/nccl/nccl' 2024-06-26T07:04:40.8418110Z Entering 'third_party/nlohmann' 2024-06-26T07:04:40.8444484Z Entering 'third_party/onnx' 2024-06-26T07:04:40.8485759Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:04:40.8513314Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:04:40.8543767Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:04:40.8571040Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:04:40.8599952Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:04:40.8625635Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:04:40.8653243Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:04:40.8682501Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:04:40.8708097Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:04:40.8734641Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:04:40.8763056Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:04:40.8791538Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:04:40.8819504Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:04:40.8863503Z Entering 'third_party/pocketfft' 2024-06-26T07:04:40.8890386Z Entering 'third_party/protobuf' 2024-06-26T07:04:40.8919355Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:04:40.8948789Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:04:40.8976692Z Entering 'third_party/psimd' 2024-06-26T07:04:40.9002307Z Entering 'third_party/pthreadpool' 2024-06-26T07:04:40.9027821Z Entering 'third_party/pybind11' 2024-06-26T07:04:40.9054365Z Entering 'third_party/python-peachpy' 2024-06-26T07:04:40.9082516Z Entering 'third_party/sleef' 2024-06-26T07:04:40.9108986Z Entering 'third_party/tensorpipe' 2024-06-26T07:04:40.9136344Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:04:40.9164352Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:04:40.9191463Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:04:40.9219686Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:04:40.9246292Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:04:40.9287953Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-06-26T07:04:40.9497097Z Entering 'android/libs/fbjni' 2024-06-26T07:04:40.9525674Z Entering 'third_party/FP16' 2024-06-26T07:04:40.9554365Z Entering 'third_party/FXdiv' 2024-06-26T07:04:40.9584105Z Entering 'third_party/NNPACK' 2024-06-26T07:04:40.9611183Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:04:40.9638840Z Entering 'third_party/XNNPACK' 2024-06-26T07:04:40.9678279Z Entering 'third_party/benchmark' 2024-06-26T07:04:40.9706593Z Entering 'third_party/cpp-httplib' 2024-06-26T07:04:40.9734701Z Entering 'third_party/cpuinfo' 2024-06-26T07:04:40.9764156Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:04:40.9792260Z Entering 'third_party/cutlass' 2024-06-26T07:04:40.9824014Z Entering 'third_party/eigen' 2024-06-26T07:04:40.9852709Z Entering 'third_party/fbgemm' 2024-06-26T07:04:40.9881483Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:04:40.9906892Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:04:40.9933168Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:04:40.9965602Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:04:40.9992158Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:04:41.0020384Z Entering 'third_party/flatbuffers' 2024-06-26T07:04:41.0052683Z Entering 'third_party/fmt' 2024-06-26T07:04:41.0078685Z Entering 'third_party/foxi' 2024-06-26T07:04:41.0104468Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:04:41.0130725Z Entering 'third_party/gloo' 2024-06-26T07:04:41.0159309Z Entering 'third_party/googletest' 2024-06-26T07:04:41.0185930Z Entering 'third_party/ideep' 2024-06-26T07:04:41.0212929Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:04:41.0247865Z Entering 'third_party/ittapi' 2024-06-26T07:04:41.0274955Z Entering 'third_party/kineto' 2024-06-26T07:04:41.0302382Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:04:41.0330246Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:41.0357544Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:41.0383827Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:41.0409872Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:41.0437387Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:04:41.0465305Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:41.0493258Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:41.0520797Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:41.0549764Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:41.0579152Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:04:41.0607692Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:04:41.0636008Z Entering 'third_party/mimalloc' 2024-06-26T07:04:41.0665216Z Entering 'third_party/nccl/nccl' 2024-06-26T07:04:41.0693482Z Entering 'third_party/nlohmann' 2024-06-26T07:04:41.0722867Z Entering 'third_party/onnx' 2024-06-26T07:04:41.0761450Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:04:41.0788682Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:04:41.0819595Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:04:41.0849760Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:04:41.0877682Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:04:41.0903397Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:04:41.0930109Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:04:41.0958030Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:04:41.0983634Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:04:41.1009953Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:04:41.1036654Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:04:41.1065421Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:04:41.1092066Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:04:41.1135276Z Entering 'third_party/pocketfft' 2024-06-26T07:04:41.1164541Z Entering 'third_party/protobuf' 2024-06-26T07:04:41.1194736Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:04:41.1222253Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:04:41.1250113Z Entering 'third_party/psimd' 2024-06-26T07:04:41.1276873Z Entering 'third_party/pthreadpool' 2024-06-26T07:04:41.1306595Z Entering 'third_party/pybind11' 2024-06-26T07:04:41.1335397Z Entering 'third_party/python-peachpy' 2024-06-26T07:04:41.1364552Z Entering 'third_party/sleef' 2024-06-26T07:04:41.1391704Z Entering 'third_party/tensorpipe' 2024-06-26T07:04:41.1420040Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:04:41.1447624Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:04:41.1475230Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:04:41.1503568Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:04:41.1529220Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:04:41.1568019Z ##[endgroup] 2024-06-26T07:04:41.1596873Z [command]/usr/bin/git log -1 --format='%H' 2024-06-26T07:04:41.1617127Z 'b8c4c54d347aa776934c60784e35936878ef18dc' 2024-06-26T07:04:41.1785744Z Prepare all required actions 2024-06-26T07:04:41.1786191Z Getting action download info 2024-06-26T07:04:41.3041697Z ##[group]Run ./.github/actions/setup-linux 2024-06-26T07:04:41.3042118Z env: 2024-06-26T07:04:41.3042395Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:41.3042972Z ##[endgroup] 2024-06-26T07:04:41.3102068Z ##[group]Run set -euo pipefail 2024-06-26T07:04:41.3102505Z set -euo pipefail 2024-06-26T07:04:41.3102894Z function get_ec2_metadata() { 2024-06-26T07:04:41.3103431Z  # Pulled from instance metadata endpoint for EC2 2024-06-26T07:04:41.3104332Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-06-26T07:04:41.3105101Z  category=$1 2024-06-26T07:04:41.3105627Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-06-26T07:04:41.3106265Z  runner_name_str=i-0f77a10328249b1bd 2024-06-26T07:04:41.3106753Z  if [[ -f /.inarc ]]; then 2024-06-26T07:04:41.3107251Z  echo "ARC Runner, no info on ec2 metadata" 2024-06-26T07:04:41.3107874Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-06-26T07:04:41.3108563Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-06-26T07:04:41.3109164Z  else 2024-06-26T07:04:41.3109654Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-06-26T07:04:41.3110231Z  fi 2024-06-26T07:04:41.3110515Z } 2024-06-26T07:04:41.3110886Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-06-26T07:04:41.3111470Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-06-26T07:04:41.3112130Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-06-26T07:04:41.3112698Z echo "system info $(uname -a)" 2024-06-26T07:04:41.3120432Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:41.3120922Z env: 2024-06-26T07:04:41.3121205Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:41.3121557Z ##[endgroup] 2024-06-26T07:04:41.3217120Z ami-id: ami-0ce0c36d7a00b20e2 2024-06-26T07:04:41.3258732Z instance-id: i-0f77a10328249b1bd 2024-06-26T07:04:41.3299031Z instance-type: g5.4xlarge 2024-06-26T07:04:41.3304716Z system info Linux ip-10-0-62-17.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-06-26T07:04:41.3324168Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:04:41.3325209Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:04:41.3333736Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:41.3334232Z env: 2024-06-26T07:04:41.3334516Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:41.3334870Z ##[endgroup] 2024-06-26T07:04:41.3396888Z ##[group]Run if systemctl is-active --quiet docker; then 2024-06-26T07:04:41.3397548Z if systemctl is-active --quiet docker; then 2024-06-26T07:04:41.3398095Z  echo "Docker daemon is running..."; 2024-06-26T07:04:41.3398547Z else 2024-06-26T07:04:41.3399038Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-06-26T07:04:41.3399618Z fi 2024-06-26T07:04:41.3407113Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:41.3407610Z env: 2024-06-26T07:04:41.3407893Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:41.3408245Z ##[endgroup] 2024-06-26T07:04:41.3444169Z Docker daemon is running... 2024-06-26T07:04:41.3486434Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:04:41.3487000Z with: 2024-06-26T07:04:41.3487274Z shell: bash 2024-06-26T07:04:41.3487573Z timeout_minutes: 5 2024-06-26T07:04:41.3487901Z max_attempts: 3 2024-06-26T07:04:41.3488221Z retry_wait_seconds: 30 2024-06-26T07:04:41.3489914Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-06-26T07:04:41.3491358Z polling_interval_seconds: 1 2024-06-26T07:04:41.3491741Z warning_on_retry: true 2024-06-26T07:04:41.3492103Z continue_on_error: false 2024-06-26T07:04:41.3492444Z env: 2024-06-26T07:04:41.3492715Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:41.3493081Z AWS_RETRY_MODE: standard 2024-06-26T07:04:41.3493451Z AWS_MAX_ATTEMPTS: 5 2024-06-26T07:04:41.3493804Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:04:41.3494177Z ##[endgroup] 2024-06-26T07:04:42.2107699Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:04:42.2108731Z Configure a credential helper to remove this warning. See 2024-06-26T07:04:42.2109914Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:04:42.2110454Z 2024-06-26T07:04:42.2110601Z Login Succeeded 2024-06-26T07:04:42.3981136Z Command completed after 1 attempt(s). 2024-06-26T07:04:42.4022816Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:04:42.4023553Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:04:42.4024211Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:04:42.4032025Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:42.4032519Z env: 2024-06-26T07:04:42.4032801Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:42.4033172Z ##[endgroup] 2024-06-26T07:04:42.4098056Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:04:42.4098839Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:04:42.4099443Z # shellcheck disable=SC2046 2024-06-26T07:04:42.4099919Z docker stop $(docker ps -q) || true 2024-06-26T07:04:42.4100423Z # Prune all of the docker images 2024-06-26T07:04:42.4100885Z docker system prune -af 2024-06-26T07:04:42.4107828Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:42.4108336Z env: 2024-06-26T07:04:42.4108622Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:42.4108977Z ##[endgroup] 2024-06-26T07:04:42.4348410Z "docker stop" requires at least 1 argument. 2024-06-26T07:04:42.4349289Z See 'docker stop --help'. 2024-06-26T07:04:42.4349643Z 2024-06-26T07:04:42.4349888Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-06-26T07:04:42.4350847Z 2024-06-26T07:04:42.4351017Z Stop one or more running containers 2024-06-26T07:04:42.4505718Z Total reclaimed space: 0B 2024-06-26T07:04:42.4543852Z ##[group]Run set +e 2024-06-26T07:04:42.4544374Z set +e 2024-06-26T07:04:42.4544871Z set -x 2024-06-26T07:04:42.4545284Z  2024-06-26T07:04:42.4545759Z PT_DOMAIN=download.pytorch.org 2024-06-26T07:04:42.4546989Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-06-26T07:04:42.4548686Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-06-26T07:04:42.4549851Z # one is returned at random 2024-06-26T07:04:42.4550699Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-06-26T07:04:42.4551468Z  2024-06-26T07:04:42.4551948Z if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:04:42.4552896Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-06-26T07:04:42.4554072Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-06-26T07:04:42.4554910Z  2024-06-26T07:04:42.4555401Z  if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:04:42.4556223Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-06-26T07:04:42.4556992Z  exit 1 2024-06-26T07:04:42.4557442Z  fi 2024-06-26T07:04:42.4557853Z fi 2024-06-26T07:04:42.4558450Z  2024-06-26T07:04:42.4559014Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-06-26T07:04:42.4559780Z  # Clean up any old records first 2024-06-26T07:04:42.4560536Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-06-26T07:04:42.4561214Z fi 2024-06-26T07:04:42.4561618Z  2024-06-26T07:04:42.4562247Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-06-26T07:04:42.4563079Z cat /etc/hosts 2024-06-26T07:04:42.4573676Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:42.4574423Z env: 2024-06-26T07:04:42.4574850Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:42.4575383Z ##[endgroup] 2024-06-26T07:04:42.4600536Z + PT_DOMAIN=download.pytorch.org 2024-06-26T07:04:42.4604210Z ++ dig -4 +short download.pytorch.org 2024-06-26T07:04:42.4604656Z ++ tail -n1 2024-06-26T07:04:42.4991689Z + RESOLVED_IP=18.160.10.22 2024-06-26T07:04:42.4992147Z + '[' -z 18.160.10.22 ']' 2024-06-26T07:04:42.4992597Z + grep -r download.pytorch.org /etc/hosts 2024-06-26T07:04:42.4997640Z 18.160.10.28 download.pytorch.org 2024-06-26T07:04:42.4998779Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-06-26T07:04:42.5096840Z + echo '18.160.10.22 download.pytorch.org' 2024-06-26T07:04:42.5098044Z + sudo tee -a /etc/hosts 2024-06-26T07:04:42.5373258Z 18.160.10.22 download.pytorch.org 2024-06-26T07:04:42.5386237Z + cat /etc/hosts 2024-06-26T07:04:42.5390846Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-06-26T07:04:42.5403488Z ::1 localhost6 localhost6.localdomain6 2024-06-26T07:04:42.5404004Z 18.160.10.22 download.pytorch.org 2024-06-26T07:04:42.5606538Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-06-26T07:04:42.5607154Z with: 2024-06-26T07:04:42.5608302Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:42.5609669Z docker-build-dir: .ci/docker 2024-06-26T07:04:42.5610059Z working-directory: . 2024-06-26T07:04:42.5610525Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:42.5611055Z force-push: false 2024-06-26T07:04:42.5611353Z env: 2024-06-26T07:04:42.5611623Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:42.5611962Z ##[endgroup] 2024-06-26T07:04:42.5630810Z ##[group]Run set -ex 2024-06-26T07:04:42.5631175Z set -ex 2024-06-26T07:04:42.5631472Z  2024-06-26T07:04:42.5632156Z # If the docker build directory or the build script doesn't exist, the action will 2024-06-26T07:04:42.5633128Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-06-26T07:04:42.5633934Z # job could then download the pre-built image as usual 2024-06-26T07:04:42.5634676Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-06-26T07:04:42.5635347Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:42.5635983Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:42.5636536Z  2024-06-26T07:04:42.5637043Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-06-26T07:04:42.5637675Z  exit 0 2024-06-26T07:04:42.5637983Z else 2024-06-26T07:04:42.5638359Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:42.5638805Z fi 2024-06-26T07:04:42.5639089Z  2024-06-26T07:04:42.5639556Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-06-26T07:04:42.5640409Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-06-26T07:04:42.5641185Z  # use it as it is, but first let's extract the tag 2024-06-26T07:04:42.5641899Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-06-26T07:04:42.5642633Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:42.5643336Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:42.5643895Z else 2024-06-26T07:04:42.5644332Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-06-26T07:04:42.5645000Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:42.5645893Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:42.5646657Z fi 2024-06-26T07:04:42.5654596Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:42.5655104Z env: 2024-06-26T07:04:42.5655389Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:42.5655756Z REPO_NAME: pytorch 2024-06-26T07:04:42.5656896Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:42.5658373Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:04:42.5658877Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:42.5659400Z ##[endgroup] 2024-06-26T07:04:42.5678597Z + [[ ! -d .ci/docker ]] 2024-06-26T07:04:42.5679176Z + [[ ! -f .ci/docker/build.sh ]] 2024-06-26T07:04:42.5679786Z + echo skip=false 2024-06-26T07:04:42.5682674Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-06-26T07:04:42.5686725Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:42.5688447Z ++ awk -F '[:,]' '{print $2}' 2024-06-26T07:04:42.5690729Z + DOCKER_TAG=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:42.5691682Z + echo docker-tag=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:42.5693314Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:42.5719340Z ##[group]Run set +e 2024-06-26T07:04:42.5719716Z set +e 2024-06-26T07:04:42.5720025Z set -x 2024-06-26T07:04:42.5720544Z  2024-06-26T07:04:42.5720832Z login() { 2024-06-26T07:04:42.5721624Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:04:42.5722340Z } 2024-06-26T07:04:42.5722618Z  2024-06-26T07:04:42.5722903Z retry () { 2024-06-26T07:04:42.5723300Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:04:42.5723749Z } 2024-06-26T07:04:42.5724020Z  2024-06-26T07:04:42.5724348Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:04:42.5724768Z  2024-06-26T07:04:42.5725236Z # Check if image already exists, if it does then skip building it 2024-06-26T07:04:42.5725947Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-06-26T07:04:42.5726458Z  exit 0 2024-06-26T07:04:42.5726765Z fi 2024-06-26T07:04:42.5727040Z  2024-06-26T07:04:42.5727608Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-06-26T07:04:42.5728437Z # be empty. The default action would be to continue rebuild the image 2024-06-26T07:04:42.5729389Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-06-26T07:04:42.5730064Z  # if we're on the base branch then use the parent commit 2024-06-26T07:04:42.5730658Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-06-26T07:04:42.5731093Z else 2024-06-26T07:04:42.5731570Z  # otherwise we're on a PR, so use the most recent base commit 2024-06-26T07:04:42.5732257Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-06-26T07:04:42.5732775Z fi 2024-06-26T07:04:42.5733057Z  2024-06-26T07:04:42.5733381Z if [[ -z "${MERGE_BASE}" ]]; then 2024-06-26T07:04:42.5733882Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:42.5734345Z  2024-06-26T07:04:42.5735002Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-06-26T07:04:42.5735767Z  exit 0 2024-06-26T07:04:42.5736070Z fi 2024-06-26T07:04:42.5736354Z  2024-06-26T07:04:42.5736808Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-06-26T07:04:42.5737982Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-06-26T07:04:42.5738784Z  exit 1 2024-06-26T07:04:42.5739095Z fi 2024-06-26T07:04:42.5739380Z  2024-06-26T07:04:42.5739887Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-06-26T07:04:42.5740832Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-06-26T07:04:42.5741708Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-06-26T07:04:42.5742675Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-06-26T07:04:42.5743774Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-06-26T07:04:42.5744528Z fi 2024-06-26T07:04:42.5744815Z  2024-06-26T07:04:42.5745183Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:04:42.5752870Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:42.5753380Z env: 2024-06-26T07:04:42.5753664Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:42.5754038Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:04:42.5754501Z BASE_REVISION: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:04:42.5755765Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:42.5757010Z DOCKER_TAG: 91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:42.5757657Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:42.5758257Z ##[endgroup] 2024-06-26T07:04:42.5780317Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:42.5781289Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:42.5782171Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:04:42.5783316Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:42.9893229Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:04:42.9894023Z Configure a credential helper to remove this warning. See 2024-06-26T07:04:42.9894835Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:04:42.9895345Z 2024-06-26T07:04:42.9895875Z Login Succeeded 2024-06-26T07:04:42.9906990Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:43.1918997Z { 2024-06-26T07:04:43.1919440Z "schemaVersion": 2, 2024-06-26T07:04:43.1920179Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-06-26T07:04:43.1920986Z "config": { 2024-06-26T07:04:43.1921597Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-06-26T07:04:43.1922143Z "size": 48681, 2024-06-26T07:04:43.1922838Z "digest": "sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd" 2024-06-26T07:04:43.1923710Z }, 2024-06-26T07:04:43.1923996Z "layers": [ 2024-06-26T07:04:43.1924265Z { 2024-06-26T07:04:43.1924704Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1925378Z "size": 28580681, 2024-06-26T07:04:43.1926133Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-06-26T07:04:43.1926751Z }, 2024-06-26T07:04:43.1927000Z { 2024-06-26T07:04:43.1927435Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1927997Z "size": 7943451, 2024-06-26T07:04:43.1928539Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-06-26T07:04:43.1929284Z }, 2024-06-26T07:04:43.1929602Z { 2024-06-26T07:04:43.1930233Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1931015Z + exit 0 2024-06-26T07:04:43.1931423Z "size": 55728572, 2024-06-26T07:04:43.1932167Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-06-26T07:04:43.1932963Z }, 2024-06-26T07:04:43.1933284Z { 2024-06-26T07:04:43.1933846Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1934573Z "size": 186, 2024-06-26T07:04:43.1935267Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-06-26T07:04:43.1936076Z }, 2024-06-26T07:04:43.1936420Z { 2024-06-26T07:04:43.1936936Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1937764Z "size": 6886, 2024-06-26T07:04:43.1938368Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-06-26T07:04:43.1939177Z }, 2024-06-26T07:04:43.1939440Z { 2024-06-26T07:04:43.1939870Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1940429Z "size": 1329001756, 2024-06-26T07:04:43.1941015Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-06-26T07:04:43.1941640Z }, 2024-06-26T07:04:43.1941881Z { 2024-06-26T07:04:43.1942316Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1943009Z "size": 62501, 2024-06-26T07:04:43.1943558Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-06-26T07:04:43.1944160Z }, 2024-06-26T07:04:43.1944410Z { 2024-06-26T07:04:43.1944840Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1945390Z "size": 1684, 2024-06-26T07:04:43.1945916Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-06-26T07:04:43.1946646Z }, 2024-06-26T07:04:43.1946902Z { 2024-06-26T07:04:43.1947329Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1947877Z "size": 1523, 2024-06-26T07:04:43.1948433Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-06-26T07:04:43.1949065Z }, 2024-06-26T07:04:43.1949309Z { 2024-06-26T07:04:43.1949743Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1950299Z "size": 2528295403, 2024-06-26T07:04:43.1950864Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-06-26T07:04:43.1951470Z }, 2024-06-26T07:04:43.1951715Z { 2024-06-26T07:04:43.1952144Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1952688Z "size": 86016, 2024-06-26T07:04:43.1953241Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-06-26T07:04:43.1953871Z }, 2024-06-26T07:04:43.1954116Z { 2024-06-26T07:04:43.1954544Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1955097Z "size": 1823, 2024-06-26T07:04:43.1955635Z "digest": "sha256:309b872da805fcdf74137a887464876fa0615dc8b24199307119eb65fc2bfb49" 2024-06-26T07:04:43.1956238Z }, 2024-06-26T07:04:43.1956479Z { 2024-06-26T07:04:43.1956906Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1957484Z "size": 246737542, 2024-06-26T07:04:43.1958065Z "digest": "sha256:e82932cb002c06a782077b8e5965f8733eb8563406a23cab260d9d8c4a994f7d" 2024-06-26T07:04:43.1958668Z }, 2024-06-26T07:04:43.1958913Z { 2024-06-26T07:04:43.1959343Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1959887Z "size": 545, 2024-06-26T07:04:43.1960435Z "digest": "sha256:ca9dd73e3e9378225ebdcfd12199ef44649e34fe55003cfbc8afe2e8a4527c5b" 2024-06-26T07:04:43.1961063Z }, 2024-06-26T07:04:43.1961310Z { 2024-06-26T07:04:43.1961740Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1962294Z "size": 1283, 2024-06-26T07:04:43.1962823Z "digest": "sha256:16d76a79d5d6616e228537785ec83a869f68a402520bbc6e826201621ca6cb98" 2024-06-26T07:04:43.1963421Z }, 2024-06-26T07:04:43.1963660Z { 2024-06-26T07:04:43.1964088Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1964643Z "size": 485, 2024-06-26T07:04:43.1965185Z "digest": "sha256:4bdf41c9ecbe1651e0986ccbea4663a0556af27b7145a679cfa45c35dc96b891" 2024-06-26T07:04:43.1965803Z }, 2024-06-26T07:04:43.1966051Z { 2024-06-26T07:04:43.1966483Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1967032Z "size": 91710937, 2024-06-26T07:04:43.1967604Z "digest": "sha256:f4863801e30cdbcd1d4e6cb1cbbcde7dbcaf6ae317ca80429295741ce60ac31d" 2024-06-26T07:04:43.1968228Z }, 2024-06-26T07:04:43.1968477Z { 2024-06-26T07:04:43.1968994Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1969624Z "size": 3233, 2024-06-26T07:04:43.1970181Z "digest": "sha256:d37dd9ed8948a0d02e1198bbedadbca36477a0980dd553196385aab0c025581d" 2024-06-26T07:04:43.1970803Z }, 2024-06-26T07:04:43.1971041Z { 2024-06-26T07:04:43.1971473Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1972031Z "size": 1905, 2024-06-26T07:04:43.1972582Z "digest": "sha256:cf2f34f225dbed17c1a9a8b9ea4ad030aba797774a716dc0a82fe212324aa751" 2024-06-26T07:04:43.1973199Z }, 2024-06-26T07:04:43.1973450Z { 2024-06-26T07:04:43.1973881Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1974432Z "size": 702, 2024-06-26T07:04:43.1974977Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:04:43.1975601Z }, 2024-06-26T07:04:43.1975851Z { 2024-06-26T07:04:43.1976342Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1976897Z "size": 2784789002, 2024-06-26T07:04:43.1977469Z "digest": "sha256:3d914950f8addfc4d657a2365c76e1143eb80b710cd3f6e9ae05640217fc6347" 2024-06-26T07:04:43.1978280Z }, 2024-06-26T07:04:43.1978537Z { 2024-06-26T07:04:43.1978973Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1979523Z "size": 381, 2024-06-26T07:04:43.1980062Z "digest": "sha256:96d0e2032184c57e62a652aaffcf0241f0d0c7a23e894ca0ad192c4a64a36583" 2024-06-26T07:04:43.1980676Z }, 2024-06-26T07:04:43.1980923Z { 2024-06-26T07:04:43.1981351Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1981896Z "size": 11879, 2024-06-26T07:04:43.1982437Z "digest": "sha256:06e12ce59790cb8b091e5d619c2a62a28329008ee184a8b9a562e3db12471f30" 2024-06-26T07:04:43.1983039Z }, 2024-06-26T07:04:43.1983286Z { 2024-06-26T07:04:43.1983711Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1984277Z "size": 803, 2024-06-26T07:04:43.1984828Z "digest": "sha256:ed712f63cde7070bdec2a6179c4c35e752c6991ddbc69fe9f2724a9903250ceb" 2024-06-26T07:04:43.1985439Z }, 2024-06-26T07:04:43.1985684Z { 2024-06-26T07:04:43.1986112Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1986667Z "size": 106, 2024-06-26T07:04:43.1987196Z "digest": "sha256:a97a71ef8a4e66234dfc2098834f8395ac5630240e60164d1a251fdb0115ba85" 2024-06-26T07:04:43.1987857Z }, 2024-06-26T07:04:43.1988109Z { 2024-06-26T07:04:43.1988542Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1989088Z "size": 503, 2024-06-26T07:04:43.1989626Z "digest": "sha256:25cd0c482ad612f936b35b4f220c580e16be982226aac8801a6a849a6bfdc221" 2024-06-26T07:04:43.1990248Z }, 2024-06-26T07:04:43.1990494Z { 2024-06-26T07:04:43.1990929Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1991488Z "size": 121477177, 2024-06-26T07:04:43.1992057Z "digest": "sha256:9bf94bc456584365fda21d0ed83b7cc690b08a71727b71c4c1e667e15028a386" 2024-06-26T07:04:43.1992659Z }, 2024-06-26T07:04:43.1992907Z { 2024-06-26T07:04:43.1993334Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1993887Z "size": 109, 2024-06-26T07:04:43.1994416Z "digest": "sha256:de45a14c9b227e4b0c9d45d42223b50a71c03522ddd9e2ca0d9517835206c823" 2024-06-26T07:04:43.1995029Z }, 2024-06-26T07:04:43.1995282Z { 2024-06-26T07:04:43.1995718Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1996266Z "size": 491, 2024-06-26T07:04:43.1996802Z "digest": "sha256:e99241e77209b0b35a0fb59a61c0821fc161190178af7e4af45e6eeaa043375e" 2024-06-26T07:04:43.1997411Z }, 2024-06-26T07:04:43.1997654Z { 2024-06-26T07:04:43.1998091Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.1998639Z "size": 291, 2024-06-26T07:04:43.1999199Z "digest": "sha256:35e8f4e2a42f9df0c969dedd2199cf71719cf5ecfa676dfa014bbdeb4fdb295f" 2024-06-26T07:04:43.1999915Z }, 2024-06-26T07:04:43.2000169Z { 2024-06-26T07:04:43.2000601Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2001153Z "size": 101, 2024-06-26T07:04:43.2001685Z "digest": "sha256:a823ffa4c541394b52953391dc0a6867e26c37fe4a71fa022a2cc3f07f0ba70d" 2024-06-26T07:04:43.2002296Z }, 2024-06-26T07:04:43.2002540Z { 2024-06-26T07:04:43.2002966Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2003518Z "size": 1473, 2024-06-26T07:04:43.2004046Z "digest": "sha256:44ba272eae2064603605a6238b3f35cc6578859d660068e702a047a9cd8ab2fd" 2024-06-26T07:04:43.2004643Z }, 2024-06-26T07:04:43.2004881Z { 2024-06-26T07:04:43.2005307Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2005871Z "size": 446710781, 2024-06-26T07:04:43.2006428Z "digest": "sha256:da02d278e5240c285bc855ce4e42cd6121d5b903787f5c14f03e575ad1268650" 2024-06-26T07:04:43.2007139Z }, 2024-06-26T07:04:43.2007394Z { 2024-06-26T07:04:43.2007822Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2008373Z "size": 161, 2024-06-26T07:04:43.2008999Z "digest": "sha256:73a0ce681c295135456e190578733273276b38b4dd5babbb39298f384d43b1c6" 2024-06-26T07:04:43.2009596Z }, 2024-06-26T07:04:43.2009841Z { 2024-06-26T07:04:43.2010265Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2010817Z "size": 564, 2024-06-26T07:04:43.2011359Z "digest": "sha256:49737bd5ae709dbf7956e21fc8b05ba009e9e6c9b130b0126a73bd3d3f5efc78" 2024-06-26T07:04:43.2011973Z }, 2024-06-26T07:04:43.2012208Z { 2024-06-26T07:04:43.2012635Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2013187Z "size": 35874600, 2024-06-26T07:04:43.2013739Z "digest": "sha256:9cd4c745f8e4f9d5c9bb8050923575afad21f5b7852975a4a1981bc9eff282c5" 2024-06-26T07:04:43.2014354Z }, 2024-06-26T07:04:43.2014598Z { 2024-06-26T07:04:43.2015030Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2015569Z "size": 104, 2024-06-26T07:04:43.2016090Z "digest": "sha256:b2950a0365345d4c17a48627c905fd8930b3b3154c727f0526eb9ca547102bb7" 2024-06-26T07:04:43.2016682Z }, 2024-06-26T07:04:43.2016924Z { 2024-06-26T07:04:43.2017345Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2018130Z "size": 425, 2024-06-26T07:04:43.2018653Z "digest": "sha256:25e19d81d9fc081962ff4cbc09e098f4035419822712c19a410f5282300dfb9e" 2024-06-26T07:04:43.2019252Z }, 2024-06-26T07:04:43.2019491Z { 2024-06-26T07:04:43.2019912Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2020462Z "size": 20262084, 2024-06-26T07:04:43.2021016Z "digest": "sha256:60156b40fcbec2c25f0170b0a82fa8cd9303e0daae8bb1811fd0fc6562e88e8e" 2024-06-26T07:04:43.2021635Z }, 2024-06-26T07:04:43.2021876Z { 2024-06-26T07:04:43.2022306Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2022843Z "size": 439, 2024-06-26T07:04:43.2023364Z "digest": "sha256:7a868239227fd19b30a19927bbc8f058d247403fe5b343c55f4b4b8987918911" 2024-06-26T07:04:43.2023956Z }, 2024-06-26T07:04:43.2024197Z { 2024-06-26T07:04:43.2024616Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2025165Z "size": 702, 2024-06-26T07:04:43.2025695Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:04:43.2026303Z }, 2024-06-26T07:04:43.2026538Z { 2024-06-26T07:04:43.2026960Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2027509Z "size": 143, 2024-06-26T07:04:43.2028030Z "digest": "sha256:83911d7c7ecf11b5932bb7449975c52e7f737a5b867691935e1ccb85ceb25b52" 2024-06-26T07:04:43.2028628Z }, 2024-06-26T07:04:43.2028878Z { 2024-06-26T07:04:43.2029301Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2029976Z "size": 135, 2024-06-26T07:04:43.2030526Z "digest": "sha256:b3c5d66c679f58c71febdd502889bd8cd3d36e5001754cdf1e30eabe21135cce" 2024-06-26T07:04:43.2031146Z }, 2024-06-26T07:04:43.2031387Z { 2024-06-26T07:04:43.2031803Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2032352Z "size": 4911495247, 2024-06-26T07:04:43.2032903Z "digest": "sha256:84b2c7a899820f6acdb22148c5e8346d8b26c33269fd01842c9573515251e1e1" 2024-06-26T07:04:43.2033503Z }, 2024-06-26T07:04:43.2033741Z { 2024-06-26T07:04:43.2034168Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2034708Z "size": 190, 2024-06-26T07:04:43.2035245Z "digest": "sha256:c1068d03a811d8dead5da9e82fdc8869205fc4ee9cb92d4f7a4acfbb5ba83950" 2024-06-26T07:04:43.2035873Z }, 2024-06-26T07:04:43.2036116Z { 2024-06-26T07:04:43.2036608Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2037146Z "size": 565, 2024-06-26T07:04:43.2037676Z "digest": "sha256:671b4a301d81074fca17d36851f81801552c4ecb2f90ff3412aa9ab6a1a91371" 2024-06-26T07:04:43.2038276Z }, 2024-06-26T07:04:43.2038517Z { 2024-06-26T07:04:43.2038936Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2039485Z "size": 43163095, 2024-06-26T07:04:43.2040041Z "digest": "sha256:67760cf5ec46ab58d3ca98637fd9b26cb47a67446bfd0db176fe5e023606aa0d" 2024-06-26T07:04:43.2040653Z }, 2024-06-26T07:04:43.2040891Z { 2024-06-26T07:04:43.2041316Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2041860Z "size": 106, 2024-06-26T07:04:43.2042389Z "digest": "sha256:ddc9740bcea2b301e99b50525059aea8d9c7121a1c8eab566561a8968438a8c2" 2024-06-26T07:04:43.2042999Z }, 2024-06-26T07:04:43.2043240Z { 2024-06-26T07:04:43.2043663Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2044230Z "size": 1213, 2024-06-26T07:04:43.2044772Z "digest": "sha256:fb2e977685609fcc01de170ba7f50a573e73e6520e47253717e92a28952c5ee8" 2024-06-26T07:04:43.2045378Z }, 2024-06-26T07:04:43.2045625Z { 2024-06-26T07:04:43.2046048Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2046589Z "size": 702, 2024-06-26T07:04:43.2047128Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:04:43.2047773Z }, 2024-06-26T07:04:43.2048029Z { 2024-06-26T07:04:43.2048452Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2049097Z "size": 138, 2024-06-26T07:04:43.2049635Z "digest": "sha256:b599c5a8c7cb461cbd7b3c98b99960b8eb04506e05e79db1cc0cfadb14a47e91" 2024-06-26T07:04:43.2050261Z }, 2024-06-26T07:04:43.2050504Z { 2024-06-26T07:04:43.2050939Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2051489Z "size": 120, 2024-06-26T07:04:43.2052037Z "digest": "sha256:e08e88c1366827455ee7ddbf92b6ae615ceb3cbd3fb11cd03ddd31bd3fa94a9c" 2024-06-26T07:04:43.2052656Z }, 2024-06-26T07:04:43.2052900Z { 2024-06-26T07:04:43.2053322Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2053875Z "size": 1904157495, 2024-06-26T07:04:43.2054440Z "digest": "sha256:12ba7431ff38de8cf6af3052a2a52e79f4e712b8c99f5bafde3c20e63b1f570f" 2024-06-26T07:04:43.2055049Z }, 2024-06-26T07:04:43.2055293Z { 2024-06-26T07:04:43.2055717Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2056263Z "size": 172, 2024-06-26T07:04:43.2056806Z "digest": "sha256:cf7c2cb896cba6a594028cc0fd246e127a2f4481d48905a3aa12c5f5872332d4" 2024-06-26T07:04:43.2057421Z }, 2024-06-26T07:04:43.2057890Z { 2024-06-26T07:04:43.2058322Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2058866Z "size": 908, 2024-06-26T07:04:43.2059421Z "digest": "sha256:3a778ddfac2af5b3a11f49ddf2f3f1ad1731f74ba691f434ab69966d1e2a189f" 2024-06-26T07:04:43.2060140Z }, 2024-06-26T07:04:43.2060387Z { 2024-06-26T07:04:43.2060815Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2061359Z "size": 702, 2024-06-26T07:04:43.2061898Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:04:43.2062507Z }, 2024-06-26T07:04:43.2062750Z { 2024-06-26T07:04:43.2063180Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2063723Z "size": 134, 2024-06-26T07:04:43.2064246Z "digest": "sha256:236c8f0245f4b40616a4322eaf1e1b3e178d5990db1fcaba39622ea4e79c06a1" 2024-06-26T07:04:43.2064850Z }, 2024-06-26T07:04:43.2065094Z { 2024-06-26T07:04:43.2065514Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2066050Z "size": 32, 2024-06-26T07:04:43.2066585Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:04:43.2067265Z }, 2024-06-26T07:04:43.2067525Z { 2024-06-26T07:04:43.2067987Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2068536Z "size": 156, 2024-06-26T07:04:43.2069079Z "digest": "sha256:dda1ce302b9a9df0c8f1145d2e3eaa03c96909bb45a792451f1fcb555b06738d" 2024-06-26T07:04:43.2069691Z }, 2024-06-26T07:04:43.2069937Z { 2024-06-26T07:04:43.2070361Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2070911Z "size": 1841, 2024-06-26T07:04:43.2071450Z "digest": "sha256:e797fe86caef19319e03f8dca92cf96838ab3bb3644f4447c6dfc04230118afa" 2024-06-26T07:04:43.2072065Z }, 2024-06-26T07:04:43.2072309Z { 2024-06-26T07:04:43.2072732Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2073283Z "size": 7529784, 2024-06-26T07:04:43.2073834Z "digest": "sha256:967190076fc0ab2bb2fc0d74e58edd3d662a537d702ff3f8b6e876230154a16f" 2024-06-26T07:04:43.2074448Z }, 2024-06-26T07:04:43.2074685Z { 2024-06-26T07:04:43.2075116Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2075664Z "size": 165, 2024-06-26T07:04:43.2076192Z "digest": "sha256:4fb0a83f06855534ef82b80dcf7c685303edb2bcbd446196066dc2ad2dd71281" 2024-06-26T07:04:43.2076794Z }, 2024-06-26T07:04:43.2077042Z { 2024-06-26T07:04:43.2077466Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2078018Z "size": 7944, 2024-06-26T07:04:43.2078556Z "digest": "sha256:68af2ec4e30a33f26947ece9f3a31b7d3b04c2bca5865d4b48618457832a1da2" 2024-06-26T07:04:43.2079173Z }, 2024-06-26T07:04:43.2079418Z { 2024-06-26T07:04:43.2079841Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2080385Z "size": 8070, 2024-06-26T07:04:43.2080915Z "digest": "sha256:cc91933e985a044c6df53caa15488d2285896f5e991f38ec068c253ceec53f14" 2024-06-26T07:04:43.2081526Z }, 2024-06-26T07:04:43.2081765Z { 2024-06-26T07:04:43.2082195Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2082739Z "size": 303, 2024-06-26T07:04:43.2083279Z "digest": "sha256:9f3187916e7dcad33cc572f8f0e949d2f507b887f99cfd6acefd14826b44960d" 2024-06-26T07:04:43.2083884Z }, 2024-06-26T07:04:43.2084126Z { 2024-06-26T07:04:43.2084553Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2085093Z "size": 7629071, 2024-06-26T07:04:43.2085618Z "digest": "sha256:52608399485738d10d779254946c9e1d536010dd45593c69bf1eb9f62f85dc24" 2024-06-26T07:04:43.2086208Z }, 2024-06-26T07:04:43.2086454Z { 2024-06-26T07:04:43.2086875Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2087421Z "size": 108, 2024-06-26T07:04:43.2088010Z "digest": "sha256:05a6f6dcc1c00bac6660923a22adc7d7c34416718f8c89daf9b3351d9365f694" 2024-06-26T07:04:43.2088620Z }, 2024-06-26T07:04:43.2088994Z { 2024-06-26T07:04:43.2089422Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2090036Z "size": 54145775, 2024-06-26T07:04:43.2090598Z "digest": "sha256:6bc9dc8315b09b27c976bfba9700767ba412d72fd0528b18bbe8f8e62005ec22" 2024-06-26T07:04:43.2091204Z }, 2024-06-26T07:04:43.2091451Z { 2024-06-26T07:04:43.2091874Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2092416Z "size": 474, 2024-06-26T07:04:43.2092948Z "digest": "sha256:1b0e27c35f07b3fbf0aeb548d1949bf1d60a420136da21789d94d196c891d981" 2024-06-26T07:04:43.2093561Z }, 2024-06-26T07:04:43.2093803Z { 2024-06-26T07:04:43.2094221Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2094765Z "size": 1374858747, 2024-06-26T07:04:43.2095312Z "digest": "sha256:95237e7311a1c5ce6937db7196c866632f44e8788514a5c0eb4a3f04da4017e6" 2024-06-26T07:04:43.2095906Z }, 2024-06-26T07:04:43.2096150Z { 2024-06-26T07:04:43.2096638Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2097183Z "size": 106, 2024-06-26T07:04:43.2097912Z "digest": "sha256:789eff6c51aada70bf75eb1acafed63c0d839b768638001500616ff6e53b854a" 2024-06-26T07:04:43.2098547Z }, 2024-06-26T07:04:43.2098790Z { 2024-06-26T07:04:43.2099217Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2099758Z "size": 558, 2024-06-26T07:04:43.2100302Z "digest": "sha256:de0f8a8c45c6077b3c5ba3b84d7f07de98353fd618dad9a1e2f368ffe00493c7" 2024-06-26T07:04:43.2100923Z }, 2024-06-26T07:04:43.2101166Z { 2024-06-26T07:04:43.2101585Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2102136Z "size": 46248513, 2024-06-26T07:04:43.2102682Z "digest": "sha256:2060028f74888d218bebdd10d723f66c33ee1c708935c4ffd4654d34912ac4dc" 2024-06-26T07:04:43.2103283Z }, 2024-06-26T07:04:43.2103523Z { 2024-06-26T07:04:43.2103948Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2104500Z "size": 111, 2024-06-26T07:04:43.2105029Z "digest": "sha256:e412c7f3b2ded440992955bd92512345cbece8a8894cd3592f714ebc758ad1a0" 2024-06-26T07:04:43.2105635Z }, 2024-06-26T07:04:43.2105880Z { 2024-06-26T07:04:43.2106304Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2106842Z "size": 32, 2024-06-26T07:04:43.2107378Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:04:43.2108042Z }, 2024-06-26T07:04:43.2108282Z { 2024-06-26T07:04:43.2108703Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2109253Z "size": 32, 2024-06-26T07:04:43.2109789Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:04:43.2110402Z }, 2024-06-26T07:04:43.2110639Z { 2024-06-26T07:04:43.2111061Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2111615Z "size": 32, 2024-06-26T07:04:43.2112149Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:04:43.2112767Z }, 2024-06-26T07:04:43.2113010Z { 2024-06-26T07:04:43.2113441Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:04:43.2113984Z "size": 32, 2024-06-26T07:04:43.2114518Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:04:43.2115129Z } 2024-06-26T07:04:43.2115374Z ] 2024-06-26T07:04:43.2115610Z } 2024-06-26T07:04:43.2229873Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:04:43.2230365Z tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:04:43.2230915Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-06-26T07:04:43.2238894Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:43.2239401Z env: 2024-06-26T07:04:43.2239685Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:43.2240834Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:43.2242009Z ##[endgroup] 2024-06-26T07:04:43.2265733Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:43.2315544Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-06-26T07:04:43.2316121Z with: 2024-06-26T07:04:43.2317190Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:43.2318493Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:43.2319010Z env: 2024-06-26T07:04:43.2319289Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:43.2319638Z ##[endgroup] 2024-06-26T07:04:43.2334825Z ##[group]Run set -x 2024-06-26T07:04:43.2335178Z set -x 2024-06-26T07:04:43.2335616Z set +e 2024-06-26T07:04:43.2335912Z  2024-06-26T07:04:43.2336190Z login() { 2024-06-26T07:04:43.2336853Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:04:43.2337723Z } 2024-06-26T07:04:43.2338006Z  2024-06-26T07:04:43.2338310Z retry () { 2024-06-26T07:04:43.2338704Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:04:43.2339146Z } 2024-06-26T07:04:43.2339426Z  2024-06-26T07:04:43.2339749Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:04:43.2340162Z  2024-06-26T07:04:43.2340440Z set -e 2024-06-26T07:04:43.2340914Z # ignore output since only exit code is used for conditional 2024-06-26T07:04:43.2341620Z # only pull docker image if it's not available locally 2024-06-26T07:04:43.2342383Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-06-26T07:04:43.2343089Z  retry docker pull "${DOCKER_IMAGE}" 2024-06-26T07:04:43.2343530Z fi 2024-06-26T07:04:43.2351154Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:04:43.2351655Z env: 2024-06-26T07:04:43.2351940Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:04:43.2353083Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:43.2354384Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:43.2354904Z ##[endgroup] 2024-06-26T07:04:43.2373333Z + set +e 2024-06-26T07:04:43.2374096Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:43.2374759Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:43.2375775Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:04:43.2376663Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:04:43.6435117Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:04:43.6435919Z Configure a credential helper to remove this warning. See 2024-06-26T07:04:43.6436745Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:04:43.6437290Z 2024-06-26T07:04:43.6437422Z Login Succeeded 2024-06-26T07:04:43.6445749Z + set -e 2024-06-26T07:04:43.6447035Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:43.6572578Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:43.6574687Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:04:43.8473480Z 91382da70d5719cd7007b6b80b71d2f48398f6b7: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-06-26T07:04:43.8477438Z 7a2c55901189: Pulling fs layer 2024-06-26T07:04:43.8477980Z 224fe954d725: Pulling fs layer 2024-06-26T07:04:43.8478859Z 75722010b82e: Pulling fs layer 2024-06-26T07:04:43.8479441Z d527cbbb87e3: Pulling fs layer 2024-06-26T07:04:43.8479993Z b57676e46aee: Pulling fs layer 2024-06-26T07:04:43.8480640Z a8c1e85b5e14: Pulling fs layer 2024-06-26T07:04:43.8481224Z a41a8d1c11c8: Pulling fs layer 2024-06-26T07:04:43.8481780Z 0c1227890755: Pulling fs layer 2024-06-26T07:04:43.8482363Z d8d1234baab3: Pulling fs layer 2024-06-26T07:04:43.8482945Z 7ed32bc8e469: Pulling fs layer 2024-06-26T07:04:43.8483399Z ec1e7978c1fe: Pulling fs layer 2024-06-26T07:04:43.8483832Z 309b872da805: Pulling fs layer 2024-06-26T07:04:43.8484378Z e82932cb002c: Pulling fs layer 2024-06-26T07:04:43.8485199Z b57676e46aee: Waiting 2024-06-26T07:04:43.8485537Z ca9dd73e3e93: Pulling fs layer 2024-06-26T07:04:43.8485999Z 16d76a79d5d6: Pulling fs layer 2024-06-26T07:04:43.8486378Z 4bdf41c9ecbe: Pulling fs layer 2024-06-26T07:04:43.8486822Z a8c1e85b5e14: Waiting 2024-06-26T07:04:43.8487158Z f4863801e30c: Pulling fs layer 2024-06-26T07:04:43.8487628Z d37dd9ed8948: Pulling fs layer 2024-06-26T07:04:43.8488011Z a41a8d1c11c8: Waiting 2024-06-26T07:04:43.8488337Z cf2f34f225db: Pulling fs layer 2024-06-26T07:04:43.8488850Z df546d7b3e34: Pulling fs layer 2024-06-26T07:04:43.8489254Z 7ed32bc8e469: Waiting 2024-06-26T07:04:43.8489647Z 3d914950f8ad: Pulling fs layer 2024-06-26T07:04:43.8490024Z 96d0e2032184: Pulling fs layer 2024-06-26T07:04:43.8490378Z 309b872da805: Waiting 2024-06-26T07:04:43.8490749Z 06e12ce59790: Pulling fs layer 2024-06-26T07:04:43.8491104Z 0c1227890755: Waiting 2024-06-26T07:04:43.8491429Z ed712f63cde7: Pulling fs layer 2024-06-26T07:04:43.8491787Z e82932cb002c: Waiting 2024-06-26T07:04:43.8492130Z a97a71ef8a4e: Pulling fs layer 2024-06-26T07:04:43.8492482Z ca9dd73e3e93: Waiting 2024-06-26T07:04:43.8492816Z 25cd0c482ad6: Pulling fs layer 2024-06-26T07:04:43.8493192Z 9bf94bc45658: Pulling fs layer 2024-06-26T07:04:43.8493550Z d8d1234baab3: Waiting 2024-06-26T07:04:43.8493881Z de45a14c9b22: Pulling fs layer 2024-06-26T07:04:43.8494248Z ec1e7978c1fe: Waiting 2024-06-26T07:04:43.8494586Z e99241e77209: Pulling fs layer 2024-06-26T07:04:43.8494965Z 35e8f4e2a42f: Pulling fs layer 2024-06-26T07:04:43.8495330Z a823ffa4c541: Pulling fs layer 2024-06-26T07:04:43.8495685Z d37dd9ed8948: Waiting 2024-06-26T07:04:43.8496047Z 16d76a79d5d6: Waiting 2024-06-26T07:04:43.8496373Z 44ba272eae20: Pulling fs layer 2024-06-26T07:04:43.8496793Z cf2f34f225db: Waiting 2024-06-26T07:04:43.8497120Z 3d914950f8ad: Waiting 2024-06-26T07:04:43.8497720Z da02d278e524: Pulling fs layer 2024-06-26T07:04:43.8498086Z d527cbbb87e3: Waiting 2024-06-26T07:04:43.8498417Z 73a0ce681c29: Pulling fs layer 2024-06-26T07:04:43.8498787Z 49737bd5ae70: Pulling fs layer 2024-06-26T07:04:43.8499155Z 9cd4c745f8e4: Pulling fs layer 2024-06-26T07:04:43.8499510Z 06e12ce59790: Waiting 2024-06-26T07:04:43.8499878Z 4bdf41c9ecbe: Waiting 2024-06-26T07:04:43.8500208Z b2950a036534: Pulling fs layer 2024-06-26T07:04:43.8500577Z 25e19d81d9fc: Pulling fs layer 2024-06-26T07:04:43.8500926Z ed712f63cde7: Waiting 2024-06-26T07:04:43.8501265Z 60156b40fcbe: Pulling fs layer 2024-06-26T07:04:43.8501632Z 7a868239227f: Pulling fs layer 2024-06-26T07:04:43.8501983Z a97a71ef8a4e: Waiting 2024-06-26T07:04:43.8502291Z 25cd0c482ad6: Waiting 2024-06-26T07:04:43.8502607Z df546d7b3e34: Waiting 2024-06-26T07:04:43.8502923Z 35e8f4e2a42f: Waiting 2024-06-26T07:04:43.8503243Z 83911d7c7ecf: Pulling fs layer 2024-06-26T07:04:43.8503616Z b3c5d66c679f: Pulling fs layer 2024-06-26T07:04:43.8503972Z a823ffa4c541: Waiting 2024-06-26T07:04:43.8504352Z 84b2c7a89982: Pulling fs layer 2024-06-26T07:04:43.8504717Z c1068d03a811: Pulling fs layer 2024-06-26T07:04:43.8505085Z 671b4a301d81: Pulling fs layer 2024-06-26T07:04:43.8505442Z 44ba272eae20: Waiting 2024-06-26T07:04:43.8505795Z 96d0e2032184: Waiting 2024-06-26T07:04:43.8506114Z 67760cf5ec46: Pulling fs layer 2024-06-26T07:04:43.8506487Z ddc9740bcea2: Pulling fs layer 2024-06-26T07:04:43.8506841Z 7a868239227f: Waiting 2024-06-26T07:04:43.8507145Z 9cd4c745f8e4: Waiting 2024-06-26T07:04:43.8507586Z 83911d7c7ecf: Waiting 2024-06-26T07:04:43.8507903Z 84b2c7a89982: Waiting 2024-06-26T07:04:43.8508229Z fb2e97768560: Pulling fs layer 2024-06-26T07:04:43.8508579Z b3c5d66c679f: Waiting 2024-06-26T07:04:43.8508887Z 671b4a301d81: Waiting 2024-06-26T07:04:43.8509213Z b599c5a8c7cb: Pulling fs layer 2024-06-26T07:04:43.8509562Z 49737bd5ae70: Waiting 2024-06-26T07:04:43.8509891Z e08e88c13668: Pulling fs layer 2024-06-26T07:04:43.8510247Z 73a0ce681c29: Waiting 2024-06-26T07:04:43.8510573Z 12ba7431ff38: Pulling fs layer 2024-06-26T07:04:43.8510920Z da02d278e524: Waiting 2024-06-26T07:04:43.8511229Z 67760cf5ec46: Waiting 2024-06-26T07:04:43.8511537Z de45a14c9b22: Waiting 2024-06-26T07:04:43.8511978Z cf7c2cb896cb: Pulling fs layer 2024-06-26T07:04:43.8512347Z 3a778ddfac2a: Pulling fs layer 2024-06-26T07:04:43.8512705Z b2950a036534: Waiting 2024-06-26T07:04:43.8513031Z 236c8f0245f4: Pulling fs layer 2024-06-26T07:04:43.8513393Z 4f4fb700ef54: Pulling fs layer 2024-06-26T07:04:43.8513750Z ddc9740bcea2: Waiting 2024-06-26T07:04:43.8514070Z e08e88c13668: Waiting 2024-06-26T07:04:43.8514382Z cf7c2cb896cb: Waiting 2024-06-26T07:04:43.8514705Z dda1ce302b9a: Pulling fs layer 2024-06-26T07:04:43.8515083Z e797fe86caef: Pulling fs layer 2024-06-26T07:04:43.8515446Z dda1ce302b9a: Waiting 2024-06-26T07:04:43.8515774Z 967190076fc0: Pulling fs layer 2024-06-26T07:04:43.8516119Z 3a778ddfac2a: Waiting 2024-06-26T07:04:43.8516447Z 4fb0a83f0685: Pulling fs layer 2024-06-26T07:04:43.8516801Z 236c8f0245f4: Waiting 2024-06-26T07:04:43.8517101Z 967190076fc0: Waiting 2024-06-26T07:04:43.8517412Z 4f4fb700ef54: Waiting 2024-06-26T07:04:43.8517737Z 68af2ec4e30a: Pulling fs layer 2024-06-26T07:04:43.8518102Z 12ba7431ff38: Waiting 2024-06-26T07:04:43.8518419Z cc91933e985a: Pulling fs layer 2024-06-26T07:04:43.8518772Z 60156b40fcbe: Waiting 2024-06-26T07:04:43.8519083Z 68af2ec4e30a: Waiting 2024-06-26T07:04:43.8519402Z 9f3187916e7d: Pulling fs layer 2024-06-26T07:04:43.8519757Z e797fe86caef: Waiting 2024-06-26T07:04:43.8520082Z 526083994857: Pulling fs layer 2024-06-26T07:04:43.8520453Z 05a6f6dcc1c0: Pulling fs layer 2024-06-26T07:04:43.8520801Z cc91933e985a: Waiting 2024-06-26T07:04:43.8521109Z e99241e77209: Waiting 2024-06-26T07:04:43.8521416Z 526083994857: Waiting 2024-06-26T07:04:43.8521720Z fb2e97768560: Waiting 2024-06-26T07:04:43.8522039Z 6bc9dc8315b0: Pulling fs layer 2024-06-26T07:04:43.8522410Z 1b0e27c35f07: Pulling fs layer 2024-06-26T07:04:43.8522819Z 05a6f6dcc1c0: Waiting 2024-06-26T07:04:43.8523144Z 95237e7311a1: Pulling fs layer 2024-06-26T07:04:43.8523520Z 789eff6c51aa: Pulling fs layer 2024-06-26T07:04:43.8523881Z 6bc9dc8315b0: Waiting 2024-06-26T07:04:43.8524213Z de0f8a8c45c6: Pulling fs layer 2024-06-26T07:04:43.8524574Z 1b0e27c35f07: Waiting 2024-06-26T07:04:43.8524904Z 2060028f7488: Pulling fs layer 2024-06-26T07:04:43.8525278Z e412c7f3b2de: Pulling fs layer 2024-06-26T07:04:43.8525675Z 789eff6c51aa: Waiting 2024-06-26T07:04:43.8525984Z 2060028f7488: Waiting 2024-06-26T07:04:43.9814085Z 224fe954d725: Verifying Checksum 2024-06-26T07:04:43.9814595Z 224fe954d725: Download complete 2024-06-26T07:04:44.0717258Z d527cbbb87e3: Verifying Checksum 2024-06-26T07:04:44.0717861Z d527cbbb87e3: Download complete 2024-06-26T07:04:44.1463049Z b57676e46aee: Verifying Checksum 2024-06-26T07:04:44.1463562Z b57676e46aee: Download complete 2024-06-26T07:04:44.1918953Z 7a2c55901189: Verifying Checksum 2024-06-26T07:04:44.1919545Z 7a2c55901189: Download complete 2024-06-26T07:04:44.2768910Z a41a8d1c11c8: Download complete 2024-06-26T07:04:44.3575509Z 0c1227890755: Verifying Checksum 2024-06-26T07:04:44.3575970Z 0c1227890755: Download complete 2024-06-26T07:04:44.4316845Z d8d1234baab3: Verifying Checksum 2024-06-26T07:04:44.4622519Z d8d1234baab3: Download complete 2024-06-26T07:04:44.4623011Z 75722010b82e: Verifying Checksum 2024-06-26T07:04:44.4623522Z 75722010b82e: Download complete 2024-06-26T07:04:44.5403699Z ec1e7978c1fe: Verifying Checksum 2024-06-26T07:04:44.5404173Z ec1e7978c1fe: Download complete 2024-06-26T07:04:44.6517968Z 309b872da805: Verifying Checksum 2024-06-26T07:04:44.6518622Z 309b872da805: Download complete 2024-06-26T07:04:45.2369295Z 7a2c55901189: Pull complete 2024-06-26T07:04:45.5298449Z 224fe954d725: Pull complete 2024-06-26T07:04:46.4602807Z 75722010b82e: Pull complete 2024-06-26T07:04:46.5152661Z d527cbbb87e3: Pull complete 2024-06-26T07:04:46.5742032Z b57676e46aee: Pull complete 2024-06-26T07:04:47.1714618Z e82932cb002c: Verifying Checksum 2024-06-26T07:04:47.1715093Z e82932cb002c: Download complete 2024-06-26T07:04:47.2501787Z ca9dd73e3e93: Download complete 2024-06-26T07:04:47.3251819Z 16d76a79d5d6: Verifying Checksum 2024-06-26T07:04:47.3872506Z 4bdf41c9ecbe: Verifying Checksum 2024-06-26T07:04:47.3873188Z 4bdf41c9ecbe: Download complete 2024-06-26T07:04:48.3543050Z f4863801e30c: Verifying Checksum 2024-06-26T07:04:48.3543727Z f4863801e30c: Download complete 2024-06-26T07:04:48.4308018Z d37dd9ed8948: Verifying Checksum 2024-06-26T07:04:48.4308583Z d37dd9ed8948: Download complete 2024-06-26T07:04:48.5051394Z cf2f34f225db: Download complete 2024-06-26T07:04:48.6031204Z df546d7b3e34: Verifying Checksum 2024-06-26T07:04:48.6031905Z df546d7b3e34: Download complete 2024-06-26T07:04:57.4951159Z a8c1e85b5e14: Verifying Checksum 2024-06-26T07:04:57.4951850Z a8c1e85b5e14: Download complete 2024-06-26T07:04:57.5828528Z 96d0e2032184: Verifying Checksum 2024-06-26T07:04:57.5829178Z 96d0e2032184: Download complete 2024-06-26T07:04:57.6600209Z 06e12ce59790: Verifying Checksum 2024-06-26T07:04:57.6600857Z 06e12ce59790: Download complete 2024-06-26T07:04:57.7275510Z ed712f63cde7: Download complete 2024-06-26T07:04:57.8048531Z a97a71ef8a4e: Verifying Checksum 2024-06-26T07:04:57.8049306Z a97a71ef8a4e: Download complete 2024-06-26T07:04:57.8708223Z 25cd0c482ad6: Download complete 2024-06-26T07:04:59.1544948Z 9bf94bc45658: Verifying Checksum 2024-06-26T07:04:59.1545422Z 9bf94bc45658: Download complete 2024-06-26T07:04:59.2229483Z de45a14c9b22: Verifying Checksum 2024-06-26T07:04:59.2230287Z de45a14c9b22: Download complete 2024-06-26T07:04:59.2929525Z e99241e77209: Verifying Checksum 2024-06-26T07:04:59.2930187Z e99241e77209: Download complete 2024-06-26T07:04:59.3626962Z 35e8f4e2a42f: Verifying Checksum 2024-06-26T07:04:59.3627534Z 35e8f4e2a42f: Download complete 2024-06-26T07:04:59.4279072Z a823ffa4c541: Download complete 2024-06-26T07:04:59.5007767Z 44ba272eae20: Verifying Checksum 2024-06-26T07:04:59.5008272Z 44ba272eae20: Download complete 2024-06-26T07:05:04.0170177Z da02d278e524: Verifying Checksum 2024-06-26T07:05:04.0170800Z da02d278e524: Download complete 2024-06-26T07:05:04.1023536Z 73a0ce681c29: Download complete 2024-06-26T07:05:04.1759047Z 49737bd5ae70: Download complete 2024-06-26T07:05:04.6236045Z 9cd4c745f8e4: Verifying Checksum 2024-06-26T07:05:04.6236763Z 9cd4c745f8e4: Download complete 2024-06-26T07:05:04.6965442Z b2950a036534: Download complete 2024-06-26T07:05:04.7640611Z 25e19d81d9fc: Verifying Checksum 2024-06-26T07:05:04.7641108Z 25e19d81d9fc: Download complete 2024-06-26T07:05:05.0727981Z 60156b40fcbe: Verifying Checksum 2024-06-26T07:05:05.0728493Z 60156b40fcbe: Download complete 2024-06-26T07:05:05.1492486Z 7a868239227f: Verifying Checksum 2024-06-26T07:05:05.1493106Z 7a868239227f: Download complete 2024-06-26T07:05:05.2152830Z 83911d7c7ecf: Download complete 2024-06-26T07:05:05.3069000Z b3c5d66c679f: Download complete 2024-06-26T07:05:09.7862247Z 7ed32bc8e469: Verifying Checksum 2024-06-26T07:05:09.7862733Z 7ed32bc8e469: Download complete 2024-06-26T07:05:09.8584412Z c1068d03a811: Download complete 2024-06-26T07:05:09.9688522Z 671b4a301d81: Download complete 2024-06-26T07:05:10.3373649Z a8c1e85b5e14: Pull complete 2024-06-26T07:05:10.4730962Z 67760cf5ec46: Verifying Checksum 2024-06-26T07:05:10.4731450Z 67760cf5ec46: Download complete 2024-06-26T07:05:10.5480630Z ddc9740bcea2: Verifying Checksum 2024-06-26T07:05:10.5481140Z ddc9740bcea2: Download complete 2024-06-26T07:05:10.5667440Z a41a8d1c11c8: Pull complete 2024-06-26T07:05:10.6185201Z fb2e97768560: Verifying Checksum 2024-06-26T07:05:10.6185820Z fb2e97768560: Download complete 2024-06-26T07:05:10.6931367Z b599c5a8c7cb: Verifying Checksum 2024-06-26T07:05:10.6931861Z b599c5a8c7cb: Download complete 2024-06-26T07:05:10.7801008Z e08e88c13668: Verifying Checksum 2024-06-26T07:05:10.7801491Z e08e88c13668: Download complete 2024-06-26T07:05:10.7802065Z 0c1227890755: Pull complete 2024-06-26T07:05:11.0060833Z d8d1234baab3: Pull complete 2024-06-26T07:05:16.4992171Z 3d914950f8ad: Verifying Checksum 2024-06-26T07:05:16.4992746Z 3d914950f8ad: Download complete 2024-06-26T07:05:16.5763308Z cf7c2cb896cb: Verifying Checksum 2024-06-26T07:05:16.6428971Z 3a778ddfac2a: Verifying Checksum 2024-06-26T07:05:16.6429630Z 3a778ddfac2a: Download complete 2024-06-26T07:05:16.7144729Z 236c8f0245f4: Verifying Checksum 2024-06-26T07:05:16.7145358Z 236c8f0245f4: Download complete 2024-06-26T07:05:16.7232104Z 4f4fb700ef54: Verifying Checksum 2024-06-26T07:05:16.7232706Z 4f4fb700ef54: Download complete 2024-06-26T07:05:16.8062969Z dda1ce302b9a: Download complete 2024-06-26T07:05:16.8754445Z e797fe86caef: Verifying Checksum 2024-06-26T07:05:16.8754960Z e797fe86caef: Download complete 2024-06-26T07:05:16.9998480Z 967190076fc0: Verifying Checksum 2024-06-26T07:05:16.9998949Z 967190076fc0: Download complete 2024-06-26T07:05:17.0836152Z 4fb0a83f0685: Verifying Checksum 2024-06-26T07:05:17.0836810Z 4fb0a83f0685: Download complete 2024-06-26T07:05:17.1557310Z 68af2ec4e30a: Download complete 2024-06-26T07:05:17.2480598Z cc91933e985a: Verifying Checksum 2024-06-26T07:05:17.2481225Z cc91933e985a: Download complete 2024-06-26T07:05:17.3683697Z 9f3187916e7d: Download complete 2024-06-26T07:05:17.5006797Z 526083994857: Verifying Checksum 2024-06-26T07:05:17.5007369Z 526083994857: Download complete 2024-06-26T07:05:17.5683540Z 05a6f6dcc1c0: Verifying Checksum 2024-06-26T07:05:17.5684011Z 05a6f6dcc1c0: Download complete 2024-06-26T07:05:18.2728117Z 6bc9dc8315b0: Verifying Checksum 2024-06-26T07:05:18.2728771Z 6bc9dc8315b0: Download complete 2024-06-26T07:05:18.3646549Z 1b0e27c35f07: Download complete 2024-06-26T07:05:40.8577272Z 12ba7431ff38: Verifying Checksum 2024-06-26T07:05:40.8578022Z 12ba7431ff38: Download complete 2024-06-26T07:05:40.9256308Z 789eff6c51aa: Download complete 2024-06-26T07:05:40.9971044Z de0f8a8c45c6: Download complete 2024-06-26T07:05:42.9100298Z 2060028f7488: Verifying Checksum 2024-06-26T07:05:42.9100755Z 2060028f7488: Download complete 2024-06-26T07:05:43.0085531Z e412c7f3b2de: Download complete 2024-06-26T07:05:46.6073295Z 95237e7311a1: Verifying Checksum 2024-06-26T07:05:46.6073864Z 95237e7311a1: Download complete 2024-06-26T07:06:09.1215023Z 84b2c7a89982: Verifying Checksum 2024-06-26T07:06:09.1215556Z 84b2c7a89982: Download complete 2024-06-26T07:06:26.3446121Z 7ed32bc8e469: Pull complete 2024-06-26T07:06:26.5592667Z ec1e7978c1fe: Pull complete 2024-06-26T07:06:26.7582119Z 309b872da805: Pull complete 2024-06-26T07:06:33.9409263Z e82932cb002c: Pull complete 2024-06-26T07:06:34.0995684Z ca9dd73e3e93: Pull complete 2024-06-26T07:06:34.2789658Z 16d76a79d5d6: Pull complete 2024-06-26T07:06:34.4507446Z 4bdf41c9ecbe: Pull complete 2024-06-26T07:06:36.8103734Z f4863801e30c: Pull complete 2024-06-26T07:06:36.9234000Z d37dd9ed8948: Pull complete 2024-06-26T07:06:37.0175893Z cf2f34f225db: Pull complete 2024-06-26T07:06:37.1018782Z df546d7b3e34: Pull complete 2024-06-26T07:07:26.3270449Z 3d914950f8ad: Pull complete 2024-06-26T07:07:26.5313595Z 96d0e2032184: Pull complete 2024-06-26T07:07:26.7530085Z 06e12ce59790: Pull complete 2024-06-26T07:07:26.9558001Z ed712f63cde7: Pull complete 2024-06-26T07:07:27.1613064Z a97a71ef8a4e: Pull complete 2024-06-26T07:07:27.3688331Z 25cd0c482ad6: Pull complete 2024-06-26T07:07:30.8450481Z 9bf94bc45658: Pull complete 2024-06-26T07:07:30.9719756Z de45a14c9b22: Pull complete 2024-06-26T07:07:31.0516100Z e99241e77209: Pull complete 2024-06-26T07:07:31.1442711Z 35e8f4e2a42f: Pull complete 2024-06-26T07:07:31.2060805Z a823ffa4c541: Pull complete 2024-06-26T07:07:31.3044977Z 44ba272eae20: Pull complete 2024-06-26T07:07:39.5280664Z da02d278e524: Pull complete 2024-06-26T07:07:39.6842645Z 73a0ce681c29: Pull complete 2024-06-26T07:07:39.8489981Z 49737bd5ae70: Pull complete 2024-06-26T07:07:40.8527393Z 9cd4c745f8e4: Pull complete 2024-06-26T07:07:41.0247853Z b2950a036534: Pull complete 2024-06-26T07:07:41.1569541Z 25e19d81d9fc: Pull complete 2024-06-26T07:07:41.6461611Z 60156b40fcbe: Pull complete 2024-06-26T07:07:41.7940517Z 7a868239227f: Pull complete 2024-06-26T07:07:42.1892477Z 83911d7c7ecf: Pull complete 2024-06-26T07:07:42.3945705Z b3c5d66c679f: Pull complete 2024-06-26T07:08:21.0033349Z 84b2c7a89982: Pull complete 2024-06-26T07:08:21.1191500Z c1068d03a811: Pull complete 2024-06-26T07:08:21.2172107Z 671b4a301d81: Pull complete 2024-06-26T07:08:23.2169488Z 67760cf5ec46: Pull complete 2024-06-26T07:08:23.2777098Z ddc9740bcea2: Pull complete 2024-06-26T07:08:23.3348850Z fb2e97768560: Pull complete 2024-06-26T07:08:23.4472379Z b599c5a8c7cb: Pull complete 2024-06-26T07:08:23.4883013Z e08e88c13668: Pull complete 2024-06-26T07:08:57.7591904Z 12ba7431ff38: Pull complete 2024-06-26T07:08:57.9729059Z cf7c2cb896cb: Pull complete 2024-06-26T07:08:58.1890341Z 3a778ddfac2a: Pull complete 2024-06-26T07:08:58.6782266Z 236c8f0245f4: Pull complete 2024-06-26T07:08:58.9044105Z 4f4fb700ef54: Pull complete 2024-06-26T07:08:59.1370981Z dda1ce302b9a: Pull complete 2024-06-26T07:08:59.3428349Z e797fe86caef: Pull complete 2024-06-26T07:08:59.7146072Z 967190076fc0: Pull complete 2024-06-26T07:08:59.9383684Z 4fb0a83f0685: Pull complete 2024-06-26T07:09:00.1689979Z 68af2ec4e30a: Pull complete 2024-06-26T07:09:00.3283217Z cc91933e985a: Pull complete 2024-06-26T07:09:00.5231969Z 9f3187916e7d: Pull complete 2024-06-26T07:09:01.6215314Z 526083994857: Pull complete 2024-06-26T07:09:01.8352808Z 05a6f6dcc1c0: Pull complete 2024-06-26T07:09:03.9130934Z 6bc9dc8315b0: Pull complete 2024-06-26T07:09:04.1198011Z 1b0e27c35f07: Pull complete 2024-06-26T07:09:17.9088836Z 95237e7311a1: Pull complete 2024-06-26T07:09:18.1005423Z 789eff6c51aa: Pull complete 2024-06-26T07:09:18.3436722Z de0f8a8c45c6: Pull complete 2024-06-26T07:09:19.1825792Z 2060028f7488: Pull complete 2024-06-26T07:09:19.3202689Z e412c7f3b2de: Pull complete 2024-06-26T07:09:20.1172013Z Digest: sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T07:09:20.1428570Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:09:20.1553026Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:09:20.1604694Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:09:20.1605626Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:09:20.1613861Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:20.1614369Z env: 2024-06-26T07:09:20.1614641Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:20.1614990Z ##[endgroup] 2024-06-26T07:09:20.1767292Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-06-26T07:09:20.1767849Z with: 2024-06-26T07:09:20.1768145Z driver-version: 550.54.15 2024-06-26T07:09:20.1768604Z env: 2024-06-26T07:09:20.1768909Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:20.1769268Z ##[endgroup] 2024-06-26T07:09:20.1839329Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:09:20.1839888Z with: 2024-06-26T07:09:20.1840173Z timeout_minutes: 10 2024-06-26T07:09:20.1840500Z max_attempts: 3 2024-06-26T07:09:20.1871563Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-06-26T07:09:20.1903114Z retry_wait_seconds: 10 2024-06-26T07:09:20.1903490Z polling_interval_seconds: 1 2024-06-26T07:09:20.1903865Z warning_on_retry: true 2024-06-26T07:09:20.1904226Z continue_on_error: false 2024-06-26T07:09:20.1904568Z env: 2024-06-26T07:09:20.1904848Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:20.1905213Z DRIVER_VERSION: 550.54.15 2024-06-26T07:09:20.1905569Z ##[endgroup] 2024-06-26T07:09:20.2424475Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-06-26T07:09:20.2425306Z + pre_install_nvidia_driver_amzn2 2024-06-26T07:09:20.2425818Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-06-26T07:09:20.5444641Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:09:20.5834520Z No Match for argument: nvidia-driver-latest-dkms 2024-06-26T07:09:20.6076195Z No Packages marked for removal 2024-06-26T07:09:20.6199455Z + install_nvidia_driver_common 2024-06-26T07:09:20.6201126Z + echo 'Before installing NVIDIA driver' 2024-06-26T07:09:20.6201652Z + lspci 2024-06-26T07:09:20.6201949Z Before installing NVIDIA driver 2024-06-26T07:09:20.6279067Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:09:20.6279977Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:09:20.6281015Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:09:20.6281930Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:09:20.6282693Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:09:20.6283448Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:09:20.6284169Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:09:20.6284957Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:09:20.6285534Z + lsmod 2024-06-26T07:09:20.6292430Z Module Size Used by 2024-06-26T07:09:20.6292853Z ib_core 266240 0 2024-06-26T07:09:20.6293243Z nvidia_modeset 1339392 0 2024-06-26T07:09:20.6293729Z veth 16384 0 2024-06-26T07:09:20.6294185Z nvidia_uvm 4599808 0 2024-06-26T07:09:20.6294655Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:09:20.6295398Z drm 421888 1 nvidia 2024-06-26T07:09:20.6295905Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:09:20.6296371Z backlight 16384 1 nvidia_modeset 2024-06-26T07:09:20.6296812Z xt_conntrack 16384 1 2024-06-26T07:09:20.6297198Z ipt_MASQUERADE 16384 1 2024-06-26T07:09:20.6297947Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:09:20.6298430Z nf_conntrack_netlink 49152 0 2024-06-26T07:09:20.6298896Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:09:20.6299361Z xfrm_user 45056 1 2024-06-26T07:09:20.6299758Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:09:20.6300177Z iptable_nat 16384 1 2024-06-26T07:09:20.6300564Z nf_conntrack_ipv4 16384 3 2024-06-26T07:09:20.6301017Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:09:20.6301633Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:09:20.6302171Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-06-26T07:09:20.6303216Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-06-26T07:09:20.6304381Z xt_addrtype 16384 2 2024-06-26T07:09:20.6304927Z iptable_filter 16384 1 2024-06-26T07:09:20.6305455Z br_netfilter 24576 0 2024-06-26T07:09:20.6306072Z bridge 172032 1 br_netfilter 2024-06-26T07:09:20.6306673Z stp 16384 1 bridge 2024-06-26T07:09:20.6307292Z llc 16384 2 bridge,stp 2024-06-26T07:09:20.6308058Z overlay 86016 0 2024-06-26T07:09:20.6308554Z sunrpc 393216 1 2024-06-26T07:09:20.6309050Z dm_mirror 28672 0 2024-06-26T07:09:20.6309564Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:09:20.6310218Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:09:20.6310898Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:09:20.6311445Z dax 69632 1 dm_mod 2024-06-26T07:09:20.6311853Z crc32_pclmul 16384 0 2024-06-26T07:09:20.6312237Z ghash_clmulni_intel 16384 0 2024-06-26T07:09:20.6312623Z pcbc 16384 0 2024-06-26T07:09:20.6312987Z aesni_intel 188416 0 2024-06-26T07:09:20.6313386Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:09:20.6313843Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:09:20.6314304Z glue_helper 16384 1 aesni_intel 2024-06-26T07:09:20.6314969Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:09:20.6315525Z evdev 20480 3 2024-06-26T07:09:20.6315946Z mousedev 24576 0 2024-06-26T07:09:20.6316327Z psmouse 32768 0 2024-06-26T07:09:20.6316691Z button 16384 0 2024-06-26T07:09:20.6317067Z ena 139264 0 2024-06-26T07:09:20.6317438Z ptp 20480 1 ena 2024-06-26T07:09:20.6317841Z pps_core 20480 1 ptp 2024-06-26T07:09:20.6318238Z crc32c_intel 24576 0 2024-06-26T07:09:20.6318607Z autofs4 49152 2 2024-06-26T07:09:20.6318981Z + modinfo nvidia 2024-06-26T07:09:20.6319648Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:09:20.6320345Z alias: char-major-195-* 2024-06-26T07:09:20.6320726Z version: 550.54.15 2024-06-26T07:09:20.6321071Z supported: external 2024-06-26T07:09:20.6321425Z license: NVIDIA 2024-06-26T07:09:20.6321839Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:09:20.6322330Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:09:20.6322798Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:09:20.6323263Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:09:20.6323768Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:09:20.6324254Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:09:20.6324852Z depends: i2c-core,drm 2024-06-26T07:09:20.6325218Z retpoline: Y 2024-06-26T07:09:20.6325527Z name: nvidia 2024-06-26T07:09:20.6326087Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:09:20.6326732Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:09:20.6327376Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:09:20.6327970Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:09:20.6328412Z parm: NVreg_RmLogonRC:int 2024-06-26T07:09:20.6328979Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:09:20.6329446Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:09:20.6329893Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:09:20.6330327Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:09:20.6330855Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:09:20.6331406Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:09:20.6331940Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:09:20.6332365Z parm: NVreg_EnableMSI:int 2024-06-26T07:09:20.6332770Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:09:20.6333225Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:09:20.6333751Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:09:20.6334329Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:09:20.6334878Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:09:20.6335479Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:09:20.6336171Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:09:20.6336781Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:09:20.6337363Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:09:20.6338093Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:09:20.6338637Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:09:20.6339178Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:09:20.6339665Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:09:20.6340124Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:09:20.6340619Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:09:20.6341093Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:09:20.6341538Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:09:20.6342054Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:09:20.6342590Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:09:20.6343073Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:09:20.6343548Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:09:20.6344045Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:09:20.6344527Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:09:20.6345016Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:09:20.6345497Z parm: NVreg_RmMsg:charp 2024-06-26T07:09:20.6345907Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:09:20.6346371Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:09:20.6346837Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:09:20.6347302Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:09:20.6347779Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:09:20.6348254Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:09:20.6348695Z parm: rm_firmware_active:charp 2024-06-26T07:09:20.6349103Z + HAS_NVIDIA_DRIVER=0 2024-06-26T07:09:20.6349482Z ++ command -v nvidia-smi 2024-06-26T07:09:20.6349875Z + '[' -x /usr/bin/nvidia-smi ']' 2024-06-26T07:09:20.6350232Z + set +e 2024-06-26T07:09:20.6350723Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-06-26T07:09:22.6348715Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-06-26T07:09:22.6349392Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:09:22.6349971Z + '[' 0 -ne 0 ']' 2024-06-26T07:09:22.6350758Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-06-26T07:09:22.6351149Z + HAS_NVIDIA_DRIVER=1 2024-06-26T07:09:22.6351876Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-06-26T07:09:22.6352574Z + set -e 2024-06-26T07:09:22.6352865Z + '[' 1 -eq 0 ']' 2024-06-26T07:09:22.6353436Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-06-26T07:09:22.6354522Z + post_install_nvidia_driver_common 2024-06-26T07:09:22.6355080Z + sudo modprobe nvidia 2024-06-26T07:09:22.6449567Z + echo 'After installing NVIDIA driver' 2024-06-26T07:09:22.6450076Z + lspci 2024-06-26T07:09:22.6450374Z After installing NVIDIA driver 2024-06-26T07:09:22.6526878Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:09:22.6527782Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:09:22.6528812Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:09:22.6529620Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:09:22.6530372Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:09:22.6531116Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:09:22.6531851Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:09:22.6532662Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:09:22.6533251Z + lsmod 2024-06-26T07:09:22.6541164Z Module Size Used by 2024-06-26T07:09:22.6541617Z ib_core 266240 0 2024-06-26T07:09:22.6542314Z nvidia_modeset 1339392 0 2024-06-26T07:09:22.6542852Z veth 16384 0 2024-06-26T07:09:22.6543349Z nvidia_uvm 4599808 0 2024-06-26T07:09:22.6543965Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:09:22.6544590Z drm 421888 1 nvidia 2024-06-26T07:09:22.6545026Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:09:22.6545491Z backlight 16384 1 nvidia_modeset 2024-06-26T07:09:22.6545955Z xt_conntrack 16384 1 2024-06-26T07:09:22.6546345Z ipt_MASQUERADE 16384 1 2024-06-26T07:09:22.6546789Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:09:22.6547257Z nf_conntrack_netlink 49152 0 2024-06-26T07:09:22.6547710Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:09:22.6548171Z xfrm_user 45056 1 2024-06-26T07:09:22.6548569Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:09:22.6548986Z iptable_nat 16384 1 2024-06-26T07:09:22.6549376Z nf_conntrack_ipv4 16384 3 2024-06-26T07:09:22.6549904Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:09:22.6550549Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:09:22.6551251Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-06-26T07:09:22.6552541Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-06-26T07:09:22.6553385Z xt_addrtype 16384 2 2024-06-26T07:09:22.6553767Z iptable_filter 16384 1 2024-06-26T07:09:22.6554142Z br_netfilter 24576 0 2024-06-26T07:09:22.6554559Z bridge 172032 1 br_netfilter 2024-06-26T07:09:22.6555004Z stp 16384 1 bridge 2024-06-26T07:09:22.6555427Z llc 16384 2 bridge,stp 2024-06-26T07:09:22.6555840Z overlay 86016 0 2024-06-26T07:09:22.6556215Z sunrpc 393216 1 2024-06-26T07:09:22.6556597Z dm_mirror 28672 0 2024-06-26T07:09:22.6556995Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:09:22.6557595Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:09:22.6558110Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:09:22.6558615Z dax 69632 1 dm_mod 2024-06-26T07:09:22.6559147Z crc32_pclmul 16384 0 2024-06-26T07:09:22.6559541Z ghash_clmulni_intel 16384 0 2024-06-26T07:09:22.6559920Z pcbc 16384 0 2024-06-26T07:09:22.6560289Z aesni_intel 188416 0 2024-06-26T07:09:22.6560678Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:09:22.6561132Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:09:22.6561585Z glue_helper 16384 1 aesni_intel 2024-06-26T07:09:22.6562151Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:09:22.6562681Z evdev 20480 3 2024-06-26T07:09:22.6563047Z mousedev 24576 0 2024-06-26T07:09:22.6563424Z psmouse 32768 0 2024-06-26T07:09:22.6563786Z button 16384 0 2024-06-26T07:09:22.6564138Z ena 139264 0 2024-06-26T07:09:22.6564508Z ptp 20480 1 ena 2024-06-26T07:09:22.6564904Z pps_core 20480 1 ptp 2024-06-26T07:09:22.6565298Z crc32c_intel 24576 0 2024-06-26T07:09:22.6565662Z autofs4 49152 2 2024-06-26T07:09:22.6566019Z + modinfo nvidia 2024-06-26T07:09:22.6566705Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:09:22.6567397Z alias: char-major-195-* 2024-06-26T07:09:22.6567770Z version: 550.54.15 2024-06-26T07:09:22.6568118Z supported: external 2024-06-26T07:09:22.6568455Z license: NVIDIA 2024-06-26T07:09:22.6569027Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:09:22.6569510Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:09:22.6570037Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:09:22.6570502Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:09:22.6570992Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:09:22.6571483Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:09:22.6571966Z depends: i2c-core,drm 2024-06-26T07:09:22.6572328Z retpoline: Y 2024-06-26T07:09:22.6572634Z name: nvidia 2024-06-26T07:09:22.6573198Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:09:22.6573860Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:09:22.6574507Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:09:22.6575096Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:09:22.6575545Z parm: NVreg_RmLogonRC:int 2024-06-26T07:09:22.6575978Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:09:22.6576434Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:09:22.6576871Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:09:22.6577312Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:09:22.6578114Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:09:22.6578679Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:09:22.6579165Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:09:22.6579606Z parm: NVreg_EnableMSI:int 2024-06-26T07:09:22.6580027Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:09:22.6580483Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:09:22.6581021Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:09:22.6581605Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:09:22.6582219Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:09:22.6582823Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:09:22.6583421Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:09:22.6597744Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:09:22.6598394Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:09:22.6598895Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:09:22.6599432Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:09:22.6599982Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:09:22.6600604Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:09:22.6601080Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:09:22.6601559Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:09:22.6602034Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:09:22.6602488Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:09:22.6602991Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:09:22.6603530Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:09:22.6604014Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:09:22.6604511Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:09:22.6605012Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:09:22.6605503Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:09:22.6606007Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:09:22.6606492Z parm: NVreg_RmMsg:charp 2024-06-26T07:09:22.6606903Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:09:22.6607388Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:09:22.6607870Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:09:22.6608329Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:09:22.6608951Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:09:22.6609433Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:09:22.6609892Z parm: rm_firmware_active:charp 2024-06-26T07:09:22.6610294Z + set +e 2024-06-26T07:09:22.6610609Z + nvidia-smi 2024-06-26T07:09:24.2262395Z Wed Jun 26 07:09:24 2024 2024-06-26T07:09:24.2263269Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:09:24.2264399Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-06-26T07:09:24.2265176Z |-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:09:24.2265986Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-06-26T07:09:24.2266878Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-06-26T07:09:24.2267568Z | | | MIG M. | 2024-06-26T07:09:24.2268122Z |=========================================+========================+======================| 2024-06-26T07:09:24.2398308Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-06-26T07:09:24.2399041Z | 0% 34C P0 57W / 300W | 0MiB / 23028MiB | 5% Default | 2024-06-26T07:09:24.2399880Z | | | N/A | 2024-06-26T07:09:24.2400721Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:09:24.2401317Z 2024-06-26T07:09:24.2402073Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:09:24.2402806Z | Processes: | 2024-06-26T07:09:24.2403514Z | GPU GI CI PID Type Process name GPU Memory | 2024-06-26T07:09:24.2404219Z | ID ID Usage | 2024-06-26T07:09:24.2404781Z |=========================================================================================| 2024-06-26T07:09:24.2405430Z | No running processes found | 2024-06-26T07:09:24.2406207Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:09:24.8780872Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-06-26T07:09:26.4366726Z NVIDIA A10G 2024-06-26T07:09:26.8810519Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:09:26.8811487Z + '[' 0 -eq 0 ']' 2024-06-26T07:09:26.8811910Z + echo 'INFO: Ignoring allowed status 0' 2024-06-26T07:09:26.8812409Z + set -e 2024-06-26T07:09:26.8812819Z INFO: Ignoring allowed status 0 2024-06-26T07:09:26.8814230Z == Installing nvidia container toolkit for amzn2 == 2024-06-26T07:09:26.8817166Z + sudo yum install -y yum-utils 2024-06-26T07:09:27.1830940Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:09:28.6583581Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-06-26T07:09:28.6584279Z Nothing to do 2024-06-26T07:09:28.8092543Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]] 2024-06-26T07:09:28.8093777Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:09:28.8094920Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:09:29.1324611Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:09:29.1590950Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:09:29.1592189Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:09:29.1593145Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:09:29.1709667Z + sudo yum install -y nvidia-docker2 2024-06-26T07:09:29.4707547Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:09:30.9287266Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-06-26T07:09:30.9287929Z Nothing to do 2024-06-26T07:09:31.0759638Z + sudo systemctl restart docker 2024-06-26T07:10:15.2931567Z Command completed after 1 attempt(s). 2024-06-26T07:10:15.2989884Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:10:15.2990664Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:10:15.2991367Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-06-26T07:10:15.2992039Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:10:15.2999783Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:15.3000285Z env: 2024-06-26T07:10:15.3000571Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:15.3001030Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:15.3001521Z ##[endgroup] 2024-06-26T07:10:15.5201064Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:10:15.5352308Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-06-26T07:10:15.5450971Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-06-26T07:10:15.6365765Z Prepare all required actions 2024-06-26T07:10:15.6366261Z Getting action download info 2024-06-26T07:10:15.7521455Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-06-26T07:10:15.9078823Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-06-26T07:10:17.2342895Z ##[group]Run ./.github/actions/download-build-artifacts 2024-06-26T07:10:17.2343375Z with: 2024-06-26T07:10:17.2343716Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:10:17.2344184Z s3-bucket: gha-artifacts 2024-06-26T07:10:17.2344536Z env: 2024-06-26T07:10:17.2344814Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:17.2345263Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:17.2345764Z ##[endgroup] 2024-06-26T07:10:17.2376765Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:10:17.2377205Z with: 2024-06-26T07:10:17.2377717Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:10:17.2378186Z s3-bucket: gha-artifacts 2024-06-26T07:10:17.2378535Z region: us-east-1 2024-06-26T07:10:17.2378836Z env: 2024-06-26T07:10:17.2379118Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:17.2379568Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:17.2380060Z ##[endgroup] 2024-06-26T07:10:17.6532173Z (node:12760) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:10:17.6532891Z 2024-06-26T07:10:17.6533181Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:10:17.6534045Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:10:17.6535268Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:10:17.7216675Z Found 1 objects with prefix pytorch/pytorch/9673646808/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-06-26T07:10:17.7218015Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:10:25.2102811Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:10:25.2109508Z Artifact download has finished successfully 2024-06-26T07:10:25.2273831Z ##[group]Run unzip -o artifacts.zip 2024-06-26T07:10:25.2274280Z unzip -o artifacts.zip 2024-06-26T07:10:25.2281999Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:25.2282503Z env: 2024-06-26T07:10:25.2282783Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:25.2283246Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:25.2283735Z ##[endgroup] 2024-06-26T07:10:25.2311380Z Archive: artifacts.zip 2024-06-26T07:10:25.2312987Z creating: dist/ 2024-06-26T07:10:27.2060113Z inflating: dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:10:27.2061021Z creating: build/custom_test_artifacts/ 2024-06-26T07:10:27.2061607Z creating: build/custom_test_artifacts/custom-op-build/ 2024-06-26T07:10:27.2062334Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-06-26T07:10:27.2063165Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:10:27.2067032Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:10:27.2067950Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-06-26T07:10:27.2068888Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:10:27.2069864Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:10:27.2070813Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:10:27.2071896Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:10:27.2073121Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:10:27.2074111Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:10:27.2075091Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:10:27.2076220Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:10:27.2077357Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:10:27.2078484Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:10:27.2079604Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:10:27.2080761Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:10:27.2081917Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:10:27.2082932Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:10:27.2083924Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:10:27.2118866Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:10:27.2156710Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:10:27.2158148Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:10:27.2201106Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:10:27.2203895Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:10:27.2207036Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:10:27.2210076Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:10:27.2211565Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:10:27.2212931Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:10:27.2214306Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:10:27.2215797Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:10:27.2217136Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:10:27.2218668Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:10:27.2219895Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:10:27.2221094Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:10:27.2222310Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:10:27.2223495Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:10:27.2224695Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:10:27.2268368Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:10:27.2327127Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:10:27.2329576Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:10:27.2330642Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:10:27.2331493Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:10:27.2332394Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-06-26T07:10:27.2333309Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-06-26T07:10:27.2334322Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-06-26T07:10:27.2335475Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-06-26T07:10:27.2336577Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-06-26T07:10:27.2337799Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-06-26T07:10:27.2338883Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-06-26T07:10:27.2339971Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-06-26T07:10:27.2341072Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-06-26T07:10:27.2342144Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-06-26T07:10:27.2343199Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-06-26T07:10:27.2351734Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-06-26T07:10:27.2481741Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-06-26T07:10:27.2483710Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-06-26T07:10:27.2485858Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-06-26T07:10:27.2488234Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-06-26T07:10:27.2490316Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-06-26T07:10:27.2491396Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-06-26T07:10:27.2492522Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-06-26T07:10:27.2493755Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-06-26T07:10:27.2494892Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-06-26T07:10:27.2496022Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-06-26T07:10:27.2497130Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-06-26T07:10:27.2505122Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-06-26T07:10:27.2584222Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-06-26T07:10:27.2585429Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:10:27.2586547Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:10:27.2587727Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-06-26T07:10:27.2588659Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-06-26T07:10:27.2589561Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-06-26T07:10:27.2590472Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-06-26T07:10:27.2591328Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-06-26T07:10:27.2592117Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-06-26T07:10:27.2592925Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-06-26T07:10:27.2698274Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-06-26T07:10:27.2758685Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-06-26T07:10:27.2759430Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-06-26T07:10:27.2760136Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-06-26T07:10:27.2760950Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:10:27.2766199Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:10:27.2767137Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-06-26T07:10:27.2768074Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:10:27.2769132Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:10:27.2770093Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:10:27.2786200Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:10:27.2787330Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:10:27.2788321Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:10:27.2789306Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:10:27.2790444Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:10:27.2791607Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:10:27.2792725Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:10:27.2793841Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:10:27.2794995Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:10:27.2796302Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:10:27.2797321Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:10:27.2798302Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:10:27.2816923Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:10:27.2854459Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:10:27.2855900Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:10:27.2898528Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:10:27.2899943Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:10:27.2901466Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:10:27.2902930Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:10:27.2904327Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:10:27.2905714Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:10:27.2907088Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:10:27.2908444Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:10:27.2909810Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:10:27.2911083Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:10:27.2912320Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:10:27.2913533Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:10:27.2914761Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:10:27.2915945Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:10:27.2917160Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:10:27.2964420Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:10:27.3022188Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:10:27.3023363Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:10:27.3024337Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:10:27.3025190Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:10:27.3026080Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-06-26T07:10:27.3027041Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-06-26T07:10:27.3028089Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-06-26T07:10:27.3029389Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-06-26T07:10:27.3030521Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-06-26T07:10:27.3031586Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-06-26T07:10:27.3032673Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-06-26T07:10:27.3033772Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-06-26T07:10:27.3034875Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-06-26T07:10:27.3035973Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-06-26T07:10:27.3037055Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-06-26T07:10:27.3047574Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-06-26T07:10:27.3109034Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-06-26T07:10:27.3110261Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:10:27.3111319Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:10:27.3112266Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-06-26T07:10:27.3113139Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-06-26T07:10:27.3114014Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-06-26T07:10:27.3114899Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-06-26T07:10:27.3115726Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-06-26T07:10:27.3116495Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-06-26T07:10:27.3117300Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-06-26T07:10:27.3162442Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-06-26T07:10:27.3163201Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-06-26T07:10:27.3163971Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-06-26T07:10:27.3164844Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:10:27.3170060Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:10:27.3171053Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-06-26T07:10:27.3172047Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:10:27.3173090Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:10:27.3174121Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:10:27.3175281Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:10:27.3176444Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:10:27.3177635Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:10:27.3178678Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:10:27.3179945Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:10:27.3181155Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:10:27.3182466Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:10:27.3183675Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:10:27.3184877Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:10:27.3186084Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:10:27.3187175Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:10:27.3188229Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:10:27.3221355Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:10:27.3258992Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:10:27.3260626Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:10:27.3303052Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:10:27.3304523Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:10:27.3305992Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:10:27.3307516Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:10:27.3308969Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:10:27.3310481Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:10:27.3311924Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:10:27.3313359Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:10:27.3314747Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:10:27.3316067Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:10:27.3317359Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:10:27.3318634Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:10:27.3319974Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:10:27.3321215Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:10:27.3322479Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:10:27.3368571Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:10:27.3426641Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:10:27.3427895Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:10:27.3428941Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:10:27.3430046Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:10:27.3431005Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-06-26T07:10:27.3432014Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-06-26T07:10:27.3433155Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-06-26T07:10:27.3434404Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-06-26T07:10:27.3435596Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-06-26T07:10:27.3436735Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-06-26T07:10:27.3437901Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-06-26T07:10:27.3439163Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-06-26T07:10:27.3440388Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-06-26T07:10:27.3441568Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-06-26T07:10:27.3442738Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-06-26T07:10:27.3443985Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-06-26T07:10:27.3555832Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-06-26T07:10:27.3557005Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-06-26T07:10:27.3558187Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-06-26T07:10:27.3559490Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-06-26T07:10:27.3560744Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-06-26T07:10:27.3561932Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-06-26T07:10:27.3563163Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-06-26T07:10:27.3564385Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-06-26T07:10:27.3565628Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-06-26T07:10:27.3566865Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-06-26T07:10:27.3568095Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-06-26T07:10:27.3579750Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-06-26T07:10:27.3632342Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-06-26T07:10:27.3633632Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:10:27.3634764Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:10:27.3635785Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-06-26T07:10:27.3636725Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-06-26T07:10:27.3637806Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-06-26T07:10:27.3638759Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-06-26T07:10:27.3639660Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-06-26T07:10:27.3640485Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-06-26T07:10:27.3641335Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-06-26T07:10:27.3737456Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-06-26T07:10:27.3779234Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-06-26T07:10:27.3780058Z creating: build/lib/ 2024-06-26T07:10:27.3863737Z inflating: build/lib/libprotobuf-lite.a 2024-06-26T07:10:27.4301660Z inflating: build/lib/libprotobuf.a 2024-06-26T07:10:27.4310868Z inflating: build/lib/libpthreadpool.a 2024-06-26T07:10:27.4318569Z inflating: build/lib/libcpuinfo.a 2024-06-26T07:10:27.4326169Z inflating: build/lib/libcpuinfo_internals.a 2024-06-26T07:10:27.4326657Z inflating: build/lib/libclog.a 2024-06-26T07:10:27.4344326Z inflating: build/lib/libnnpack.a 2024-06-26T07:10:27.4346310Z inflating: build/lib/libnnpack_reference_layers.a 2024-06-26T07:10:27.4408419Z inflating: build/lib/libgtest.a 2024-06-26T07:10:27.4480447Z inflating: build/lib/libbenchmark.a 2024-06-26T07:10:27.4541807Z inflating: build/lib/libasmjit.a 2024-06-26T07:10:27.4548869Z inflating: build/lib/libittnotify.a 2024-06-26T07:10:27.4575323Z inflating: build/lib/libtensorpipe_uv.a 2024-06-26T07:10:27.4697284Z inflating: build/lib/libgloo.a 2024-06-26T07:10:27.4718339Z inflating: build/lib/libfmt.a 2024-06-26T07:10:27.4807103Z inflating: build/lib/libc10.so 2024-06-26T07:10:27.4808372Z inflating: build/lib/libcaffe2_nvrtc.so 2024-06-26T07:10:27.4809390Z inflating: build/lib/libfoxi_loader.a 2024-06-26T07:10:27.4809924Z inflating: build/lib/libtorch_global_deps.so 2024-06-26T07:10:27.4828609Z inflating: build/lib/libpytorch_qnnpack.a 2024-06-26T07:10:27.4845697Z inflating: build/lib/libgmock.a 2024-06-26T07:10:27.4846146Z inflating: build/lib/libgtest_main.a 2024-06-26T07:10:27.4847107Z inflating: build/lib/libbenchmark_main.a 2024-06-26T07:10:27.5330675Z inflating: build/lib/libprotoc.a 2024-06-26T07:10:27.5700071Z inflating: build/lib/libgloo_cuda.a 2024-06-26T07:10:28.5425054Z inflating: build/lib/libdnnl.a 2024-06-26T07:10:28.5974433Z inflating: build/lib/libtensorpipe.a 2024-06-26T07:10:28.6026763Z inflating: build/lib/libc10_cuda.so 2024-06-26T07:10:28.6027215Z inflating: build/lib/libgmock_main.a 2024-06-26T07:10:28.7228895Z inflating: build/lib/libfbgemm.a 2024-06-26T07:10:28.7473537Z inflating: build/lib/libtensorpipe_cuda.a 2024-06-26T07:10:28.7971342Z inflating: build/lib/libkineto.a 2024-06-26T07:10:28.8148901Z inflating: build/lib/libXNNPACK.a 2024-06-26T07:10:28.8189378Z inflating: build/lib/libonnx_proto.a 2024-06-26T07:10:28.8854030Z inflating: build/lib/libonnx.a 2024-06-26T07:10:31.2131689Z inflating: build/lib/libtorch_cpu.so 2024-06-26T07:10:31.2135177Z inflating: build/lib/libunbox_lib.a 2024-06-26T07:10:31.2139167Z inflating: build/lib/libshm.so 2024-06-26T07:10:33.2004156Z inflating: build/lib/libtorch_cuda.so 2024-06-26T07:10:33.2004682Z inflating: build/lib/libtorch.so 2024-06-26T07:10:33.2007001Z inflating: build/lib/libc10d_cuda_test.so 2024-06-26T07:10:33.9956796Z inflating: build/lib/libtorch_cuda_linalg.so 2024-06-26T07:10:34.1847718Z inflating: build/lib/libtorch_python.so 2024-06-26T07:10:34.1913771Z inflating: build/lib/libtorchbind_test.so 2024-06-26T07:10:34.1933150Z inflating: build/lib/libjitbackend_test.so 2024-06-26T07:10:34.1957661Z inflating: build/lib/libbackend_with_compiler.so 2024-06-26T07:10:34.1990900Z inflating: build/lib/libnnapi_backend.so 2024-06-26T07:10:34.1991371Z creating: build/bin/ 2024-06-26T07:10:34.2040155Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-06-26T07:10:34.2089436Z inflating: build/bin/c10_DeviceGuard_test 2024-06-26T07:10:34.2138200Z inflating: build/bin/c10_Device_test 2024-06-26T07:10:34.2195159Z inflating: build/bin/c10_DispatchKeySet_test 2024-06-26T07:10:34.2246997Z inflating: build/bin/c10_Scalar_test 2024-06-26T07:10:34.2294397Z inflating: build/bin/c10_StreamGuard_test 2024-06-26T07:10:34.2343927Z inflating: build/bin/c10_SymInt_test 2024-06-26T07:10:34.2396583Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-06-26T07:10:34.2450925Z inflating: build/bin/c10_InlineStreamGuard_test 2024-06-26T07:10:34.2504996Z inflating: build/bin/c10_SizesAndStrides_test 2024-06-26T07:10:34.2573638Z inflating: build/bin/c10_cow_test 2024-06-26T07:10:34.2625134Z inflating: build/bin/c10_Bitset_test 2024-06-26T07:10:34.2673083Z inflating: build/bin/c10_ConstexprCrc_test 2024-06-26T07:10:34.2720746Z inflating: build/bin/c10_DeadlockDetection_test 2024-06-26T07:10:34.2770367Z inflating: build/bin/c10_Half_test 2024-06-26T07:10:34.2824421Z inflating: build/bin/c10_LeftRight_test 2024-06-26T07:10:34.2877641Z inflating: build/bin/c10_Metaprogramming_test 2024-06-26T07:10:34.2925879Z inflating: build/bin/c10_Synchronized_test 2024-06-26T07:10:34.2979948Z inflating: build/bin/c10_ThreadLocal_test 2024-06-26T07:10:34.3029605Z inflating: build/bin/c10_TypeIndex_test 2024-06-26T07:10:34.3078965Z inflating: build/bin/c10_TypeList_test 2024-06-26T07:10:34.3126199Z inflating: build/bin/c10_TypeTraits_test 2024-06-26T07:10:34.3176391Z inflating: build/bin/c10_accumulate_test 2024-06-26T07:10:34.3230001Z inflating: build/bin/c10_bfloat16_test 2024-06-26T07:10:34.3278617Z inflating: build/bin/c10_bit_cast_test 2024-06-26T07:10:34.3333138Z inflating: build/bin/c10_complex_math_test 2024-06-26T07:10:34.3387097Z inflating: build/bin/c10_complex_test 2024-06-26T07:10:34.3438555Z inflating: build/bin/c10_exception_test 2024-06-26T07:10:34.3487006Z inflating: build/bin/c10_flags_test 2024-06-26T07:10:34.3535196Z inflating: build/bin/c10_generic_math_test 2024-06-26T07:10:34.3693632Z inflating: build/bin/c10_intrusive_ptr_test 2024-06-26T07:10:34.3742816Z inflating: build/bin/c10_irange_test 2024-06-26T07:10:34.3794518Z inflating: build/bin/c10_lazy_test 2024-06-26T07:10:34.3849982Z inflating: build/bin/c10_logging_test 2024-06-26T07:10:34.3922285Z inflating: build/bin/c10_optional_test 2024-06-26T07:10:34.3982694Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-06-26T07:10:34.4034771Z inflating: build/bin/c10_registry_test 2024-06-26T07:10:34.4180456Z inflating: build/bin/c10_small_vector_test 2024-06-26T07:10:34.4230194Z inflating: build/bin/c10_ssize_test 2024-06-26T07:10:34.4280930Z inflating: build/bin/c10_string_util_test 2024-06-26T07:10:34.4337593Z inflating: build/bin/c10_string_view_test 2024-06-26T07:10:34.4386212Z inflating: build/bin/c10_tempfile_test 2024-06-26T07:10:34.4433786Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-06-26T07:10:34.4487863Z inflating: build/bin/c10_typeid_test 2024-06-26T07:10:34.4920320Z inflating: build/bin/protoc-3.13.0.0 2024-06-26T07:10:34.5352816Z inflating: build/bin/protoc 2024-06-26T07:10:34.5404028Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-06-26T07:10:34.5455428Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-06-26T07:10:34.5507375Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-06-26T07:10:34.5558294Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-06-26T07:10:34.5610103Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-06-26T07:10:34.5661267Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-06-26T07:10:34.5708238Z inflating: build/bin/c10_cuda_CUDATest 2024-06-26T07:10:34.5759435Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-06-26T07:10:34.6077802Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-06-26T07:10:34.6410813Z inflating: build/bin/vec_test_all_types_AVX512 2024-06-26T07:10:34.6757081Z inflating: build/bin/vec_test_all_types_AVX2 2024-06-26T07:10:34.6808103Z inflating: build/bin/FileStoreTest 2024-06-26T07:10:34.6862485Z inflating: build/bin/TCPStoreTest 2024-06-26T07:10:34.6914354Z inflating: build/bin/HashStoreTest 2024-06-26T07:10:34.6927487Z inflating: build/bin/ProcessGroupMPITest 2024-06-26T07:10:34.6980255Z inflating: build/bin/test_edge_op_registration 2024-06-26T07:10:34.6984308Z inflating: build/bin/torch_shm_manager 2024-06-26T07:10:34.6987142Z inflating: build/bin/example_allreduce 2024-06-26T07:10:34.7041152Z inflating: build/bin/test_dist_autograd 2024-06-26T07:10:34.7108224Z inflating: build/bin/test_cpp_rpc 2024-06-26T07:10:34.7110522Z inflating: build/bin/parallel_benchmark 2024-06-26T07:10:34.7175870Z inflating: build/bin/test_mobile_nnc 2024-06-26T07:10:34.7184718Z inflating: build/bin/aot_model_compiler_test 2024-06-26T07:10:34.7517712Z inflating: build/bin/test_lazy 2024-06-26T07:10:34.8660616Z inflating: build/bin/test_api 2024-06-26T07:10:34.8732213Z inflating: build/bin/Dict_test 2024-06-26T07:10:34.8782665Z inflating: build/bin/Dimname_test 2024-06-26T07:10:34.8845258Z inflating: build/bin/MaybeOwned_test 2024-06-26T07:10:34.8901142Z inflating: build/bin/NamedTensor_test 2024-06-26T07:10:34.8959139Z inflating: build/bin/apply_utils_test 2024-06-26T07:10:34.9015996Z inflating: build/bin/atest 2024-06-26T07:10:34.9077837Z inflating: build/bin/basic 2024-06-26T07:10:34.9131296Z inflating: build/bin/broadcast_test 2024-06-26T07:10:34.9180285Z inflating: build/bin/cpu_allocator_test 2024-06-26T07:10:34.9236854Z inflating: build/bin/cpu_generator_test 2024-06-26T07:10:34.9288447Z inflating: build/bin/cpu_profiling_allocator_test 2024-06-26T07:10:34.9379578Z inflating: build/bin/cpu_rng_test 2024-06-26T07:10:34.9427831Z inflating: build/bin/dispatch_key_set_test 2024-06-26T07:10:34.9476855Z inflating: build/bin/dlconvertor_test 2024-06-26T07:10:34.9533378Z inflating: build/bin/extension_backend_test 2024-06-26T07:10:34.9587176Z inflating: build/bin/half_test 2024-06-26T07:10:34.9680719Z inflating: build/bin/ivalue_test 2024-06-26T07:10:34.9729140Z inflating: build/bin/lazy_tensor_test 2024-06-26T07:10:34.9781629Z inflating: build/bin/math_kernel_test 2024-06-26T07:10:34.9834403Z inflating: build/bin/memory_format_test 2024-06-26T07:10:34.9885933Z inflating: build/bin/memory_overlapping_test 2024-06-26T07:10:34.9937880Z inflating: build/bin/mobile_memory_cleanup 2024-06-26T07:10:34.9992215Z inflating: build/bin/native_test 2024-06-26T07:10:35.0041656Z inflating: build/bin/operator_name_test 2024-06-26T07:10:35.0092047Z inflating: build/bin/operators_test 2024-06-26T07:10:35.0141887Z inflating: build/bin/packedtensoraccessor_test 2024-06-26T07:10:35.0207147Z inflating: build/bin/pow_test 2024-06-26T07:10:35.0262896Z inflating: build/bin/quantized_test 2024-06-26T07:10:35.0311513Z inflating: build/bin/reduce_ops_test 2024-06-26T07:10:35.0361196Z inflating: build/bin/reportMemoryUsage_test 2024-06-26T07:10:35.0416250Z inflating: build/bin/scalar_tensor_test 2024-06-26T07:10:35.0473774Z inflating: build/bin/scalar_test 2024-06-26T07:10:35.0524006Z inflating: build/bin/StorageUtils_test 2024-06-26T07:10:35.0574797Z inflating: build/bin/stride_properties_test 2024-06-26T07:10:35.0651463Z inflating: build/bin/tensor_iterator_test 2024-06-26T07:10:35.0703906Z inflating: build/bin/test_parallel 2024-06-26T07:10:35.0706815Z inflating: build/bin/thread_init_test 2024-06-26T07:10:35.0761030Z inflating: build/bin/type_ptr_test 2024-06-26T07:10:35.0819450Z inflating: build/bin/type_test 2024-06-26T07:10:35.0870206Z inflating: build/bin/undefined_tensor_test 2024-06-26T07:10:35.0871752Z inflating: build/bin/verify_api_visibility 2024-06-26T07:10:35.0939719Z inflating: build/bin/legacy_vmap_test 2024-06-26T07:10:35.0990065Z inflating: build/bin/weakref_test 2024-06-26T07:10:35.1040115Z inflating: build/bin/wrapdim_test 2024-06-26T07:10:35.1090493Z inflating: build/bin/xla_tensor_test 2024-06-26T07:10:35.1148815Z inflating: build/bin/IListRef_test 2024-06-26T07:10:35.1251874Z inflating: build/bin/List_test 2024-06-26T07:10:35.1317123Z inflating: build/bin/KernelFunction_test 2024-06-26T07:10:35.1435511Z inflating: build/bin/kernel_function_legacy_test 2024-06-26T07:10:35.1529302Z inflating: build/bin/kernel_function_test 2024-06-26T07:10:35.1653014Z inflating: build/bin/kernel_lambda_legacy_test 2024-06-26T07:10:35.1754178Z inflating: build/bin/kernel_lambda_test 2024-06-26T07:10:35.1812719Z inflating: build/bin/kernel_stackbased_test 2024-06-26T07:10:35.1905840Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-06-26T07:10:35.1955378Z inflating: build/bin/CppSignature_test 2024-06-26T07:10:35.2009760Z inflating: build/bin/backend_fallback_test 2024-06-26T07:10:35.2057235Z inflating: build/bin/op_allowlist_test 2024-06-26T07:10:35.2119257Z inflating: build/bin/inline_container_test 2024-06-26T07:10:35.2410906Z inflating: build/bin/op_registration_test 2024-06-26T07:10:35.2462496Z inflating: build/bin/cuda_apply_test 2024-06-26T07:10:35.2511819Z inflating: build/bin/cuda_allocator_test 2024-06-26T07:10:35.2569671Z inflating: build/bin/cuda_atomic_ops_test 2024-06-26T07:10:35.2622522Z inflating: build/bin/cuda_caching_host_allocator_test 2024-06-26T07:10:35.2690791Z inflating: build/bin/cuda_complex_math_test 2024-06-26T07:10:35.2747593Z inflating: build/bin/cuda_complex_test 2024-06-26T07:10:35.2795512Z inflating: build/bin/cuda_device_test 2024-06-26T07:10:35.2851038Z inflating: build/bin/cuda_cub_test 2024-06-26T07:10:35.2900212Z inflating: build/bin/cuda_dlconvertor_test 2024-06-26T07:10:35.2962811Z inflating: build/bin/cuda_distributions_test 2024-06-26T07:10:35.3017852Z inflating: build/bin/cuda_generator_test 2024-06-26T07:10:35.3065949Z inflating: build/bin/cuda_half_test 2024-06-26T07:10:35.3115901Z inflating: build/bin/cuda_integer_divider_test 2024-06-26T07:10:35.3163046Z inflating: build/bin/cuda_optional_test 2024-06-26T07:10:35.3213197Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-06-26T07:10:35.3264300Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-06-26T07:10:35.3312458Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-06-26T07:10:35.3371226Z inflating: build/bin/cuda_stream_test 2024-06-26T07:10:35.3419626Z inflating: build/bin/cuda_cudnn_test 2024-06-26T07:10:35.3469643Z inflating: build/bin/cuda_vectorized_test 2024-06-26T07:10:35.3483852Z inflating: build/bin/tutorial_tensorexpr 2024-06-26T07:10:35.3547533Z inflating: build/bin/ProcessGroupGlooTest 2024-06-26T07:10:35.3603603Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-06-26T07:10:35.3664628Z inflating: build/bin/ProcessGroupNCCLTest 2024-06-26T07:10:35.3724421Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-06-26T07:10:35.4529071Z inflating: build/bin/test_tensorexpr 2024-06-26T07:10:35.5088466Z inflating: build/bin/test_jit 2024-06-26T07:10:35.5089038Z creating: .additional_ci_files/ 2024-06-26T07:10:35.5142545Z inflating: .additional_ci_files/test-times.json 2024-06-26T07:10:35.5353985Z inflating: .additional_ci_files/test-class-times.json 2024-06-26T07:10:35.5380510Z ##[group]Run rm artifacts.zip 2024-06-26T07:10:35.5381040Z rm artifacts.zip 2024-06-26T07:10:35.5390132Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:35.5390675Z env: 2024-06-26T07:10:35.5390963Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:35.5391421Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:35.5391902Z ##[endgroup] 2024-06-26T07:10:35.6096850Z ##[group]Run df -H 2024-06-26T07:10:35.6097180Z df -H 2024-06-26T07:10:35.6106364Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:35.6106865Z env: 2024-06-26T07:10:35.6107141Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:35.6107602Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:35.6108087Z ##[endgroup] 2024-06-26T07:10:35.6138380Z Filesystem Size Used Avail Use% Mounted on 2024-06-26T07:10:35.6138904Z devtmpfs 34G 0 34G 0% /dev 2024-06-26T07:10:35.6139378Z tmpfs 34G 2.8M 34G 1% /dev/shm 2024-06-26T07:10:35.6140095Z tmpfs 34G 455k 34G 1% /run 2024-06-26T07:10:35.6140626Z tmpfs 34G 0 34G 0% /sys/fs/cgroup 2024-06-26T07:10:35.6141111Z /dev/nvme0n1p1 162G 66G 96G 41% / 2024-06-26T07:10:35.6178867Z Prepare all required actions 2024-06-26T07:10:35.6179311Z Getting action download info 2024-06-26T07:10:35.7358431Z ##[group]Run ./.github/actions/download-td-artifacts 2024-06-26T07:10:35.7358890Z with: 2024-06-26T07:10:35.7359151Z env: 2024-06-26T07:10:35.7359429Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:35.7359880Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:35.7360357Z ##[endgroup] 2024-06-26T07:10:35.7391109Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:10:35.7391641Z with: 2024-06-26T07:10:35.7391971Z name: td_results 2024-06-26T07:10:35.7392365Z s3-bucket: gha-artifacts 2024-06-26T07:10:35.7392799Z region: us-east-1 2024-06-26T07:10:35.7393167Z env: 2024-06-26T07:10:35.7393499Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:35.7394032Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:35.7394521Z ##[endgroup] 2024-06-26T07:10:36.1405380Z (node:12781) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:10:36.1406066Z 2024-06-26T07:10:36.1406347Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:10:36.1407066Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:10:36.1407923Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:10:36.2068147Z Found 0 objects with prefix pytorch/pytorch/9673646808/td_results/ 2024-06-26T07:10:36.2074485Z Artifact download has finished successfully 2024-06-26T07:10:36.2214382Z ##[group]Run mkdir -p .additional_ci_files 2024-06-26T07:10:36.2214872Z mkdir -p .additional_ci_files 2024-06-26T07:10:36.2215442Z mv td_results.json .additional_ci_files/td_results.json 2024-06-26T07:10:36.2223541Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:36.2224053Z env: 2024-06-26T07:10:36.2224330Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:36.2224796Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:36.2225280Z ##[endgroup] 2024-06-26T07:10:36.2258951Z mv: cannot stat 'td_results.json': No such file or directory 2024-06-26T07:10:36.2269473Z ##[error]Process completed with exit code 1. 2024-06-26T07:10:36.2300563Z ##[group]Run .github/scripts/parse_ref.py 2024-06-26T07:10:36.2301042Z .github/scripts/parse_ref.py 2024-06-26T07:10:36.2307950Z shell: /usr/bin/bash -e {0} 2024-06-26T07:10:36.2308307Z env: 2024-06-26T07:10:36.2308593Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:36.2309045Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:36.2309527Z ##[endgroup] 2024-06-26T07:10:36.2539283Z Prepare all required actions 2024-06-26T07:10:36.2575862Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-06-26T07:10:36.2576325Z with: 2024-06-26T07:10:36.2576875Z github-token: *** 2024-06-26T07:10:36.2577316Z env: 2024-06-26T07:10:36.2577918Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:36.2578450Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:36.2578944Z ##[endgroup] 2024-06-26T07:10:36.2593943Z ##[group]Run set -eux 2024-06-26T07:10:36.2594288Z set -eux 2024-06-26T07:10:36.2594880Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-06-26T07:10:36.2602649Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:36.2603151Z env: 2024-06-26T07:10:36.2603435Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:36.2603891Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:36.2604536Z GITHUB_TOKEN: *** 2024-06-26T07:10:36.2604850Z ##[endgroup] 2024-06-26T07:10:36.2624498Z + python3 .github/scripts/get_workflow_job_id.py 9673646808 i-0f77a10328249b1bd 2024-06-26T07:10:37.5540725Z setting job-id=26688911389 2024-06-26T07:10:37.5542162Z setting job-name=cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:37.5734669Z Prepare all required actions 2024-06-26T07:10:37.5735115Z Getting action download info 2024-06-26T07:10:37.7309134Z ##[group]Run ./.github/actions/filter-test-configs 2024-06-26T07:10:37.7309591Z with: 2024-06-26T07:10:37.7310028Z github-token: *** 2024-06-26T07:10:37.7318155Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:10:37.7327084Z job-name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:37.7327796Z env: 2024-06-26T07:10:37.7328085Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:37.7328550Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:37.7329089Z ##[endgroup] 2024-06-26T07:10:37.7368948Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:10:37.7369494Z with: 2024-06-26T07:10:37.7369766Z shell: bash 2024-06-26T07:10:37.7370069Z timeout_minutes: 10 2024-06-26T07:10:37.7370402Z max_attempts: 5 2024-06-26T07:10:37.7370852Z retry_wait_seconds: 30 2024-06-26T07:10:37.7371956Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:10:37.7373162Z polling_interval_seconds: 1 2024-06-26T07:10:37.7373538Z warning_on_retry: true 2024-06-26T07:10:37.7373887Z continue_on_error: false 2024-06-26T07:10:37.7374224Z env: 2024-06-26T07:10:37.7374507Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:37.7374964Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:37.7375601Z GITHUB_TOKEN: *** 2024-06-26T07:10:37.7375926Z ##[endgroup] 2024-06-26T07:10:37.7865247Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:10:38.0045462Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:10:38.0196410Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1) 2024-06-26T07:10:38.0345334Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-06-26T07:10:38.0354471Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7) 2024-06-26T07:10:38.0367538Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12) 2024-06-26T07:10:38.0389161Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.6.2) 2024-06-26T07:10:38.0397738Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19) 2024-06-26T07:10:38.7865221Z Command completed after 1 attempt(s). 2024-06-26T07:10:38.7905281Z ##[group]Run set -x 2024-06-26T07:10:38.7905614Z set -x 2024-06-26T07:10:38.7905921Z  2024-06-26T07:10:38.7906491Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:10:38.7907169Z # in runner workspace 2024-06-26T07:10:38.7907705Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-06-26T07:10:38.7915506Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:38.7916004Z env: 2024-06-26T07:10:38.7916276Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:38.7916738Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:38.7917226Z ##[endgroup] 2024-06-26T07:10:38.7953588Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-06-26T07:10:38.8152601Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:10:38.8153154Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:10:38.8153621Z echo "Job name: ${JOB_NAME}" 2024-06-26T07:10:38.8154028Z  2024-06-26T07:10:38.8154588Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:10:38.8155265Z # in runner workspace 2024-06-26T07:10:38.8155841Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-06-26T07:10:38.8156477Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-06-26T07:10:38.8156932Z  --job-name "${JOB_NAME}" \ 2024-06-26T07:10:38.8165239Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-06-26T07:10:38.8173979Z  --selected-test-configs "" \ 2024-06-26T07:10:38.8174438Z  --pr-number "${PR_NUMBER}" \ 2024-06-26T07:10:38.8174855Z  --tag "${TAG}" \ 2024-06-26T07:10:38.8175254Z  --event-name "${EVENT_NAME}" \ 2024-06-26T07:10:38.8175700Z  --schedule "${SCHEDULE}" \ 2024-06-26T07:10:38.8176130Z  --branch "${HEAD_BRANCH}" 2024-06-26T07:10:38.8184085Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:38.8184599Z env: 2024-06-26T07:10:38.8184871Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:38.8185338Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:38.8186010Z GITHUB_TOKEN: *** 2024-06-26T07:10:38.8186666Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:38.8187384Z PR_NUMBER: 2024-06-26T07:10:38.8187699Z TAG: ciflow/inductor/129470 2024-06-26T07:10:38.8188069Z EVENT_NAME: push 2024-06-26T07:10:38.8188379Z SCHEDULE: 2024-06-26T07:10:38.8188671Z HEAD_BRANCH: 2024-06-26T07:10:38.8188973Z ##[endgroup] 2024-06-26T07:10:38.8209092Z Workflow: inductor 2024-06-26T07:10:38.8210351Z Job name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:39.0243111Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-06-26T07:10:39.1633952Z ##[group]Run echo "Filtered matrix:" 2024-06-26T07:10:39.1634419Z echo "Filtered matrix:" 2024-06-26T07:10:39.1642627Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-06-26T07:10:39.1651259Z  2024-06-26T07:10:39.1651638Z echo 2024-06-26T07:10:39.1652028Z echo "Is the current job unstable? False" 2024-06-26T07:10:39.1652482Z  2024-06-26T07:10:39.1652761Z echo 2024-06-26T07:10:39.1653128Z echo "Is keep-going label set? False" 2024-06-26T07:10:39.1653564Z  2024-06-26T07:10:39.1653838Z echo 2024-06-26T07:10:39.1654157Z echo "Renabled issues? " 2024-06-26T07:10:39.1662086Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:39.1662619Z env: 2024-06-26T07:10:39.1662908Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:39.1663365Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:39.1663848Z ##[endgroup] 2024-06-26T07:10:39.1683264Z Filtered matrix: 2024-06-26T07:10:39.1693262Z {include: [{config: inductor, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-06-26T07:10:39.1701639Z 2024-06-26T07:10:39.1701798Z Is the current job unstable? False 2024-06-26T07:10:39.1702083Z 2024-06-26T07:10:39.1702355Z Is keep-going label set? False 2024-06-26T07:10:39.1702625Z 2024-06-26T07:10:39.1702749Z Renabled issues? 2024-06-26T07:10:39.1741266Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:10:39.1742140Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:10:39.1749150Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:39.1749643Z env: 2024-06-26T07:10:39.1749926Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:39.1750379Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:39.1750871Z JOB_TIMEOUT: 240 2024-06-26T07:10:39.1751174Z ##[endgroup] 2024-06-26T07:10:39.1824081Z ##[group]Run set -x 2024-06-26T07:10:39.1824579Z set -x 2024-06-26T07:10:39.1824897Z  2024-06-26T07:10:39.1825253Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-06-26T07:10:39.1825798Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-06-26T07:10:39.1826363Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-06-26T07:10:39.1826876Z  TEST_COMMAND=.ci/onnx/test.sh 2024-06-26T07:10:39.1827292Z else 2024-06-26T07:10:39.1827633Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:10:39.1828071Z fi 2024-06-26T07:10:39.1828351Z  2024-06-26T07:10:39.1828824Z # detached container should get cleaned up by teardown_ec2_linux 2024-06-26T07:10:39.1829580Z # TODO: Stop building test binaries as part of the build phase 2024-06-26T07:10:39.1830263Z # Used for GPU_FLAG since that doesn't play nice 2024-06-26T07:10:39.1830856Z # shellcheck disable=SC2086,SC2090 2024-06-26T07:10:39.1831317Z container_name=$(docker run \ 2024-06-26T07:10:39.1831743Z  ${GPU_FLAG:-} \ 2024-06-26T07:10:39.1832123Z  -e BUILD_ENVIRONMENT \ 2024-06-26T07:10:39.1832532Z  -e PR_NUMBER \ 2024-06-26T07:10:39.1832894Z  -e GITHUB_ACTIONS \ 2024-06-26T07:10:39.1833296Z  -e GITHUB_REPOSITORY \ 2024-06-26T07:10:39.1833713Z  -e GITHUB_WORKFLOW \ 2024-06-26T07:10:39.1834111Z  -e GITHUB_JOB \ 2024-06-26T07:10:39.1834472Z  -e GITHUB_RUN_ID \ 2024-06-26T07:10:39.1834867Z  -e GITHUB_RUN_NUMBER \ 2024-06-26T07:10:39.1835282Z  -e GITHUB_RUN_ATTEMPT \ 2024-06-26T07:10:39.1835678Z  -e JOB_ID \ 2024-06-26T07:10:39.1836022Z  -e JOB_NAME \ 2024-06-26T07:10:39.1836376Z  -e BASE_SHA \ 2024-06-26T07:10:39.1836729Z  -e BRANCH \ 2024-06-26T07:10:39.1837061Z  -e SHA1 \ 2024-06-26T07:10:39.1837416Z  -e AWS_DEFAULT_REGION \ 2024-06-26T07:10:39.1837830Z  -e IN_WHEEL_TEST \ 2024-06-26T07:10:39.1838223Z  -e SHARD_NUMBER \ 2024-06-26T07:10:39.1838591Z  -e TEST_CONFIG \ 2024-06-26T07:10:39.1838972Z  -e NUM_TEST_SHARDS \ 2024-06-26T07:10:39.1839377Z  -e REENABLED_ISSUES \ 2024-06-26T07:10:39.1839796Z  -e CONTINUE_THROUGH_ERROR \ 2024-06-26T07:10:39.1840220Z  -e VERBOSE_TEST_LOGS \ 2024-06-26T07:10:39.1840624Z  -e NO_TEST_TIMEOUT \ 2024-06-26T07:10:39.1841006Z  -e NO_TD \ 2024-06-26T07:10:39.1841354Z  -e TD_DISTRIBUTED \ 2024-06-26T07:10:39.1841738Z  -e PR_LABELS \ 2024-06-26T07:10:39.1842150Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-06-26T07:10:39.1842650Z  -e SCCACHE_BUCKET \ 2024-06-26T07:10:39.1843055Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-06-26T07:10:39.1843470Z  -e XLA_CUDA \ 2024-06-26T07:10:39.1843882Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-06-26T07:10:39.1844399Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-06-26T07:10:39.1844925Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-06-26T07:10:39.1845447Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-06-26T07:10:39.1845931Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-06-26T07:10:39.1846359Z  -e DASHBOARD_TAG \ 2024-06-26T07:10:39.1846837Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-06-26T07:10:39.1847400Z  --security-opt seccomp=unconfined \ 2024-06-26T07:10:39.1847867Z  --cap-add=SYS_PTRACE \ 2024-06-26T07:10:39.1848384Z  --ipc=host \ 2024-06-26T07:10:39.1848885Z  --shm-size="${SHM_SIZE}" \ 2024-06-26T07:10:39.1849294Z  --tty \ 2024-06-26T07:10:39.1849617Z  --detach \ 2024-06-26T07:10:39.1849983Z  --name="${container_name}" \ 2024-06-26T07:10:39.1850400Z  --user jenkins \ 2024-06-26T07:10:39.1850958Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-06-26T07:10:39.1851534Z  -w /var/lib/jenkins/workspace \ 2024-06-26T07:10:39.1851986Z  "${DOCKER_IMAGE}" 2024-06-26T07:10:39.1852333Z ) 2024-06-26T07:10:39.1852756Z # Propagate download.pytorch.org IP to container 2024-06-26T07:10:39.1853679Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-06-26T07:10:39.1854644Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-06-26T07:10:39.1855553Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-06-26T07:10:39.1863586Z shell: /usr/bin/bash -e {0} 2024-06-26T07:10:39.1863945Z env: 2024-06-26T07:10:39.1864230Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:39.1864683Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:39.1865301Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:10:39.1865803Z PR_NUMBER: 2024-06-26T07:10:39.1866129Z GITHUB_REPOSITORY: pytorch/pytorch 2024-06-26T07:10:39.1866547Z GITHUB_WORKFLOW: inductor 2024-06-26T07:10:39.1866909Z GITHUB_JOB: test 2024-06-26T07:10:39.1867234Z GITHUB_RUN_ID: 9673646808 2024-06-26T07:10:39.1867596Z GITHUB_RUN_NUMBER: 76977 2024-06-26T07:10:39.1867949Z GITHUB_RUN_ATTEMPT: 1 2024-06-26T07:10:39.1868291Z JOB_ID: 26688911389 2024-06-26T07:10:39.1868946Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:10:39.1869659Z BRANCH: 2024-06-26T07:10:39.1869988Z SHA1: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:39.1870506Z BASE_SHA: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:10:39.1870990Z TEST_CONFIG: inductor_torchbench 2024-06-26T07:10:39.1871386Z SHARD_NUMBER: 1 2024-06-26T07:10:39.1871690Z NUM_TEST_SHARDS: 2 2024-06-26T07:10:39.1872017Z REENABLED_ISSUES: 2024-06-26T07:10:39.1872362Z CONTINUE_THROUGH_ERROR: False 2024-06-26T07:10:39.1872797Z VERBOSE_TEST_LOGS: False 2024-06-26T07:10:39.1873161Z NO_TEST_TIMEOUT: False 2024-06-26T07:10:39.1873498Z NO_TD: False 2024-06-26T07:10:39.1873803Z TD_DISTRIBUTED: False 2024-06-26T07:10:39.1874221Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-06-26T07:10:39.1874717Z SCCACHE_S3_KEY_PREFIX: inductor 2024-06-26T07:10:39.1875102Z SHM_SIZE: 2g 2024-06-26T07:10:39.1876181Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:10:39.1877340Z XLA_CUDA: 2024-06-26T07:10:39.1877826Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:10:39.1878435Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-06-26T07:10:39.1878881Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-06-26T07:10:39.1879285Z DASHBOARD_TAG: 2024-06-26T07:10:39.1879798Z HUGGING_FACE_HUB_TOKEN: *** 2024-06-26T07:10:39.1880169Z ##[endgroup] 2024-06-26T07:10:39.1899743Z + [[ inductor_torchbench == \m\u\l\t\i\g\p\u ]] 2024-06-26T07:10:39.1900537Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-06-26T07:10:39.1901063Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:10:39.1907233Z +++ nproc --ignore=2 2024-06-26T07:10:39.1921357Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9673646808 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:10:50.4605102Z + container_name=604b66743ea1a58a12a4e607c457b971331d736afd449b38895678368be0fe52 2024-06-26T07:10:50.4606921Z + docker exec -i 604b66743ea1a58a12a4e607c457b971331d736afd449b38895678368be0fe52 sudo bash -c '/bin/cat >> /etc/hosts' 2024-06-26T07:10:50.4607760Z + grep download.pytorch.org /etc/hosts 2024-06-26T07:10:50.5376586Z + echo DOCKER_CONTAINER_ID=604b66743ea1a58a12a4e607c457b971331d736afd449b38895678368be0fe52 2024-06-26T07:10:50.5378862Z ++ echo dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:10:50.5380625Z + docker exec -t 604b66743ea1a58a12a4e607c457b971331d736afd449b38895678368be0fe52 sh -c 'pip install dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-06-26T07:10:50.9184583Z Processing ./dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+gitb8c4c54) 2024-06-26T07:10:51.2300622Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.13.1) 2024-06-26T07:10:51.2304363Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (4.12.2) 2024-06-26T07:10:51.2307777Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.12.1) 2024-06-26T07:10:51.2310925Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.8.8) 2024-06-26T07:10:51.2313444Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.1.4) 2024-06-26T07:10:51.2316844Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2024.2.0) 2024-06-26T07:10:51.2347224Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.3.0) 2024-06-26T07:10:51.2406753Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.21.2) 2024-06-26T07:10:51.2832132Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.1.5) 2024-06-26T07:10:51.2988419Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.3.0) 2024-06-26T07:10:52.6021298Z Installing collected packages: torch 2024-06-26T07:11:01.2992645Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-06-26T07:11:01.2994336Z timm 0.9.7 requires torchvision, which is not installed. 2024-06-26T07:11:01.2995022Z Successfully installed torch-2.5.0a0+gitb8c4c54 2024-06-26T07:11:01.3770789Z ++ dirname .ci/pytorch/test.sh 2024-06-26T07:11:01.3774966Z + source .ci/pytorch/common.sh 2024-06-26T07:11:01.3776828Z +++ dirname .ci/pytorch/common.sh 2024-06-26T07:11:01.3782071Z ++ source .ci/pytorch/common_utils.sh 2024-06-26T07:11:01.3784253Z +++ declare -f -t trap_add 2024-06-26T07:11:01.3788615Z ++ set -ex 2024-06-26T07:11:01.3789062Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:11:01.3789566Z ++ BUILD_TEST_LIBTORCH=0 2024-06-26T07:11:01.3790181Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-06-26T07:11:01.3792521Z ++ stat -c %u /var/lib/jenkins/workspace 2024-06-26T07:11:01.3801686Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-06-26T07:11:01.3802118Z + trap_add cleanup_workspace EXIT 2024-06-26T07:11:01.3802627Z + trap_add_cmd=cleanup_workspace 2024-06-26T07:11:01.3803165Z + shift 2024-06-26T07:11:01.3803548Z + for trap_add_name in "$@" 2024-06-26T07:11:01.3807798Z +++ trap -p EXIT 2024-06-26T07:11:01.3809973Z ++ eval 'extract_trap_cmd ' 2024-06-26T07:11:01.3810484Z +++ extract_trap_cmd 2024-06-26T07:11:01.3810958Z +++ printf '%s\n' '' 2024-06-26T07:11:01.3811340Z ++ printf '%s\n' cleanup_workspace 2024-06-26T07:11:01.3813000Z + trap -- ' 2024-06-26T07:11:01.3813477Z cleanup_workspace' EXIT 2024-06-26T07:11:01.3814105Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-06-26T07:11:01.9621368Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-06-26T07:11:01.9632642Z + echo 'Environment variables:' 2024-06-26T07:11:01.9633065Z Environment variables: 2024-06-26T07:11:01.9633452Z + env 2024-06-26T07:11:01.9639578Z INSTALLED_DB=yes 2024-06-26T07:11:01.9640079Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:11:01.9640573Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:11:01.9641041Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:11:01.9641639Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:11:01.9642315Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:11:01.9642810Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:11:01.9643323Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:11:01.9643897Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:11:01.9646649Z HOSTNAME=604b66743ea1 2024-06-26T07:11:01.9647598Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_ec614a92-22ee-4f50-b5fc-3e91e7583d86 2024-06-26T07:11:01.9648608Z GITHUB_ACTION=__self 2024-06-26T07:11:01.9649272Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:11:01.9653273Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:11:01.9657347Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:11:01.9658323Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:11:01.9658844Z GITHUB_RUN_NUMBER=76977 2024-06-26T07:11:01.9659272Z TEST_CONFIG=inductor_torchbench 2024-06-26T07:11:01.9659842Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:11:01.9660447Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:11:01.9660959Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:11:01.9661401Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:11:01.9661820Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:11:01.9662501Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:11:01.9663003Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:11:01.9663464Z GITHUB_REF_TYPE=tag 2024-06-26T07:11:01.9663792Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:11:01.9664158Z NCCL_VERSION=2.17.1-1 2024-06-26T07:11:01.9664545Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:01.9664979Z XLA_CUDA= 2024-06-26T07:11:01.9665581Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:11:01.9667543Z *** 2024-06-26T07:11:01.9667829Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:11:01.9668226Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:11:01.9668601Z GITHUB_ACTIONS=true 2024-06-26T07:11:01.9669034Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:01.9669574Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:11:01.9670097Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:11:01.9670556Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:01.9671040Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:11:01.9671527Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:01.9672371Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:11:01.9673151Z UCC_HOME=/usr 2024-06-26T07:11:01.9673481Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:11:01.9673874Z VERBOSE_TEST_LOGS=False 2024-06-26T07:11:01.9674227Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:11:01.9674802Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:11:01.9675424Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:11:01.9676086Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:11:01.9676465Z SHARD_NUMBER=1 2024-06-26T07:11:01.9676763Z GITHUB_REF_PROTECTED=false 2024-06-26T07:11:01.9677112Z HOME=/var/lib/jenkins 2024-06-26T07:11:01.9677472Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:11:01.9677907Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:11:01.9678380Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:11:01.9678866Z SCCACHE_S3_KEY_PREFIX=inductor 2024-06-26T07:11:01.9679228Z CUDA_VERSION=12.1.1 2024-06-26T07:11:01.9679655Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:11:01.9680149Z NUM_TEST_SHARDS=2 2024-06-26T07:11:01.9680447Z UCX_HOME=/usr 2024-06-26T07:11:01.9681106Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:11:01.9682218Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_ec614a92-22ee-4f50-b5fc-3e91e7583d86 2024-06-26T07:11:01.9683426Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:11:01.9684623Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_ec614a92-22ee-4f50-b5fc-3e91e7583d86 2024-06-26T07:11:01.9685727Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:11:01.9686428Z GITHUB_EVENT_NAME=push 2024-06-26T07:11:01.9686750Z DASHBOARD_TAG= 2024-06-26T07:11:01.9687058Z GITHUB_RUN_ID=9673646808 2024-06-26T07:11:01.9687509Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:11:01.9688048Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:11:01.9689343Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_ec614a92-22ee-4f50-b5fc-3e91e7583d86 2024-06-26T07:11:01.9690371Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:11:01.9690822Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:11:01.9691196Z PR_NUMBER= 2024-06-26T07:11:01.9691480Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:11:01.9691819Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:11:01.9692263Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:11:01.9692716Z TERM=xterm 2024-06-26T07:11:01.9693051Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:11:01.9693469Z INSTALLED_VISION=yes 2024-06-26T07:11:01.9693787Z BRANCH= 2024-06-26T07:11:01.9694113Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:11:01.9694518Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:11:01.9695053Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:11:01.9695839Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:11:01.9696816Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:11:01.9697357Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:11:01.9698125Z REENABLED_ISSUES= 2024-06-26T07:11:01.9698428Z SHLVL=1 2024-06-26T07:11:01.9698814Z MAX_JOBS=14 2024-06-26T07:11:01.9699149Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:11:01.9699506Z NVARCH=x86_64 2024-06-26T07:11:01.9699801Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:11:01.9700257Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:01.9700793Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:11:01.9701269Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:11:01.9701928Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:11:01.9702501Z GITHUB_JOB=test 2024-06-26T07:11:01.9702913Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:11:01.9703464Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:11:01.9704043Z NO_TEST_TIMEOUT=False 2024-06-26T07:11:01.9704392Z TD_DISTRIBUTED=False 2024-06-26T07:11:01.9704771Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:11:01.9705199Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:11:01.9705692Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:11:01.9706065Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:11:01.9706411Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:11:01.9706753Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:11:01.9707776Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:01.9708884Z GITHUB_BASE_REF= 2024-06-26T07:11:01.9709199Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:11:01.9709562Z CI=true 2024-06-26T07:11:01.9709925Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:11:01.9710348Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:11:01.9710707Z JOB_ID=26688911389 2024-06-26T07:11:01.9711017Z INSTALLED_PROTOBUF=yes 2024-06-26T07:11:01.9711344Z GITHUB_HEAD_REF= 2024-06-26T07:11:01.9711738Z GITHUB_ACTION_REF= 2024-06-26T07:11:01.9712287Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:11:01.9712763Z GITHUB_WORKFLOW=inductor 2024-06-26T07:11:01.9713119Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:11:01.9714170Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_ec614a92-22ee-4f50-b5fc-3e91e7583d86 2024-06-26T07:11:01.9715076Z NO_TD=False 2024-06-26T07:11:01.9715383Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:11:01.9715765Z _=/usr/bin/env 2024-06-26T07:11:01.9716266Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-06-26T07:11:01.9783901Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-06-26T07:11:01.9784979Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T07:11:01.9785940Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-06-26T07:11:01.9786813Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-06-26T07:11:01.9787585Z + BUILD_DIR=build 2024-06-26T07:11:01.9787942Z + BUILD_RENAMED_DIR=build_renamed 2024-06-26T07:11:01.9788336Z + BUILD_BIN_DIR=build/bin 2024-06-26T07:11:01.9788683Z + SHARD_NUMBER=1 2024-06-26T07:11:01.9788988Z + NUM_TEST_SHARDS=2 2024-06-26T07:11:01.9789311Z + export VALGRIND=ON 2024-06-26T07:11:01.9789633Z + VALGRIND=ON 2024-06-26T07:11:01.9790094Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-06-26T07:11:01.9790587Z + [[ 0 == \1 ]] 2024-06-26T07:11:01.9790882Z + [[ False == \1 ]] 2024-06-26T07:11:01.9791356Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-06-26T07:11:01.9791872Z ++ realpath build/custom_test_artifacts 2024-06-26T07:11:01.9794149Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-06-26T07:11:01.9795002Z + [[ -n '' ]] 2024-06-26T07:11:01.9795353Z + echo 'Environment variables' 2024-06-26T07:11:01.9795727Z Environment variables 2024-06-26T07:11:01.9796047Z + env 2024-06-26T07:11:01.9800821Z INSTALLED_DB=yes 2024-06-26T07:11:01.9801721Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:11:01.9802259Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:11:01.9802747Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:11:01.9803565Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:11:01.9804361Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:11:01.9805024Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:11:01.9805762Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:11:01.9806357Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:11:01.9807043Z HOSTNAME=604b66743ea1 2024-06-26T07:11:01.9808088Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_ec614a92-22ee-4f50-b5fc-3e91e7583d86 2024-06-26T07:11:01.9809051Z GITHUB_ACTION=__self 2024-06-26T07:11:01.9809434Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:11:01.9813169Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:11:01.9831987Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:11:01.9832525Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:11:01.9832895Z GITHUB_RUN_NUMBER=76977 2024-06-26T07:11:01.9833246Z TEST_CONFIG=inductor_torchbench 2024-06-26T07:11:01.9833649Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:11:01.9834119Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:11:01.9834580Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:11:01.9835012Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:11:01.9835440Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:11:01.9835891Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:11:01.9836389Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:11:01.9836844Z GITHUB_REF_TYPE=tag 2024-06-26T07:11:01.9837175Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:11:01.9837559Z NCCL_VERSION=2.17.1-1 2024-06-26T07:11:01.9837941Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:01.9838384Z XLA_CUDA= 2024-06-26T07:11:01.9838872Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:11:01.9839287Z *** 2024-06-26T07:11:01.9839569Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:11:01.9839973Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:11:01.9840338Z GITHUB_ACTIONS=true 2024-06-26T07:11:01.9840671Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:01.9841150Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:11:01.9841683Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:11:01.9842148Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:01.9842658Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:11:01.9843132Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:01.9843918Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:11:01.9844633Z UCC_HOME=/usr 2024-06-26T07:11:01.9844970Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:11:01.9845364Z VERBOSE_TEST_LOGS=False 2024-06-26T07:11:01.9845709Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:11:01.9846171Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:11:01.9846673Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:11:01.9847280Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:11:01.9847662Z SHARD_NUMBER=1 2024-06-26T07:11:01.9847973Z GITHUB_REF_PROTECTED=false 2024-06-26T07:11:01.9848319Z HOME=/var/lib/jenkins 2024-06-26T07:11:01.9848797Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:11:01.9849242Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:11:01.9849805Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:11:01.9850283Z SCCACHE_S3_KEY_PREFIX=inductor 2024-06-26T07:11:01.9850653Z CUDA_VERSION=12.1.1 2024-06-26T07:11:01.9851080Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:11:01.9851523Z NUM_TEST_SHARDS=2 2024-06-26T07:11:01.9851818Z UCX_HOME=/usr 2024-06-26T07:11:01.9852325Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:11:01.9853400Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_ec614a92-22ee-4f50-b5fc-3e91e7583d86 2024-06-26T07:11:01.9854606Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:11:01.9855809Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_ec614a92-22ee-4f50-b5fc-3e91e7583d86 2024-06-26T07:11:01.9856914Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:11:01.9857782Z GITHUB_EVENT_NAME=push 2024-06-26T07:11:01.9858137Z DASHBOARD_TAG= 2024-06-26T07:11:01.9858434Z GITHUB_RUN_ID=9673646808 2024-06-26T07:11:01.9858894Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:11:01.9859475Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:11:01.9860475Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_ec614a92-22ee-4f50-b5fc-3e91e7583d86 2024-06-26T07:11:01.9861377Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:11:01.9861785Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:11:01.9862169Z PR_NUMBER= 2024-06-26T07:11:01.9862447Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:11:01.9862770Z VALGRIND=ON 2024-06-26T07:11:01.9863077Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:11:01.9863524Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:11:01.9863974Z TERM=xterm 2024-06-26T07:11:01.9864315Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:11:01.9864734Z INSTALLED_VISION=yes 2024-06-26T07:11:01.9865048Z BRANCH= 2024-06-26T07:11:01.9865354Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:11:01.9865765Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:11:01.9866193Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:11:01.9866938Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:11:01.9867686Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:11:01.9868185Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:11:01.9868651Z REENABLED_ISSUES= 2024-06-26T07:11:01.9868952Z SHLVL=1 2024-06-26T07:11:01.9869208Z MAX_JOBS=14 2024-06-26T07:11:01.9869529Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:11:01.9869892Z NVARCH=x86_64 2024-06-26T07:11:01.9870183Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:11:01.9870635Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:01.9871164Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:11:01.9871656Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:11:01.9872308Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:11:01.9872886Z GITHUB_JOB=test 2024-06-26T07:11:01.9873288Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:11:01.9873848Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:11:01.9874357Z NO_TEST_TIMEOUT=False 2024-06-26T07:11:01.9874688Z TD_DISTRIBUTED=False 2024-06-26T07:11:01.9875066Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:11:01.9875515Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:11:01.9875927Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:11:01.9876293Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:11:01.9876648Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:11:01.9877101Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:11:01.9878127Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:01.9879176Z GITHUB_BASE_REF= 2024-06-26T07:11:01.9879505Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:11:01.9879871Z CI=true 2024-06-26T07:11:01.9880258Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:11:01.9880681Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:11:01.9881059Z JOB_ID=26688911389 2024-06-26T07:11:01.9881369Z INSTALLED_PROTOBUF=yes 2024-06-26T07:11:01.9881700Z GITHUB_HEAD_REF= 2024-06-26T07:11:01.9882006Z GITHUB_ACTION_REF= 2024-06-26T07:11:01.9882435Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:11:01.9882911Z GITHUB_WORKFLOW=inductor 2024-06-26T07:11:01.9883278Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:11:01.9884172Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_ec614a92-22ee-4f50-b5fc-3e91e7583d86 2024-06-26T07:11:01.9884983Z NO_TD=False 2024-06-26T07:11:01.9885289Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:11:01.9885653Z _=/usr/bin/env 2024-06-26T07:11:01.9886001Z + echo 'Testing pytorch' 2024-06-26T07:11:01.9886337Z Testing pytorch 2024-06-26T07:11:01.9886666Z + export LANG=C.UTF-8 2024-06-26T07:11:01.9887002Z + LANG=C.UTF-8 2024-06-26T07:11:01.9887287Z + PR_NUMBER= 2024-06-26T07:11:01.9887640Z + [[ inductor_torchbench == \d\e\f\a\u\l\t ]] 2024-06-26T07:11:01.9888157Z + [[ inductor_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-06-26T07:11:01.9888738Z + [[ inductor_torchbench == \s\l\o\w ]] 2024-06-26T07:11:01.9889406Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-06-26T07:11:01.9890086Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-06-26T07:11:01.9890632Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:11:01.9891103Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:11:01.9891543Z + [[ inductor_torchbench == *crossref* ]] 2024-06-26T07:11:01.9892112Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:11:01.9892735Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-06-26T07:11:01.9893382Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:11:01.9893923Z + pip_install --user ninja==1.10.2 2024-06-26T07:11:01.9894449Z + pip install --progress-bar off --user ninja==1.10.2 2024-06-26T07:11:02.3423440Z Collecting ninja==1.10.2 2024-06-26T07:11:02.3591404Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-06-26T07:11:02.3738299Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-06-26T07:11:03.6751555Z Installing collected packages: ninja 2024-06-26T07:11:03.6816801Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-06-26T07:11:03.6818238Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-06-26T07:11:03.6878906Z Successfully installed ninja-1.10.2 2024-06-26T07:11:03.7742853Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:03.7744994Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:03.7746509Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:11:03.7747016Z + install_tlparse 2024-06-26T07:11:03.7747390Z + pip_install --user tlparse==0.3.7 2024-06-26T07:11:03.7747936Z + pip install --progress-bar off --user tlparse==0.3.7 2024-06-26T07:11:04.1205837Z Collecting tlparse==0.3.7 2024-06-26T07:11:04.1353539Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-06-26T07:11:04.1452763Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-06-26T07:11:05.4407485Z Installing collected packages: tlparse 2024-06-26T07:11:05.4778786Z Successfully installed tlparse-0.3.7 2024-06-26T07:11:05.5604817Z ++ python -m site --user-base 2024-06-26T07:11:05.5759732Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:05.5761740Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-06-26T07:11:05.5762514Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-06-26T07:11:05.5763179Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:11:05.5764106Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-06-26T07:11:05.5765245Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-06-26T07:11:05.5765970Z + cd test 2024-06-26T07:11:05.5766495Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-06-26T07:11:07.0603814Z + [[ inductor_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-06-26T07:11:07.0604444Z + [[ inductor_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-06-26T07:11:07.0605292Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:11:07.0605909Z + pushd test 2024-06-26T07:11:07.0606280Z ~/workspace/test ~/workspace 2024-06-26T07:11:07.0606881Z ++ python -c 'import torch; print(torch.version.cuda)' 2024-06-26T07:11:08.5460220Z + CUDA_VERSION=12.1 2024-06-26T07:11:08.5460829Z + '[' 12.1 == 12.4 ']' 2024-06-26T07:11:08.5461229Z + ISCUDA124= 2024-06-26T07:11:08.5461526Z + popd 2024-06-26T07:11:08.5461795Z ~/workspace 2024-06-26T07:11:08.5465209Z + DYNAMO_BENCHMARK_FLAGS=() 2024-06-26T07:11:08.5466507Z + [[ inductor_torchbench == *dynamo_eager* ]] 2024-06-26T07:11:08.5467185Z + [[ inductor_torchbench == *aot_eager* ]] 2024-06-26T07:11:08.5467714Z + [[ inductor_torchbench == *aot_inductor* ]] 2024-06-26T07:11:08.5468255Z + [[ inductor_torchbench == *inductor* ]] 2024-06-26T07:11:08.5468707Z + [[ inductor_torchbench != *perf* ]] 2024-06-26T07:11:08.5469279Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-06-26T07:11:08.5469785Z + [[ inductor_torchbench == *dynamic* ]] 2024-06-26T07:11:08.5470280Z + [[ inductor_torchbench == *cpu_inductor* ]] 2024-06-26T07:11:08.5470872Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-06-26T07:11:08.5493068Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:11:08.5493751Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-06-26T07:11:08.5496706Z + cd test 2024-06-26T07:11:08.5497259Z + python -c 'import torch; print(torch.__config__.show())' 2024-06-26T07:11:09.8790082Z PyTorch built with: 2024-06-26T07:11:09.8790719Z - GCC 9.4 2024-06-26T07:11:09.8791208Z - C++ Version: 201703 2024-06-26T07:11:09.8792162Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:11:09.8793239Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:11:09.8793908Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:11:09.8794439Z - LAPACK is enabled (usually provided by MKL) 2024-06-26T07:11:09.8794924Z - NNPACK is enabled 2024-06-26T07:11:09.8795315Z - CPU capability usage: AVX2 2024-06-26T07:11:09.8795729Z - CUDA Runtime 12.1 2024-06-26T07:11:09.8796251Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-06-26T07:11:09.8796858Z - CuDNN 90.1 (built against CUDA 12.4) 2024-06-26T07:11:09.8797299Z - Magma 2.6.1 2024-06-26T07:11:09.8804459Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-06-26T07:11:09.8811177Z 2024-06-26T07:11:10.1544952Z + cd test 2024-06-26T07:11:10.1545622Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-06-26T07:11:11.3535445Z ATen/Parallel: 2024-06-26T07:11:11.3535937Z at::get_num_threads() : 8 2024-06-26T07:11:11.3536467Z at::get_num_interop_threads() : 16 2024-06-26T07:11:11.3537046Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:11:11.3537932Z omp_get_max_threads() : 8 2024-06-26T07:11:11.3539244Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:11:11.3540163Z mkl_get_max_threads() : 8 2024-06-26T07:11:11.3540781Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:11:11.3541413Z std::thread::hardware_concurrency() : 16 2024-06-26T07:11:11.3541836Z Environment variables: 2024-06-26T07:11:11.3542183Z OMP_NUM_THREADS : [not set] 2024-06-26T07:11:11.3542553Z MKL_NUM_THREADS : [not set] 2024-06-26T07:11:11.3542938Z ATen parallel backend: OpenMP 2024-06-26T07:11:11.3543188Z 2024-06-26T07:11:11.5949268Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:11:11.5949970Z + [[ inductor_torchbench == *backward* ]] 2024-06-26T07:11:11.5950550Z + [[ inductor_torchbench == *xla* ]] 2024-06-26T07:11:11.5951137Z + [[ inductor_torchbench == *executorch* ]] 2024-06-26T07:11:11.5951774Z + [[ inductor_torchbench == \j\i\t\_\l\e\g\a\c\y ]] 2024-06-26T07:11:11.5952423Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:11:11.5952959Z + [[ inductor_torchbench == distributed ]] 2024-06-26T07:11:11.5953469Z + [[ inductor_torchbench == *inductor_distributed* ]] 2024-06-26T07:11:11.5954046Z + [[ inductor_torchbench == *inductor-halide* ]] 2024-06-26T07:11:11.5954645Z + [[ inductor_torchbench == *inductor-micro-benchmark* ]] 2024-06-26T07:11:11.5955172Z + [[ inductor_torchbench == *huggingface* ]] 2024-06-26T07:11:11.5955628Z + [[ inductor_torchbench == *timm* ]] 2024-06-26T07:11:11.5956082Z + [[ inductor_torchbench == *torchbench* ]] 2024-06-26T07:11:11.5956558Z + [[ inductor_torchbench == *cpu_inductor* ]] 2024-06-26T07:11:11.5957003Z + install_torchaudio cuda 2024-06-26T07:11:11.5957341Z + local commit 2024-06-26T07:11:11.5957658Z ++ get_pinned_commit audio 2024-06-26T07:11:11.5958031Z ++ cat .github/ci_commit_pins/audio.txt 2024-06-26T07:11:11.5963614Z + commit=b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:11:11.5964103Z + [[ cuda == \c\u\d\a ]] 2024-06-26T07:11:11.5964529Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-06-26T07:11:11.5965426Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:11:11.5966845Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:11:11.9065126Z Collecting git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:11:11.9068683Z Cloning https://github.com/pytorch/audio.git (to revision b829e936f7cc61b48149f5f957a451a38bf2a178) to /tmp/pip-req-build-ak5wwr8w 2024-06-26T07:11:11.9083706Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-ak5wwr8w 2024-06-26T07:11:12.5535666Z Running command git rev-parse -q --verify 'sha^b829e936f7cc61b48149f5f957a451a38bf2a178' 2024-06-26T07:11:12.5550762Z Running command git fetch -q https://github.com/pytorch/audio.git b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:11:12.9358640Z Running command git checkout -q b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:11:13.0493402Z Resolved https://github.com/pytorch/audio.git to commit b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:11:13.0494410Z Running command git submodule update --init --recursive -q 2024-06-26T07:11:15.2112732Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:11:15.2144803Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.4.0a0+b829e93) (2.5.0a0+gitb8c4c54) 2024-06-26T07:11:15.2207506Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (3.13.1) 2024-06-26T07:11:15.2214081Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (4.12.2) 2024-06-26T07:11:15.2216824Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (1.12.1) 2024-06-26T07:11:15.2220989Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (2.8.8) 2024-06-26T07:11:15.2224989Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (3.1.4) 2024-06-26T07:11:15.2228592Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (2024.2.0) 2024-06-26T07:11:15.2651177Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.4.0a0+b829e93) (2.1.5) 2024-06-26T07:11:15.2804614Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchaudio==2.4.0a0+b829e93) (1.3.0) 2024-06-26T07:11:15.2873326Z Building wheels for collected packages: torchaudio 2024-06-26T07:13:41.7726870Z Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-06-26T07:13:41.7756371Z [?25h Created wheel for torchaudio: filename=torchaudio-2.4.0a0+b829e93-cp310-cp310-linux_x86_64.whl size=2308809 sha256=5348a18d734ce2f6e6b4ed18840709449d86c07f1b25e50621ef51dd62b2f0dc 2024-06-26T07:13:41.7757972Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/30/b3/9c/882f183f00af2e582825654a92a04400924519922db53ff522 2024-06-26T07:13:41.7784859Z Successfully built torchaudio 2024-06-26T07:13:42.9785911Z Installing collected packages: torchaudio 2024-06-26T07:13:43.1862647Z Successfully installed torchaudio-2.4.0a0+b829e93 2024-06-26T07:13:43.3174088Z + install_torchtext 2024-06-26T07:13:43.3174596Z + local data_commit 2024-06-26T07:13:43.3174917Z + local text_commit 2024-06-26T07:13:43.3177961Z ++ get_pinned_commit data 2024-06-26T07:13:43.3178450Z ++ cat .github/ci_commit_pins/data.txt 2024-06-26T07:13:43.3188106Z + data_commit=11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:13:43.3190191Z ++ get_pinned_commit text 2024-06-26T07:13:43.3190823Z ++ cat .github/ci_commit_pins/text.txt 2024-06-26T07:13:43.3200191Z + text_commit=b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:13:43.3201345Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:13:43.3203743Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:13:43.6319757Z Collecting git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:13:43.6323417Z Cloning https://github.com/pytorch/data.git (to revision 11bb5b847ea8b9e0d9bb82db3304daf383008d3f) to /tmp/pip-req-build-mbwq0y5q 2024-06-26T07:13:43.6339337Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/data.git /tmp/pip-req-build-mbwq0y5q 2024-06-26T07:13:44.0291633Z Running command git rev-parse -q --verify 'sha^11bb5b847ea8b9e0d9bb82db3304daf383008d3f' 2024-06-26T07:13:44.0310955Z Running command git fetch -q https://github.com/pytorch/data.git 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:13:44.2577956Z Running command git checkout -q 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:13:44.3872632Z Resolved https://github.com/pytorch/data.git to commit 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:13:44.3873718Z Running command git submodule update --init --recursive -q 2024-06-26T07:14:56.3645979Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:14:56.3697311Z [?25hRequirement already satisfied: urllib3>=1.25 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (1.26.19) 2024-06-26T07:14:56.3700384Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.32.3) 2024-06-26T07:14:56.3706199Z Requirement already satisfied: torch>2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.5.0a0+gitb8c4c54) 2024-06-26T07:14:56.3766790Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.13.1) 2024-06-26T07:14:56.3772635Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (4.12.2) 2024-06-26T07:14:56.3775933Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.12.1) 2024-06-26T07:14:56.3779587Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.8.8) 2024-06-26T07:14:56.3783179Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.1.4) 2024-06-26T07:14:56.3786385Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2024.2.0) 2024-06-26T07:14:56.3971838Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.3.2) 2024-06-26T07:14:56.3978510Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.7) 2024-06-26T07:14:56.3984764Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (2024.6.2) 2024-06-26T07:14:56.4442457Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.1.5) 2024-06-26T07:14:56.4604652Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.3.0) 2024-06-26T07:14:56.4686033Z Building wheels for collected packages: torchdata 2024-06-26T07:14:58.2000869Z Building wheel for torchdata (setup.py) ... [?25l- \ | done 2024-06-26T07:14:58.2007340Z [?25h Created wheel for torchdata: filename=torchdata-0.7.0a0+11bb5b8-py3-none-any.whl size=182956 sha256=0363b57e98336e4e5b402e6c9d4af55ad5b490e6cc20092da0a0e6cc7e2c2ae0 2024-06-26T07:14:58.2009407Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/ca/59/a3/c8250bfc8d3d4d639498d4beb2e0f0e70b9a508ac61fde85ce 2024-06-26T07:14:58.2035105Z Successfully built torchdata 2024-06-26T07:14:59.4146286Z Installing collected packages: torchdata 2024-06-26T07:14:59.5080042Z Successfully installed torchdata-0.7.0a0+11bb5b8 2024-06-26T07:15:02.0101374Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:15:02.0102818Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:15:02.3218606Z Collecting git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:15:02.3223054Z Cloning https://github.com/pytorch/text.git (to revision b0ebddc648d279826089db91775375221777a2db) to /tmp/pip-req-build-2z_3h6ov 2024-06-26T07:15:02.3237609Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-2z_3h6ov 2024-06-26T07:15:03.0706561Z Running command git rev-parse -q --verify 'sha^b0ebddc648d279826089db91775375221777a2db' 2024-06-26T07:15:03.0720820Z Running command git fetch -q https://github.com/pytorch/text.git b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:15:03.5253640Z Running command git checkout -q b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:15:03.6706477Z Resolved https://github.com/pytorch/text.git to commit b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:15:03.6707525Z Running command git submodule update --init --recursive -q 2024-06-26T07:15:10.4342241Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:15:10.4393599Z [?25hRequirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (4.66.4) 2024-06-26T07:15:10.4396823Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.32.3) 2024-06-26T07:15:10.4400561Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.5.0a0+gitb8c4c54) 2024-06-26T07:15:10.4403331Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (1.21.2) 2024-06-26T07:15:10.4406633Z Requirement already satisfied: torchdata in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (0.7.0a0+11bb5b8) 2024-06-26T07:15:10.4467921Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.3.2) 2024-06-26T07:15:10.4473281Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.7) 2024-06-26T07:15:10.4480059Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (1.26.19) 2024-06-26T07:15:10.4485226Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (2024.6.2) 2024-06-26T07:15:10.4542269Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.13.1) 2024-06-26T07:15:10.4549181Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (4.12.2) 2024-06-26T07:15:10.4552047Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (1.12.1) 2024-06-26T07:15:10.4555345Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2.8.8) 2024-06-26T07:15:10.4558733Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.1.4) 2024-06-26T07:15:10.4561810Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2024.2.0) 2024-06-26T07:15:10.5251868Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchtext==0.16.0a0+b0ebddc) (2.1.5) 2024-06-26T07:15:10.5414242Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchtext==0.16.0a0+b0ebddc) (1.3.0) 2024-06-26T07:15:10.5497306Z Building wheels for collected packages: torchtext 2024-06-26T07:15:47.1544343Z Building wheel for torchtext (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-06-26T07:15:47.1571063Z [?25h Created wheel for torchtext: filename=torchtext-0.16.0a0+b0ebddc-cp310-cp310-linux_x86_64.whl size=2045304 sha256=c23ae9f8ec876479cc0ff67081cd446d50247cba8b5a14bcbdd8a00d59f010a2 2024-06-26T07:15:47.1574041Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/e5/3d/d7/faafad098ec9437ecdf4495c0f5e72b817fffbb06063e32cbf 2024-06-26T07:15:47.1596988Z Successfully built torchtext 2024-06-26T07:15:48.3559733Z Installing collected packages: torchtext 2024-06-26T07:15:48.4630669Z Successfully installed torchtext-0.16.0a0+b0ebddc 2024-06-26T07:15:48.5847198Z + install_torchvision 2024-06-26T07:15:48.5848722Z + local orig_preload 2024-06-26T07:15:48.5849079Z + local commit 2024-06-26T07:15:48.5849515Z ++ get_pinned_commit vision 2024-06-26T07:15:48.5849916Z ++ cat .github/ci_commit_pins/vision.txt 2024-06-26T07:15:48.5857252Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:15:48.5858158Z + orig_preload= 2024-06-26T07:15:48.5858667Z + '[' -n '' ']' 2024-06-26T07:15:48.5859850Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:15:48.5861401Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:15:48.8971961Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:15:48.8975725Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-u3gem4r7 2024-06-26T07:15:48.8991085Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-u3gem4r7 2024-06-26T07:15:50.9876399Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-06-26T07:15:50.9895502Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:15:52.3087697Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:15:52.5602809Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:15:54.8890661Z Preparing metadata (setup.py) ... [?25l- \ done 2024-06-26T07:15:54.8938368Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2) 2024-06-26T07:15:54.8940931Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+gitb8c4c54) 2024-06-26T07:15:54.8946156Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-06-26T07:15:54.9148567Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-06-26T07:15:54.9154914Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-06-26T07:15:54.9157142Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12.1) 2024-06-26T07:15:54.9160322Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-06-26T07:15:54.9162621Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-06-26T07:15:54.9165749Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.2.0) 2024-06-26T07:15:54.9587331Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-06-26T07:15:54.9741034Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-06-26T07:15:54.9815003Z Building wheels for collected packages: torchvision 2024-06-26T07:17:09.1220186Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2024-06-26T07:17:09.1247322Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2113062 sha256=fb2de3440ea96ced0dcbeb43c74dd1ecec51f7657390a32fb4af07ce001f762e 2024-06-26T07:17:09.1249392Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-06-26T07:17:09.1284584Z Successfully built torchvision 2024-06-26T07:17:10.3291993Z Installing collected packages: torchvision 2024-06-26T07:17:10.7157331Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-06-26T07:17:10.8255062Z + '[' -n '' ']' 2024-06-26T07:17:10.8256551Z + id=0 2024-06-26T07:17:10.8257114Z + pip_install opencv-python==4.8.0.74 2024-06-26T07:17:10.8258066Z + pip install --progress-bar off opencv-python==4.8.0.74 2024-06-26T07:17:11.2541367Z Collecting opencv-python==4.8.0.74 2024-06-26T07:17:11.2734918Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB) 2024-06-26T07:17:11.2915212Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.21.2) 2024-06-26T07:17:11.2992049Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB) 2024-06-26T07:17:13.0584366Z Installing collected packages: opencv-python 2024-06-26T07:17:13.8810184Z Successfully installed opencv-python-4.8.0.74 2024-06-26T07:17:13.9787400Z + [[ inductor_torchbench == *inductor_torchbench_smoketest_perf* ]] 2024-06-26T07:17:13.9788154Z + [[ inductor_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]] 2024-06-26T07:17:13.9788927Z + [[ inductor_torchbench == *torchbench_gcp_smoketest* ]] 2024-06-26T07:17:13.9789496Z + checkout_install_torchbench 2024-06-26T07:17:13.9792367Z + local commit 2024-06-26T07:17:13.9792849Z ++ get_pinned_commit torchbench 2024-06-26T07:17:13.9793297Z ++ cat .github/ci_commit_pins/torchbench.txt 2024-06-26T07:17:13.9802174Z + commit=23512dbebd44a11eb84afbf53c3c071dd105297e 2024-06-26T07:17:13.9802989Z + git clone https://github.com/pytorch/benchmark torchbench 2024-06-26T07:17:13.9813759Z Cloning into 'torchbench'... 2024-06-26T07:17:14.1135512Z remote: Enumerating objects: 30187, done. 2024-06-26T07:17:14.1137277Z remote: Counting objects: 0% (1/546) 2024-06-26T07:17:14.1138285Z remote: Counting objects: 1% (6/546) 2024-06-26T07:17:14.1138853Z remote: Counting objects: 2% (11/546) 2024-06-26T07:17:14.1139358Z remote: Counting objects: 3% (17/546) 2024-06-26T07:17:14.1139858Z remote: Counting objects: 4% (22/546) 2024-06-26T07:17:14.1140507Z remote: Counting objects: 5% (28/546) 2024-06-26T07:17:14.1141076Z remote: Counting objects: 6% (33/546) 2024-06-26T07:17:14.1141710Z remote: Counting objects: 7% (39/546) 2024-06-26T07:17:14.1142362Z remote: Counting objects: 8% (44/546) 2024-06-26T07:17:14.1142925Z remote: Counting objects: 9% (50/546) 2024-06-26T07:17:14.1143467Z remote: Counting objects: 10% (55/546) 2024-06-26T07:17:14.1143960Z remote: Counting objects: 11% (61/546) 2024-06-26T07:17:14.1144590Z remote: Counting objects: 12% (66/546) 2024-06-26T07:17:14.1145094Z remote: Counting objects: 13% (71/546) 2024-06-26T07:17:14.1145579Z remote: Counting objects: 14% (77/546) 2024-06-26T07:17:14.1146077Z remote: Counting objects: 15% (82/546) 2024-06-26T07:17:14.1146556Z remote: Counting objects: 16% (88/546) 2024-06-26T07:17:14.1147040Z remote: Counting objects: 17% (93/546) 2024-06-26T07:17:14.1147535Z remote: Counting objects: 18% (99/546) 2024-06-26T07:17:14.1148026Z remote: Counting objects: 19% (104/546) 2024-06-26T07:17:14.1148521Z remote: Counting objects: 20% (110/546) 2024-06-26T07:17:14.1149047Z remote: Counting objects: 21% (115/546) 2024-06-26T07:17:14.1149575Z remote: Counting objects: 22% (121/546) 2024-06-26T07:17:14.1150076Z remote: Counting objects: 23% (126/546) 2024-06-26T07:17:14.1150575Z remote: Counting objects: 24% (132/546) 2024-06-26T07:17:14.1151074Z remote: Counting objects: 25% (137/546) 2024-06-26T07:17:14.1151666Z remote: Counting objects: 26% (142/546) 2024-06-26T07:17:14.1152171Z remote: Counting objects: 27% (148/546) 2024-06-26T07:17:14.1152661Z remote: Counting objects: 28% (153/546) 2024-06-26T07:17:14.1153161Z remote: Counting objects: 29% (159/546) 2024-06-26T07:17:14.1153656Z remote: Counting objects: 30% (164/546) 2024-06-26T07:17:14.1154144Z remote: Counting objects: 31% (170/546) 2024-06-26T07:17:14.1157182Z remote: Counting objects: 32% (175/546) 2024-06-26T07:17:14.1157871Z remote: Counting objects: 33% (181/546) 2024-06-26T07:17:14.1158525Z remote: Counting objects: 34% (186/546) 2024-06-26T07:17:14.1159029Z remote: Counting objects: 35% (192/546) 2024-06-26T07:17:14.1159511Z remote: Counting objects: 36% (197/546) 2024-06-26T07:17:14.1159996Z remote: Counting objects: 37% (203/546) 2024-06-26T07:17:14.1160496Z remote: Counting objects: 38% (208/546) 2024-06-26T07:17:14.1160974Z remote: Counting objects: 39% (213/546) 2024-06-26T07:17:14.1161464Z remote: Counting objects: 40% (219/546) 2024-06-26T07:17:14.1161948Z remote: Counting objects: 41% (224/546) 2024-06-26T07:17:14.1162433Z remote: Counting objects: 42% (230/546) 2024-06-26T07:17:14.1162922Z remote: Counting objects: 43% (235/546) 2024-06-26T07:17:14.1163410Z remote: Counting objects: 44% (241/546) 2024-06-26T07:17:14.1163896Z remote: Counting objects: 45% (246/546) 2024-06-26T07:17:14.1164381Z remote: Counting objects: 46% (252/546) 2024-06-26T07:17:14.1164862Z remote: Counting objects: 47% (257/546) 2024-06-26T07:17:14.1165347Z remote: Counting objects: 48% (263/546) 2024-06-26T07:17:14.1165836Z remote: Counting objects: 49% (268/546) 2024-06-26T07:17:14.1166324Z remote: Counting objects: 50% (273/546) 2024-06-26T07:17:14.1166805Z remote: Counting objects: 51% (279/546) 2024-06-26T07:17:14.1167453Z remote: Counting objects: 52% (284/546) 2024-06-26T07:17:14.1167941Z remote: Counting objects: 53% (290/546) 2024-06-26T07:17:14.1168428Z remote: Counting objects: 54% (295/546) 2024-06-26T07:17:14.1169059Z remote: Counting objects: 55% (301/546) 2024-06-26T07:17:14.1169549Z remote: Counting objects: 56% (306/546) 2024-06-26T07:17:14.1170135Z remote: Counting objects: 57% (312/546) 2024-06-26T07:17:14.1170620Z remote: Counting objects: 58% (317/546) 2024-06-26T07:17:14.1171108Z remote: Counting objects: 59% (323/546) 2024-06-26T07:17:14.1171596Z remote: Counting objects: 60% (328/546) 2024-06-26T07:17:14.1172090Z remote: Counting objects: 61% (334/546) 2024-06-26T07:17:14.1172570Z remote: Counting objects: 62% (339/546) 2024-06-26T07:17:14.1173058Z remote: Counting objects: 63% (344/546) 2024-06-26T07:17:14.1173542Z remote: Counting objects: 64% (350/546) 2024-06-26T07:17:14.1174027Z remote: Counting objects: 65% (355/546) 2024-06-26T07:17:14.1174515Z remote: Counting objects: 66% (361/546) 2024-06-26T07:17:14.1175000Z remote: Counting objects: 67% (366/546) 2024-06-26T07:17:14.1175485Z remote: Counting objects: 68% (372/546) 2024-06-26T07:17:14.1175971Z remote: Counting objects: 69% (377/546) 2024-06-26T07:17:14.1176452Z remote: Counting objects: 70% (383/546) 2024-06-26T07:17:14.1176943Z remote: Counting objects: 71% (388/546) 2024-06-26T07:17:14.1177432Z remote: Counting objects: 72% (394/546) 2024-06-26T07:17:14.1178292Z remote: Counting objects: 73% (399/546) 2024-06-26T07:17:14.1178788Z remote: Counting objects: 74% (405/546) 2024-06-26T07:17:14.1179280Z remote: Counting objects: 75% (410/546) 2024-06-26T07:17:14.1179770Z remote: Counting objects: 76% (415/546) 2024-06-26T07:17:14.1180250Z remote: Counting objects: 77% (421/546) 2024-06-26T07:17:14.1180738Z remote: Counting objects: 78% (426/546) 2024-06-26T07:17:14.1181224Z remote: Counting objects: 79% (432/546) 2024-06-26T07:17:14.1181715Z remote: Counting objects: 80% (437/546) 2024-06-26T07:17:14.1182196Z remote: Counting objects: 81% (443/546) 2024-06-26T07:17:14.1182682Z remote: Counting objects: 82% (448/546) 2024-06-26T07:17:14.1183169Z remote: Counting objects: 83% (454/546) 2024-06-26T07:17:14.1183657Z remote: Counting objects: 84% (459/546) 2024-06-26T07:17:14.1184143Z remote: Counting objects: 85% (465/546) 2024-06-26T07:17:14.1184630Z remote: Counting objects: 86% (470/546) 2024-06-26T07:17:14.1185117Z remote: Counting objects: 87% (476/546) 2024-06-26T07:17:14.1185604Z remote: Counting objects: 88% (481/546) 2024-06-26T07:17:14.1186083Z remote: Counting objects: 89% (486/546) 2024-06-26T07:17:14.1186570Z remote: Counting objects: 90% (492/546) 2024-06-26T07:17:14.1187055Z remote: Counting objects: 91% (497/546) 2024-06-26T07:17:14.1187542Z remote: Counting objects: 92% (503/546) 2024-06-26T07:17:14.1188022Z remote: Counting objects: 93% (508/546) 2024-06-26T07:17:14.1188514Z remote: Counting objects: 94% (514/546) 2024-06-26T07:17:14.1189006Z remote: Counting objects: 95% (519/546) 2024-06-26T07:17:14.1189531Z remote: Counting objects: 96% (525/546) 2024-06-26T07:17:14.1190018Z remote: Counting objects: 97% (530/546) 2024-06-26T07:17:14.1190504Z remote: Counting objects: 98% (536/546) 2024-06-26T07:17:14.1190996Z remote: Counting objects: 99% (541/546) 2024-06-26T07:17:14.1191482Z remote: Counting objects: 100% (546/546) 2024-06-26T07:17:14.1192001Z remote: Counting objects: 100% (546/546), done. 2024-06-26T07:17:14.1192545Z remote: Compressing objects: 0% (1/319) 2024-06-26T07:17:14.1193063Z remote: Compressing objects: 1% (4/319) 2024-06-26T07:17:14.1193569Z remote: Compressing objects: 2% (7/319) 2024-06-26T07:17:14.1194085Z remote: Compressing objects: 3% (10/319) 2024-06-26T07:17:14.1194610Z remote: Compressing objects: 4% (13/319) 2024-06-26T07:17:14.1195125Z remote: Compressing objects: 5% (16/319) 2024-06-26T07:17:14.1195742Z remote: Compressing objects: 6% (20/319) 2024-06-26T07:17:14.1196490Z remote: Compressing objects: 7% (23/319) 2024-06-26T07:17:14.1197015Z remote: Compressing objects: 8% (26/319) 2024-06-26T07:17:14.1198738Z remote: Compressing objects: 9% (29/319) 2024-06-26T07:17:14.1203557Z remote: Compressing objects: 10% (32/319) 2024-06-26T07:17:14.1205944Z remote: Compressing objects: 11% (36/319) 2024-06-26T07:17:14.1208413Z remote: Compressing objects: 12% (39/319) 2024-06-26T07:17:14.1211202Z remote: Compressing objects: 13% (42/319) 2024-06-26T07:17:14.1211798Z remote: Compressing objects: 14% (45/319) 2024-06-26T07:17:14.1212450Z remote: Compressing objects: 15% (48/319) 2024-06-26T07:17:14.1212966Z remote: Compressing objects: 16% (52/319) 2024-06-26T07:17:14.1213483Z remote: Compressing objects: 17% (55/319) 2024-06-26T07:17:14.1214013Z remote: Compressing objects: 18% (58/319) 2024-06-26T07:17:14.1214679Z remote: Compressing objects: 19% (61/319) 2024-06-26T07:17:14.1215368Z remote: Compressing objects: 20% (64/319) 2024-06-26T07:17:14.1215889Z remote: Compressing objects: 21% (67/319) 2024-06-26T07:17:14.1216402Z remote: Compressing objects: 22% (71/319) 2024-06-26T07:17:14.1217004Z remote: Compressing objects: 23% (74/319) 2024-06-26T07:17:14.1217863Z remote: Compressing objects: 24% (77/319) 2024-06-26T07:17:14.1218481Z remote: Compressing objects: 25% (80/319) 2024-06-26T07:17:14.1219009Z remote: Compressing objects: 26% (83/319) 2024-06-26T07:17:14.1219596Z remote: Compressing objects: 27% (87/319) 2024-06-26T07:17:14.1220209Z remote: Compressing objects: 28% (90/319) 2024-06-26T07:17:14.1220746Z remote: Compressing objects: 29% (93/319) 2024-06-26T07:17:14.1221264Z remote: Compressing objects: 30% (96/319) 2024-06-26T07:17:14.1221776Z remote: Compressing objects: 31% (99/319) 2024-06-26T07:17:14.1222450Z remote: Compressing objects: 32% (103/319) 2024-06-26T07:17:14.1222999Z remote: Compressing objects: 33% (106/319) 2024-06-26T07:17:14.1223526Z remote: Compressing objects: 34% (109/319) 2024-06-26T07:17:14.1224054Z remote: Compressing objects: 35% (112/319) 2024-06-26T07:17:14.1224697Z remote: Compressing objects: 36% (115/319) 2024-06-26T07:17:14.1225378Z remote: Compressing objects: 37% (119/319) 2024-06-26T07:17:14.1225912Z remote: Compressing objects: 38% (122/319) 2024-06-26T07:17:14.1226438Z remote: Compressing objects: 39% (125/319) 2024-06-26T07:17:14.1227000Z remote: Compressing objects: 40% (128/319) 2024-06-26T07:17:14.1227748Z remote: Compressing objects: 41% (131/319) 2024-06-26T07:17:14.1228337Z remote: Compressing objects: 42% (134/319) 2024-06-26T07:17:14.1228856Z remote: Compressing objects: 43% (138/319) 2024-06-26T07:17:14.1229427Z remote: Compressing objects: 44% (141/319) 2024-06-26T07:17:14.1230047Z remote: Compressing objects: 45% (144/319) 2024-06-26T07:17:14.1230575Z remote: Compressing objects: 46% (147/319) 2024-06-26T07:17:14.1231097Z remote: Compressing objects: 47% (150/319) 2024-06-26T07:17:14.1231617Z remote: Compressing objects: 48% (154/319) 2024-06-26T07:17:14.1232167Z remote: Compressing objects: 49% (157/319) 2024-06-26T07:17:14.1232759Z remote: Compressing objects: 50% (160/319) 2024-06-26T07:17:14.1233283Z remote: Compressing objects: 51% (163/319) 2024-06-26T07:17:14.1233810Z remote: Compressing objects: 52% (166/319) 2024-06-26T07:17:14.1234332Z remote: Compressing objects: 53% (170/319) 2024-06-26T07:17:14.1234933Z remote: Compressing objects: 54% (173/319) 2024-06-26T07:17:14.1235468Z remote: Compressing objects: 55% (176/319) 2024-06-26T07:17:14.1235988Z remote: Compressing objects: 56% (179/319) 2024-06-26T07:17:14.1236507Z remote: Compressing objects: 57% (182/319) 2024-06-26T07:17:14.1237019Z remote: Compressing objects: 58% (186/319) 2024-06-26T07:17:14.1237605Z remote: Compressing objects: 59% (189/319) 2024-06-26T07:17:14.1238264Z remote: Compressing objects: 60% (192/319) 2024-06-26T07:17:14.1238785Z remote: Compressing objects: 61% (195/319) 2024-06-26T07:17:14.1239300Z remote: Compressing objects: 62% (198/319) 2024-06-26T07:17:14.1239918Z remote: Compressing objects: 63% (201/319) 2024-06-26T07:17:14.1240445Z remote: Compressing objects: 64% (205/319) 2024-06-26T07:17:14.1241047Z remote: Compressing objects: 65% (208/319) 2024-06-26T07:17:14.1241563Z remote: Compressing objects: 66% (211/319) 2024-06-26T07:17:14.1242088Z remote: Compressing objects: 67% (214/319) 2024-06-26T07:17:14.1242718Z remote: Compressing objects: 68% (217/319) 2024-06-26T07:17:14.1243239Z remote: Compressing objects: 69% (221/319) 2024-06-26T07:17:14.1243751Z remote: Compressing objects: 70% (224/319) 2024-06-26T07:17:14.1244271Z remote: Compressing objects: 71% (227/319) 2024-06-26T07:17:14.1244851Z remote: Compressing objects: 72% (230/319) 2024-06-26T07:17:14.1245459Z remote: Compressing objects: 73% (233/319) 2024-06-26T07:17:14.1245986Z remote: Compressing objects: 74% (237/319) 2024-06-26T07:17:14.1246504Z remote: Compressing objects: 75% (240/319) 2024-06-26T07:17:14.1247024Z remote: Compressing objects: 76% (243/319) 2024-06-26T07:17:14.1247534Z remote: Compressing objects: 77% (246/319) 2024-06-26T07:17:14.1248160Z remote: Compressing objects: 78% (249/319) 2024-06-26T07:17:14.1248806Z remote: Compressing objects: 79% (253/319) 2024-06-26T07:17:14.1249334Z remote: Compressing objects: 80% (256/319) 2024-06-26T07:17:14.1265817Z remote: Compressing objects: 81% (259/319) 2024-06-26T07:17:14.1266487Z remote: Compressing objects: 82% (262/319) 2024-06-26T07:17:14.1267021Z remote: Compressing objects: 83% (265/319) 2024-06-26T07:17:14.1267554Z remote: Compressing objects: 84% (268/319) 2024-06-26T07:17:14.1268075Z remote: Compressing objects: 85% (272/319) 2024-06-26T07:17:14.1268603Z remote: Compressing objects: 86% (275/319) 2024-06-26T07:17:14.1269142Z remote: Compressing objects: 87% (278/319) 2024-06-26T07:17:14.1269663Z remote: Compressing objects: 88% (281/319) 2024-06-26T07:17:14.1270181Z remote: Compressing objects: 89% (284/319) 2024-06-26T07:17:14.1270705Z remote: Compressing objects: 90% (288/319) 2024-06-26T07:17:14.1271230Z remote: Compressing objects: 91% (291/319) 2024-06-26T07:17:14.1271759Z remote: Compressing objects: 92% (294/319) 2024-06-26T07:17:14.1272279Z remote: Compressing objects: 93% (297/319) 2024-06-26T07:17:14.1272803Z remote: Compressing objects: 94% (300/319) 2024-06-26T07:17:14.1273335Z remote: Compressing objects: 95% (304/319) 2024-06-26T07:17:14.1273854Z remote: Compressing objects: 96% (307/319) 2024-06-26T07:17:14.1274374Z remote: Compressing objects: 97% (310/319) 2024-06-26T07:17:14.1274895Z remote: Compressing objects: 98% (313/319) 2024-06-26T07:17:14.1275412Z remote: Compressing objects: 99% (316/319) 2024-06-26T07:17:14.1275932Z remote: Compressing objects: 100% (319/319) 2024-06-26T07:17:14.1276490Z remote: Compressing objects: 100% (319/319), done. 2024-06-26T07:17:14.1310028Z Receiving objects: 0% (1/30187) 2024-06-26T07:17:14.1346720Z Receiving objects: 1% (302/30187) 2024-06-26T07:17:14.1386385Z Receiving objects: 2% (604/30187) 2024-06-26T07:17:14.1420527Z Receiving objects: 3% (906/30187) 2024-06-26T07:17:14.1450804Z Receiving objects: 4% (1208/30187) 2024-06-26T07:17:14.1486360Z Receiving objects: 5% (1510/30187) 2024-06-26T07:17:14.1520556Z Receiving objects: 6% (1812/30187) 2024-06-26T07:17:14.1562799Z Receiving objects: 7% (2114/30187) 2024-06-26T07:17:14.1595398Z Receiving objects: 8% (2415/30187) 2024-06-26T07:17:14.1625901Z Receiving objects: 9% (2717/30187) 2024-06-26T07:17:14.1661790Z Receiving objects: 10% (3019/30187) 2024-06-26T07:17:14.1723719Z Receiving objects: 11% (3321/30187) 2024-06-26T07:17:14.1786572Z Receiving objects: 12% (3623/30187) 2024-06-26T07:17:14.1843027Z Receiving objects: 13% (3925/30187) 2024-06-26T07:17:14.2012394Z Receiving objects: 14% (4227/30187) 2024-06-26T07:17:14.2114732Z Receiving objects: 15% (4529/30187) 2024-06-26T07:17:14.2211858Z Receiving objects: 16% (4830/30187) 2024-06-26T07:17:14.2264162Z Receiving objects: 17% (5132/30187) 2024-06-26T07:17:14.2305203Z Receiving objects: 18% (5434/30187) 2024-06-26T07:17:14.2330564Z Receiving objects: 19% (5736/30187) 2024-06-26T07:17:14.2393245Z Receiving objects: 20% (6038/30187) 2024-06-26T07:17:14.2412844Z Receiving objects: 21% (6340/30187) 2024-06-26T07:17:14.2433032Z Receiving objects: 22% (6642/30187) 2024-06-26T07:17:14.3493106Z Receiving objects: 23% (6944/30187) 2024-06-26T07:17:15.1264903Z Receiving objects: 24% (7245/30187) 2024-06-26T07:17:15.2312335Z Receiving objects: 24% (7430/30187), 80.19 MiB | 80.27 MiB/s 2024-06-26T07:17:15.7143938Z Receiving objects: 25% (7547/30187), 80.19 MiB | 80.27 MiB/s 2024-06-26T07:17:15.7808137Z Receiving objects: 26% (7849/30187), 118.80 MiB | 79.25 MiB/s 2024-06-26T07:17:15.8472090Z Receiving objects: 27% (8151/30187), 118.80 MiB | 79.25 MiB/s 2024-06-26T07:17:15.9128044Z Receiving objects: 28% (8453/30187), 118.80 MiB | 79.25 MiB/s 2024-06-26T07:17:15.9786704Z Receiving objects: 29% (8755/30187), 118.80 MiB | 79.25 MiB/s 2024-06-26T07:17:16.0444541Z Receiving objects: 30% (9057/30187), 118.80 MiB | 79.25 MiB/s 2024-06-26T07:17:16.1103312Z Receiving objects: 31% (9358/30187), 118.80 MiB | 79.25 MiB/s 2024-06-26T07:17:16.1264743Z Receiving objects: 32% (9660/30187), 118.80 MiB | 79.25 MiB/s 2024-06-26T07:17:16.1765929Z Receiving objects: 32% (9734/30187), 161.00 MiB | 80.54 MiB/s 2024-06-26T07:17:16.3978259Z Receiving objects: 33% (9962/30187), 161.00 MiB | 80.54 MiB/s 2024-06-26T07:17:16.4702328Z Receiving objects: 34% (10264/30187), 161.00 MiB | 80.54 MiB/s 2024-06-26T07:17:16.4824005Z Receiving objects: 35% (10566/30187), 161.00 MiB | 80.54 MiB/s 2024-06-26T07:17:16.6994602Z Receiving objects: 36% (10868/30187), 161.00 MiB | 80.54 MiB/s 2024-06-26T07:17:16.9696029Z Receiving objects: 37% (11170/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:16.9899138Z Receiving objects: 38% (11472/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:16.9912694Z Receiving objects: 39% (11773/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:16.9939180Z Receiving objects: 40% (12075/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:16.9952420Z Receiving objects: 41% (12377/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:16.9965251Z Receiving objects: 42% (12679/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:16.9975808Z Receiving objects: 43% (12981/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:16.9985742Z Receiving objects: 44% (13283/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.0004320Z Receiving objects: 45% (13585/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.0015535Z Receiving objects: 46% (13887/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.0020272Z Receiving objects: 47% (14188/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.0025787Z Receiving objects: 48% (14490/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.0046084Z Receiving objects: 49% (14792/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.0062708Z Receiving objects: 50% (15094/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.0072424Z Receiving objects: 51% (15396/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.0268098Z Receiving objects: 52% (15698/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.0272424Z Receiving objects: 53% (16000/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.0278968Z Receiving objects: 54% (16301/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.0284595Z Receiving objects: 55% (16603/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.0289914Z Receiving objects: 56% (16905/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.0306527Z Receiving objects: 57% (17207/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.0316223Z Receiving objects: 58% (17509/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.0324707Z Receiving objects: 59% (17811/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.0338004Z Receiving objects: 60% (18113/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.0346323Z Receiving objects: 61% (18415/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.0355537Z Receiving objects: 62% (18716/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.0482507Z Receiving objects: 63% (19018/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.0488091Z Receiving objects: 64% (19320/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.0494285Z Receiving objects: 65% (19622/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.0500492Z Receiving objects: 66% (19924/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.0509163Z Receiving objects: 67% (20226/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.0516660Z Receiving objects: 68% (20528/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.0758627Z Receiving objects: 69% (20830/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.1146892Z Receiving objects: 70% (21131/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.1152923Z Receiving objects: 71% (21433/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.1163725Z Receiving objects: 72% (21735/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.1176826Z Receiving objects: 73% (22037/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.1188911Z Receiving objects: 74% (22339/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.1198779Z Receiving objects: 75% (22641/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.1207051Z Receiving objects: 76% (22943/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.1225141Z Receiving objects: 77% (23244/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.1230921Z Receiving objects: 78% (23546/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.1238768Z Receiving objects: 79% (23848/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.1260660Z Receiving objects: 80% (24150/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.1265258Z Receiving objects: 81% (24452/30187), 192.63 MiB | 77.08 MiB/s 2024-06-26T07:17:17.3194296Z Receiving objects: 81% (24456/30187), 229.05 MiB | 76.37 MiB/s 2024-06-26T07:17:17.3403808Z Receiving objects: 82% (24754/30187), 229.05 MiB | 76.37 MiB/s 2024-06-26T07:17:17.3435984Z Receiving objects: 83% (25056/30187), 229.05 MiB | 76.37 MiB/s 2024-06-26T07:17:17.3445556Z Receiving objects: 84% (25358/30187), 229.05 MiB | 76.37 MiB/s 2024-06-26T07:17:17.3456004Z Receiving objects: 85% (25659/30187), 229.05 MiB | 76.37 MiB/s 2024-06-26T07:17:17.3485248Z Receiving objects: 86% (25961/30187), 229.05 MiB | 76.37 MiB/s 2024-06-26T07:17:17.3513768Z Receiving objects: 87% (26263/30187), 229.05 MiB | 76.37 MiB/s 2024-06-26T07:17:17.3640913Z Receiving objects: 88% (26565/30187), 229.05 MiB | 76.37 MiB/s 2024-06-26T07:17:17.3646671Z Receiving objects: 89% (26867/30187), 229.05 MiB | 76.37 MiB/s 2024-06-26T07:17:17.3653956Z Receiving objects: 90% (27169/30187), 229.05 MiB | 76.37 MiB/s 2024-06-26T07:17:17.3659219Z Receiving objects: 91% (27471/30187), 229.05 MiB | 76.37 MiB/s 2024-06-26T07:17:17.3665659Z Receiving objects: 92% (27773/30187), 229.05 MiB | 76.37 MiB/s 2024-06-26T07:17:17.3671397Z Receiving objects: 93% (28074/30187), 229.05 MiB | 76.37 MiB/s 2024-06-26T07:17:17.3782055Z Receiving objects: 94% (28376/30187), 229.05 MiB | 76.37 MiB/s 2024-06-26T07:17:17.3796076Z Receiving objects: 95% (28678/30187), 229.05 MiB | 76.37 MiB/s 2024-06-26T07:17:17.3806073Z Receiving objects: 96% (28980/30187), 229.05 MiB | 76.37 MiB/s 2024-06-26T07:17:17.3823967Z Receiving objects: 97% (29282/30187), 229.05 MiB | 76.37 MiB/s 2024-06-26T07:17:17.3838159Z Receiving objects: 98% (29584/30187), 229.05 MiB | 76.37 MiB/s 2024-06-26T07:17:17.3878234Z Receiving objects: 99% (29886/30187), 229.05 MiB | 76.37 MiB/s 2024-06-26T07:17:17.3879271Z remote: Total 30187 (delta 255), reused 486 (delta 215), pack-reused 29641 2024-06-26T07:17:17.3895678Z Receiving objects: 100% (30187/30187), 229.05 MiB | 76.37 MiB/s 2024-06-26T07:17:17.3896648Z Receiving objects: 100% (30187/30187), 245.25 MiB | 75.17 MiB/s, done. 2024-06-26T07:17:17.3922726Z Resolving deltas: 0% (0/16204) 2024-06-26T07:17:17.3931892Z Resolving deltas: 1% (164/16204) 2024-06-26T07:17:17.3947257Z Resolving deltas: 2% (327/16204) 2024-06-26T07:17:17.3961801Z Resolving deltas: 3% (510/16204) 2024-06-26T07:17:17.3983688Z Resolving deltas: 4% (649/16204) 2024-06-26T07:17:17.3993439Z Resolving deltas: 5% (824/16204) 2024-06-26T07:17:17.4002531Z Resolving deltas: 6% (973/16204) 2024-06-26T07:17:17.4013268Z Resolving deltas: 7% (1144/16204) 2024-06-26T07:17:17.4023441Z Resolving deltas: 8% (1299/16204) 2024-06-26T07:17:17.4032139Z Resolving deltas: 9% (1484/16204) 2024-06-26T07:17:17.4043451Z Resolving deltas: 10% (1631/16204) 2024-06-26T07:17:17.4054602Z Resolving deltas: 11% (1783/16204) 2024-06-26T07:17:17.4063565Z Resolving deltas: 12% (1948/16204) 2024-06-26T07:17:17.4076740Z Resolving deltas: 13% (2108/16204) 2024-06-26T07:17:17.4086038Z Resolving deltas: 14% (2270/16204) 2024-06-26T07:17:17.4104706Z Resolving deltas: 15% (2440/16204) 2024-06-26T07:17:17.4121248Z Resolving deltas: 16% (2618/16204) 2024-06-26T07:17:17.4135991Z Resolving deltas: 17% (2762/16204) 2024-06-26T07:17:17.4165292Z Resolving deltas: 18% (2920/16204) 2024-06-26T07:17:17.4192759Z Resolving deltas: 19% (3088/16204) 2024-06-26T07:17:17.4213218Z Resolving deltas: 20% (3241/16204) 2024-06-26T07:17:17.4213895Z Resolving deltas: 21% (3535/16204) 2024-06-26T07:17:17.4222834Z Resolving deltas: 22% (3565/16204) 2024-06-26T07:17:17.4238393Z Resolving deltas: 23% (3728/16204) 2024-06-26T07:17:17.4241445Z Resolving deltas: 24% (4020/16204) 2024-06-26T07:17:17.4253783Z Resolving deltas: 25% (4074/16204) 2024-06-26T07:17:17.4290993Z Resolving deltas: 26% (4224/16204) 2024-06-26T07:17:17.4309455Z Resolving deltas: 27% (4390/16204) 2024-06-26T07:17:17.4322602Z Resolving deltas: 28% (4560/16204) 2024-06-26T07:17:17.4335930Z Resolving deltas: 29% (4703/16204) 2024-06-26T07:17:17.4354889Z Resolving deltas: 30% (4866/16204) 2024-06-26T07:17:17.4366452Z Resolving deltas: 31% (5066/16204) 2024-06-26T07:17:17.4375994Z Resolving deltas: 32% (5199/16204) 2024-06-26T07:17:17.4385964Z Resolving deltas: 33% (5372/16204) 2024-06-26T07:17:17.4396359Z Resolving deltas: 34% (5518/16204) 2024-06-26T07:17:17.4416590Z Resolving deltas: 35% (5673/16204) 2024-06-26T07:17:17.4431838Z Resolving deltas: 36% (5839/16204) 2024-06-26T07:17:17.4448718Z Resolving deltas: 37% (5998/16204) 2024-06-26T07:17:17.4455766Z Resolving deltas: 38% (6163/16204) 2024-06-26T07:17:17.4476851Z Resolving deltas: 39% (6327/16204) 2024-06-26T07:17:17.4487600Z Resolving deltas: 40% (6490/16204) 2024-06-26T07:17:17.4494855Z Resolving deltas: 41% (6692/16204) 2024-06-26T07:17:17.4516193Z Resolving deltas: 42% (6830/16204) 2024-06-26T07:17:17.4521048Z Resolving deltas: 43% (7024/16204) 2024-06-26T07:17:17.4528178Z Resolving deltas: 44% (7147/16204) 2024-06-26T07:17:17.4539364Z Resolving deltas: 45% (7296/16204) 2024-06-26T07:17:17.4563088Z Resolving deltas: 46% (7465/16204) 2024-06-26T07:17:17.4580597Z Resolving deltas: 47% (7623/16204) 2024-06-26T07:17:17.4590286Z Resolving deltas: 48% (7788/16204) 2024-06-26T07:17:17.4611674Z Resolving deltas: 49% (7940/16204) 2024-06-26T07:17:17.4623706Z Resolving deltas: 50% (8115/16204) 2024-06-26T07:17:17.4630146Z Resolving deltas: 51% (8266/16204) 2024-06-26T07:17:17.4640835Z Resolving deltas: 52% (8433/16204) 2024-06-26T07:17:17.4659507Z Resolving deltas: 53% (8592/16204) 2024-06-26T07:17:17.4664339Z Resolving deltas: 54% (8835/16204) 2024-06-26T07:17:17.4670037Z Resolving deltas: 55% (8943/16204) 2024-06-26T07:17:17.4678062Z Resolving deltas: 56% (9081/16204) 2024-06-26T07:17:17.4692906Z Resolving deltas: 57% (9242/16204) 2024-06-26T07:17:17.4700940Z Resolving deltas: 58% (9400/16204) 2024-06-26T07:17:17.4710823Z Resolving deltas: 59% (9613/16204) 2024-06-26T07:17:17.4718938Z Resolving deltas: 61% (9885/16204) 2024-06-26T07:17:17.4731189Z Resolving deltas: 62% (10057/16204) 2024-06-26T07:17:17.4742487Z Resolving deltas: 63% (10320/16204) 2024-06-26T07:17:17.4752676Z Resolving deltas: 65% (10533/16204) 2024-06-26T07:17:17.4760881Z Resolving deltas: 66% (10761/16204) 2024-06-26T07:17:17.4780899Z Resolving deltas: 67% (10941/16204) 2024-06-26T07:17:17.4788578Z Resolving deltas: 69% (11239/16204) 2024-06-26T07:17:17.4807212Z Resolving deltas: 70% (11343/16204) 2024-06-26T07:17:17.4814309Z Resolving deltas: 71% (11505/16204) 2024-06-26T07:17:17.4825911Z Resolving deltas: 72% (11674/16204) 2024-06-26T07:17:17.4840801Z Resolving deltas: 73% (11829/16204) 2024-06-26T07:17:17.4854036Z Resolving deltas: 74% (11993/16204) 2024-06-26T07:17:17.4886651Z Resolving deltas: 75% (12176/16204) 2024-06-26T07:17:17.4891014Z Resolving deltas: 78% (12715/16204) 2024-06-26T07:17:17.4902786Z Resolving deltas: 79% (12802/16204) 2024-06-26T07:17:17.4910800Z Resolving deltas: 80% (12968/16204) 2024-06-26T07:17:17.4921599Z Resolving deltas: 81% (13134/16204) 2024-06-26T07:17:17.4930396Z Resolving deltas: 82% (13291/16204) 2024-06-26T07:17:17.4938085Z Resolving deltas: 83% (13452/16204) 2024-06-26T07:17:17.4948733Z Resolving deltas: 84% (13619/16204) 2024-06-26T07:17:17.4968998Z Resolving deltas: 85% (13775/16204) 2024-06-26T07:17:17.4979167Z Resolving deltas: 86% (13936/16204) 2024-06-26T07:17:17.4986112Z Resolving deltas: 87% (14106/16204) 2024-06-26T07:17:17.4994056Z Resolving deltas: 88% (14265/16204) 2024-06-26T07:17:17.5001060Z Resolving deltas: 89% (14440/16204) 2024-06-26T07:17:17.5011402Z Resolving deltas: 90% (14590/16204) 2024-06-26T07:17:17.5018908Z Resolving deltas: 91% (14763/16204) 2024-06-26T07:17:17.5030358Z Resolving deltas: 92% (14917/16204) 2024-06-26T07:17:17.5034631Z Resolving deltas: 93% (15126/16204) 2024-06-26T07:17:17.5042464Z Resolving deltas: 94% (15253/16204) 2024-06-26T07:17:17.5050680Z Resolving deltas: 95% (15402/16204) 2024-06-26T07:17:17.5057746Z Resolving deltas: 96% (15561/16204) 2024-06-26T07:17:17.5063921Z Resolving deltas: 97% (15718/16204) 2024-06-26T07:17:17.5071626Z Resolving deltas: 98% (15881/16204) 2024-06-26T07:17:17.5104054Z Resolving deltas: 99% (16042/16204) 2024-06-26T07:17:17.5104579Z Resolving deltas: 100% (16204/16204) 2024-06-26T07:17:17.5105034Z Resolving deltas: 100% (16204/16204), done. 2024-06-26T07:17:18.5668079Z + pushd torchbench 2024-06-26T07:17:18.5668503Z ~/workspace/torchbench ~/workspace 2024-06-26T07:17:18.5669041Z + git checkout 23512dbebd44a11eb84afbf53c3c071dd105297e 2024-06-26T07:17:18.6257869Z Note: switching to '23512dbebd44a11eb84afbf53c3c071dd105297e'. 2024-06-26T07:17:18.6258500Z 2024-06-26T07:17:18.6258988Z You are in 'detached HEAD' state. You can look around, make experimental 2024-06-26T07:17:18.6259777Z changes and commit them, and you can discard any commits you make in this 2024-06-26T07:17:18.6260661Z state without impacting any branches by switching back to a branch. 2024-06-26T07:17:18.6261109Z 2024-06-26T07:17:18.6261451Z If you want to create a new branch to retain commits you create, you may 2024-06-26T07:17:18.6262355Z do so (now or later) by using -c with the switch command. Example: 2024-06-26T07:17:18.6262778Z 2024-06-26T07:17:18.6262967Z git switch -c 2024-06-26T07:17:18.6263240Z 2024-06-26T07:17:18.6263377Z Or undo this operation with: 2024-06-26T07:17:18.6263635Z 2024-06-26T07:17:18.6263753Z git switch - 2024-06-26T07:17:18.6263940Z 2024-06-26T07:17:18.6264288Z Turn off this advice by setting config variable advice.detachedHead to false 2024-06-26T07:17:18.6264781Z 2024-06-26T07:17:18.6264953Z HEAD is now at 23512dbe Pin numpy version 2024-06-26T07:17:18.6265401Z + '[' '' ']' 2024-06-26T07:17:18.6265762Z + python install.py --continue_on_fail 2024-06-26T07:17:20.5718244Z checking packages torch, torchvision, torchaudio are installed...OK 2024-06-26T07:17:36.1606499Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK 2024-06-26T07:17:38.5510106Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK 2024-06-26T07:17:41.1166733Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK 2024-06-26T07:17:43.4739612Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK 2024-06-26T07:17:43.4895841Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK 2024-06-26T07:17:52.0955727Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK 2024-06-26T07:17:55.4549971Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK 2024-06-26T07:17:58.8238623Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK 2024-06-26T07:18:02.1768030Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK 2024-06-26T07:18:02.1771687Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found 2024-06-26T07:18:03.9735945Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK 2024-06-26T07:18:07.5299158Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK 2024-06-26T07:18:07.5450767Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK 2024-06-26T07:19:01.8529013Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK 2024-06-26T07:19:10.4979441Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK 2024-06-26T07:19:18.1631629Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK 2024-06-26T07:19:25.4692728Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK 2024-06-26T07:19:33.9472834Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK 2024-06-26T07:19:41.3850469Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK 2024-06-26T07:19:48.2071840Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK 2024-06-26T07:19:55.6014922Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK 2024-06-26T07:20:03.2992831Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK 2024-06-26T07:20:11.0549927Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK 2024-06-26T07:20:18.4478441Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK 2024-06-26T07:20:25.8098292Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK 2024-06-26T07:20:29.7558895Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK 2024-06-26T07:21:09.5215449Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK 2024-06-26T07:21:15.4764204Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK 2024-06-26T07:21:22.0931390Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK 2024-06-26T07:21:32.9152056Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK 2024-06-26T07:21:35.1112288Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK 2024-06-26T07:21:37.5853339Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK 2024-06-26T07:21:45.4506186Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK 2024-06-26T07:21:53.5150957Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK 2024-06-26T07:22:00.1638297Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK 2024-06-26T07:22:08.7615503Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK 2024-06-26T07:22:15.7971458Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK 2024-06-26T07:22:22.0651611Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK 2024-06-26T07:22:29.3517059Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK 2024-06-26T07:22:45.5810427Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK 2024-06-26T07:22:52.8829155Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK 2024-06-26T07:22:58.2041736Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK 2024-06-26T07:23:04.3054419Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK 2024-06-26T07:23:12.0797291Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK 2024-06-26T07:23:12.0799810Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found 2024-06-26T07:23:25.3514824Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK 2024-06-26T07:23:30.5998595Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK 2024-06-26T07:23:30.6000306Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found 2024-06-26T07:23:39.1381917Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK 2024-06-26T07:23:41.4223047Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK 2024-06-26T07:23:43.7245488Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK 2024-06-26T07:24:23.3376156Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK 2024-06-26T07:25:42.9558144Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK 2024-06-26T07:25:42.9719131Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK 2024-06-26T07:25:45.4897821Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK 2024-06-26T07:25:45.5041046Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK 2024-06-26T07:25:45.5197448Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK 2024-06-26T07:25:45.5350954Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK 2024-06-26T07:25:45.5504979Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK 2024-06-26T07:25:45.5655578Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK 2024-06-26T07:26:05.9415638Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK 2024-06-26T07:26:05.9418366Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found 2024-06-26T07:26:08.2418715Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK 2024-06-26T07:26:11.5508062Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK 2024-06-26T07:26:11.5659531Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK 2024-06-26T07:26:11.5805983Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK 2024-06-26T07:26:11.5962893Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK 2024-06-26T07:26:11.6115414Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK 2024-06-26T07:26:11.6266594Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK 2024-06-26T07:26:18.8180902Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK 2024-06-26T07:26:21.3394381Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK 2024-06-26T07:26:25.8754148Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK 2024-06-26T07:26:25.8908565Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK 2024-06-26T07:26:25.9064336Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK 2024-06-26T07:26:25.9215114Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK 2024-06-26T07:26:25.9365522Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK 2024-06-26T07:26:25.9518491Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK 2024-06-26T07:26:37.6272652Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK 2024-06-26T07:26:49.7490497Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK 2024-06-26T07:26:49.7646852Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK 2024-06-26T07:26:49.7650121Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found 2024-06-26T07:26:49.7654166Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found 2024-06-26T07:26:53.1047356Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK 2024-06-26T07:26:55.8889872Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK 2024-06-26T07:26:55.9049045Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK 2024-06-26T07:27:18.8429637Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK 2024-06-26T07:27:25.7628219Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK 2024-06-26T07:27:31.3645641Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK 2024-06-26T07:27:39.7100252Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK 2024-06-26T07:27:39.7256753Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK 2024-06-26T07:27:39.7417182Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK 2024-06-26T07:27:39.7572115Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK 2024-06-26T07:27:39.7728189Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK 2024-06-26T07:27:39.7882287Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK 2024-06-26T07:27:39.8034130Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK 2024-06-26T07:27:39.8188074Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK 2024-06-26T07:27:52.3980732Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK 2024-06-26T07:28:05.1230187Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK 2024-06-26T07:28:05.1383700Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK 2024-06-26T07:28:08.0936949Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK 2024-06-26T07:28:11.3062090Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK 2024-06-26T07:28:11.7221679Z + popd 2024-06-26T07:28:11.7222138Z ~/workspace 2024-06-26T07:28:11.7222514Z + [[ inductor_torchbench != *cpu_inductor* ]] 2024-06-26T07:28:11.7225510Z + install_torchrec_and_fbgemm 2024-06-26T07:28:11.7226099Z + local torchrec_commit 2024-06-26T07:28:11.7226578Z ++ get_pinned_commit torchrec 2024-06-26T07:28:11.7226992Z ++ cat .github/ci_commit_pins/torchrec.txt 2024-06-26T07:28:11.7309752Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T07:28:11.7310420Z + local fbgemm_commit 2024-06-26T07:28:11.7312184Z ++ get_pinned_commit fbgemm 2024-06-26T07:28:11.7312725Z ++ cat .github/ci_commit_pins/fbgemm.txt 2024-06-26T07:28:11.7321801Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:28:11.7322650Z + pip_uninstall torchrec-nightly 2024-06-26T07:28:11.7323113Z + pip uninstall -y torchrec-nightly 2024-06-26T07:28:12.0889771Z WARNING: Skipping torchrec-nightly as it is not installed. 2024-06-26T07:28:12.1301920Z + pip_uninstall fbgemm-gpu-nightly 2024-06-26T07:28:12.1302479Z + pip uninstall -y fbgemm-gpu-nightly 2024-06-26T07:28:12.4840274Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed. 2024-06-26T07:28:12.5262999Z + pip_install setuptools-git-versioning scikit-build pyre-extensions 2024-06-26T07:28:12.5263950Z + pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions 2024-06-26T07:28:12.9740408Z Collecting setuptools-git-versioning 2024-06-26T07:28:12.9902136Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl.metadata (5.8 kB) 2024-06-26T07:28:13.0970314Z Collecting scikit-build 2024-06-26T07:28:13.0998474Z Downloading scikit_build-0.18.0-py3-none-any.whl.metadata (17 kB) 2024-06-26T07:28:13.2059861Z Collecting pyre-extensions 2024-06-26T07:28:13.2090387Z Downloading pyre_extensions-0.0.30-py3-none-any.whl.metadata (4.0 kB) 2024-06-26T07:28:13.2555888Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.1) 2024-06-26T07:28:13.2558383Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (69.5.1) 2024-06-26T07:28:13.3038066Z Collecting toml>=0.10.2 (from setuptools-git-versioning) 2024-06-26T07:28:13.3065676Z Downloading toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB) 2024-06-26T07:28:13.3670551Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.9.0) 2024-06-26T07:28:13.3682563Z Requirement already satisfied: tomli in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (2.0.1) 2024-06-26T07:28:13.3688567Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.43.0) 2024-06-26T07:28:13.4274714Z Collecting typing-inspect (from pyre-extensions) 2024-06-26T07:28:13.4304662Z Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB) 2024-06-26T07:28:13.4789414Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.12.2) 2024-06-26T07:28:13.5012987Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0) 2024-06-26T07:28:13.5058834Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl (10 kB) 2024-06-26T07:28:13.5603416Z Downloading scikit_build-0.18.0-py3-none-any.whl (85 kB) 2024-06-26T07:28:13.6170602Z Downloading pyre_extensions-0.0.30-py3-none-any.whl (12 kB) 2024-06-26T07:28:13.6754428Z Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB) 2024-06-26T07:28:13.7343814Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2024-06-26T07:28:15.9223670Z Installing collected packages: typing-inspect, toml, scikit-build, setuptools-git-versioning, pyre-extensions 2024-06-26T07:28:16.2855771Z Successfully installed pyre-extensions-0.0.30 scikit-build-0.18.0 setuptools-git-versioning-2.0.0 toml-0.10.2 typing-inspect-0.9.0 2024-06-26T07:28:16.4040640Z + CUDA_PATH=/usr/local/cuda-12.1 2024-06-26T07:28:16.4041824Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-06-26T07:28:16.4043667Z + pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-06-26T07:28:16.7394797Z Collecting fbgemm-gpu 2024-06-26T07:28:16.7396911Z Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-lodecyh_/fbgemm-gpu_28967aa158fe411f8bdad4d163ba0978 2024-06-26T07:28:16.7411828Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-lodecyh_/fbgemm-gpu_28967aa158fe411f8bdad4d163ba0978 2024-06-26T07:28:17.7343443Z Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd' 2024-06-26T07:28:17.7357504Z Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:28:18.1284257Z Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:28:18.6128038Z Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:28:18.6129316Z Running command git submodule update --init --recursive -q 2024-06-26T07:28:26.6015532Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:28:26.6037679Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.21.2) 2024-06-26T07:28:26.6043824Z Building wheels for collected packages: fbgemm-gpu 2024-06-26T08:10:48.7831404Z Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done 2024-06-26T08:10:48.9629900Z [?25h Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=252632963 sha256=fc34a69add99592cfcb891f2a24da6ccbab7b5b37463a51105f01beddf718373 2024-06-26T08:10:48.9634606Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39 2024-06-26T08:10:48.9650786Z Successfully built fbgemm-gpu 2024-06-26T08:10:51.1071875Z Installing collected packages: fbgemm-gpu 2024-06-26T08:10:54.3318426Z Successfully installed fbgemm-gpu-0.4.1rc0.post421 2024-06-26T08:10:56.2974982Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:10:56.2976485Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:10:56.6327564Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:10:56.6331226Z Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-gxim2mcq 2024-06-26T08:10:56.6347291Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-gxim2mcq 2024-06-26T08:10:57.2494481Z Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2' 2024-06-26T08:10:57.2510212Z Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:10:57.5226906Z Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:10:57.8491996Z Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:10:58.0757140Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T08:10:58.0807029Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9) 2024-06-26T08:10:58.0808710Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.0.3) 2024-06-26T08:10:58.0812825Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0) 2024-06-26T08:10:58.1330914Z Collecting torchmetrics (from torchrec==0.3.2) 2024-06-26T08:10:58.1508619Z Downloading torchmetrics-1.4.0.post0-py3-none-any.whl.metadata (19 kB) 2024-06-26T08:10:58.1579118Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.66.4) 2024-06-26T08:10:58.1581951Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421) 2024-06-26T08:10:58.1591611Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.21.2) 2024-06-26T08:10:58.1611567Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (2.10.0) 2024-06-26T08:10:58.2264967Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0) 2024-06-26T08:10:58.2269216Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-06-26T08:10:58.2273614Z Requirement already satisfied: tzdata>=2022.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-06-26T08:10:58.3041026Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.1) 2024-06-26T08:10:58.3045524Z Requirement already satisfied: torch>=1.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.5.0a0+gitb8c4c54) 2024-06-26T08:10:58.3230516Z Collecting lightning-utilities>=0.8.0 (from torchmetrics->torchrec==0.3.2) 2024-06-26T08:10:58.3298090Z Downloading lightning_utilities-0.11.2-py3-none-any.whl.metadata (4.7 kB) 2024-06-26T08:10:58.3459554Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (69.5.1) 2024-06-26T08:10:58.3461970Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.12.2) 2024-06-26T08:10:58.3493118Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.16.0) 2024-06-26T08:10:58.3561179Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.13.1) 2024-06-26T08:10:58.3564239Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.12.1) 2024-06-26T08:10:58.3566237Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.8.8) 2024-06-26T08:10:58.3569222Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.1.4) 2024-06-26T08:10:58.3571900Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2024.2.0) 2024-06-26T08:10:58.4224310Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.1.5) 2024-06-26T08:10:58.4525874Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.3.0) 2024-06-26T08:10:58.4673096Z Downloading torchmetrics-1.4.0.post0-py3-none-any.whl (868 kB) 2024-06-26T08:10:58.4878324Z Downloading lightning_utilities-0.11.2-py3-none-any.whl (26 kB) 2024-06-26T08:10:58.4975561Z Building wheels for collected packages: torchrec 2024-06-26T08:10:58.9303537Z Building wheel for torchrec (setup.py) ... [?25l- \ | done 2024-06-26T08:10:58.9311752Z [?25h Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374488 sha256=ad0ed813fe16fdc3834443546df84aacdf8fbe1d354fb703b5d9989dfc2cf65c 2024-06-26T08:10:58.9313358Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8 2024-06-26T08:10:58.9334212Z Successfully built torchrec 2024-06-26T08:11:01.0040235Z Installing collected packages: lightning-utilities, torchmetrics, torchrec 2024-06-26T08:11:01.5572662Z Successfully installed lightning-utilities-0.11.2 torchmetrics-1.4.0.post0 torchrec-0.3.2 2024-06-26T08:11:01.6902217Z ++ pwd 2024-06-26T08:11:01.6904151Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench 2024-06-26T08:11:01.6904677Z + test_dynamo_benchmark torchbench 0 2024-06-26T08:11:01.6908526Z ++ pwd 2024-06-26T08:11:01.6922527Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:11:01.6923115Z + local suite=torchbench 2024-06-26T08:11:01.6924801Z + shift 2024-06-26T08:11:01.6925114Z + local shard_id=0 2024-06-26T08:11:01.6925421Z + shift 2024-06-26T08:11:01.6925758Z + [[ inductor_torchbench == *perf_compare* ]] 2024-06-26T08:11:01.6927837Z + [[ inductor_torchbench == *perf* ]] 2024-06-26T08:11:01.6928477Z + [[ inductor_torchbench == *cpu_inductor* ]] 2024-06-26T08:11:01.6929128Z + [[ inductor_torchbench == *aot_inductor* ]] 2024-06-26T08:11:01.6929851Z + test_single_dynamo_benchmark inference torchbench 0 --inference --bfloat16 2024-06-26T08:11:01.6930420Z ++ pwd 2024-06-26T08:11:01.6930884Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:11:01.6931746Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:11:01.6942958Z + local name=inference 2024-06-26T08:11:01.6943306Z + shift 2024-06-26T08:11:01.6943678Z + local suite=torchbench 2024-06-26T08:11:01.6944159Z + shift 2024-06-26T08:11:01.6944559Z + local shard_id=0 2024-06-26T08:11:01.6944934Z + shift 2024-06-26T08:11:01.6945204Z + partition_flags=() 2024-06-26T08:11:01.6945662Z + local partition_flags 2024-06-26T08:11:01.6946060Z + [[ -n 2 ]] 2024-06-26T08:11:01.6946361Z + [[ -n 0 ]] 2024-06-26T08:11:01.6946940Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-06-26T08:11:01.6947614Z + [[ inductor_torchbench == *perf_compare* ]] 2024-06-26T08:11:01.6948077Z + [[ inductor_torchbench == *perf* ]] 2024-06-26T08:11:01.6948528Z + [[ inductor_torchbench == *aot_inductor* ]] 2024-06-26T08:11:01.6950091Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --inductor --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv 2024-06-26T08:11:07.3232031Z 2024-06-26T08:11:07.8760844Z loading model: 0it [00:00, ?it/s]/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/optim/apply_optimizer_in_backward.py:47: DeprecationWarning: `TorchScript` support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the `torch.compile` optimizer instead. 2024-06-26T08:11:07.8762866Z from torch.distributed.optim import _apply_optimizer_in_backward 2024-06-26T08:11:08.0659037Z 2024-06-26T08:11:08.0659912Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:11:08.0660641Z cuda eval torchrec_dlrm 2024-06-26T08:11:08.0670257Z Traceback (most recent call last): 2024-06-26T08:11:08.0671376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:11:08.0672492Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:11:08.0673697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 447, in forward_pass 2024-06-26T08:11:08.0674742Z return mod(*inputs) 2024-06-26T08:11:08.0676078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:11:08.0677400Z return self._call_impl(*args, **kwargs) 2024-06-26T08:11:08.0678820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:11:08.0680043Z return forward_call(*args, **kwargs) 2024-06-26T08:11:08.0681299Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 893, in forward 2024-06-26T08:11:08.0682656Z logits = self.model(batch.dense_features, batch.sparse_features) 2024-06-26T08:11:08.0684344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:11:08.0685737Z return self._call_impl(*args, **kwargs) 2024-06-26T08:11:08.0687132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:11:08.0688439Z return forward_call(*args, **kwargs) 2024-06-26T08:11:08.0689938Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 572, in forward 2024-06-26T08:11:08.0691221Z concatenated_dense = self.inter_arch( 2024-06-26T08:11:08.0692689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:11:08.0694110Z return self._call_impl(*args, **kwargs) 2024-06-26T08:11:08.0695545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:11:08.0696854Z return forward_call(*args, **kwargs) 2024-06-26T08:11:08.0698547Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 284, in forward 2024-06-26T08:11:08.0700253Z return self.crossnet(combined_values.reshape([B, -1])) 2024-06-26T08:11:08.0701943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:11:08.0703250Z return self._call_impl(*args, **kwargs) 2024-06-26T08:11:08.0704896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:11:08.0706284Z return forward_call(*args, **kwargs) 2024-06-26T08:11:08.0707654Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/crossnet.py", line 175, in forward 2024-06-26T08:11:08.0709177Z x_l_v = torch.nn.functional.linear(x_l, self.V_kernels[layer]) 2024-06-26T08:11:08.0710513Z RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::BFloat16 2024-06-26T08:11:08.0711405Z 2024-06-26T08:11:08.0711883Z The above exception was the direct cause of the following exception: 2024-06-26T08:11:08.0712639Z 2024-06-26T08:11:08.0712870Z Traceback (most recent call last): 2024-06-26T08:11:08.0713922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:11:08.0714996Z ) = runner.load_model( 2024-06-26T08:11:08.0716062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:11:08.0717292Z self.validate_model(model, example_inputs) 2024-06-26T08:11:08.0718498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:11:08.0719710Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:11:08.0720416Z RuntimeError: Eager run failed 2024-06-26T08:11:08.0720786Z 2024-06-26T08:11:08.0720955Z eager_fail_to_run 2024-06-26T08:11:11.3683336Z 2024-06-26T08:11:13.3226445Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:11:13.3226929Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:11:13.3291032Z cuda eval BERT_pytorch 2024-06-26T08:11:31.3914657Z pass 2024-06-26T08:11:31.3926844Z TIMING: code_gen:4.6807 inductor_compile:8.41352 backend_compile:13.75378 entire_frame_compile:16.33147 2024-06-26T08:11:31.3928141Z STATS: call_* op count: 1076 | FakeTensor.__torch_dispatch__:3212 | FakeTensorMode.__torch_dispatch__:36008 | ProxyTorchDispatchMode.__torch_dispatch__:12905 2024-06-26T08:11:31.3929413Z Dynamo produced 1 graphs covering 1076 ops with 0 graph breaks (0 unique) 2024-06-26T08:11:35.2881750Z 2024-06-26T08:11:37.8601925Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:11:37.8602394Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:11:37.8654235Z cuda eval Background_Matting 2024-06-26T08:11:37.8658494Z pass_due_to_skip 2024-06-26T08:11:37.9174267Z TIMING: 2024-06-26T08:11:37.9174756Z STATS: call_* op count: 0 2024-06-26T08:11:37.9175324Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T08:11:41.1441832Z 2024-06-26T08:11:43.7615029Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:11:43.7615524Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:11:43.7642134Z cuda eval LearningToPaint 2024-06-26T08:11:50.9850455Z pass 2024-06-26T08:11:50.9869092Z TIMING: code_gen:1.6636 inductor_compile:2.87086 backend_compile:5.21361 entire_frame_compile:5.85745 2024-06-26T08:11:50.9870394Z STATS: call_* op count: 142 | FakeTensor.__torch_dispatch__:1544 | FakeTensorMode.__torch_dispatch__:11677 | ProxyTorchDispatchMode.__torch_dispatch__:3014 2024-06-26T08:11:50.9871520Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-06-26T08:11:54.7012683Z 2024-06-26T08:11:55.9181025Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/vgg16-397923af.pth 2024-06-26T08:11:55.9413338Z 2024-06-26T08:11:55.9413493Z 2024-06-26T08:11:56.0455272Z 0% 0.00/528M [00:00 will be ignored 2024-06-26T08:41:57.2003545Z pass 2024-06-26T08:41:57.2258211Z TIMING: entire_frame_compile:53.73096 code_gen:34.82023 inductor_compile:49.55593 backend_compile:47.18921 2024-06-26T08:41:57.2260354Z STATS: call_* op count: 672 | FakeTensorMode.__torch_dispatch__:64218 | FakeTensor.__torch_dispatch__:11004 | ProxyTorchDispatchMode.__torch_dispatch__:17164 2024-06-26T08:41:57.2261510Z Dynamo produced 3 graphs covering 672 ops with 7 graph breaks (5 unique) 2024-06-26T08:42:03.1439099Z 2024-06-26T08:42:04.3775846Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:42:04.3776324Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:42:04.3776775Z cuda train alexnet 2024-06-26T08:42:21.1734749Z pass 2024-06-26T08:42:21.1925538Z TIMING: entire_frame_compile:4.80241 code_gen:7.55241 inductor_compile:8.1519 backend_compile:4.5336 2024-06-26T08:42:21.1926846Z STATS: call_* op count: 26 | FakeTensor.__torch_dispatch__:449 | FakeTensorMode.__torch_dispatch__:3397 | ProxyTorchDispatchMode.__torch_dispatch__:950 2024-06-26T08:42:21.1931795Z Dynamo produced 2 graphs covering 26 ops with 6 graph breaks (5 unique) 2024-06-26T08:42:24.8162357Z 2024-06-26T08:42:26.5178781Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:42:26.5179264Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:42:26.5179718Z cuda train basic_gnn_edgecnn 2024-06-26T08:42:29.6415954Z skipping cudagraphs due to ['incompatible ops'] 2024-06-26T08:42:31.4799171Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:42:31.4799946Z File "/tmp/jenkins_pyg/tmpfbomwhti.py", line 190, in propagate 2024-06-26T08:42:31.4800597Z kwargs = self._collect(edge_index, the_size, in_kwargs) 2024-06-26T08:42:31.4801272Z File "/tmp/jenkins_pyg/tmpfbomwhti.py", line 128, in _collect 2024-06-26T08:42:31.4801830Z x_i = self._lift(x_i, edge_def, i) 2024-06-26T08:42:31.4802365Z File "/tmp/jenkins_pyg/tmpfbomwhti.py", line 88, in _lift 2024-06-26T08:42:31.4802935Z return src.index_select(self.node_dim, index) 2024-06-26T08:42:31.4803281Z 2024-06-26T08:42:31.4914281Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:42:31.4915229Z File "/tmp/jenkins_pyg/tmpfbomwhti.py", line 190, in propagate 2024-06-26T08:42:31.4915886Z kwargs = self._collect(edge_index, the_size, in_kwargs) 2024-06-26T08:42:31.4916509Z File "/tmp/jenkins_pyg/tmpfbomwhti.py", line 128, in _collect 2024-06-26T08:42:31.4917055Z x_i = self._lift(x_i, edge_def, i) 2024-06-26T08:42:31.4917575Z File "/tmp/jenkins_pyg/tmpfbomwhti.py", line 88, in _lift 2024-06-26T08:42:31.4918136Z return src.index_select(self.node_dim, index) 2024-06-26T08:42:31.4918471Z 2024-06-26T08:42:31.6122603Z W0626 08:42:31.611000 140390683140736 torch/_logging/_internal.py:1040] [24/0] Profiler function will be ignored 2024-06-26T08:42:36.7844150Z pass 2024-06-26T08:42:36.7853604Z TIMING: entire_frame_compile:7.50295 inductor_compile:5.04087 backend_compile:5.80339 code_gen:3.58866 2024-06-26T08:42:36.7854915Z STATS: call_* op count: 103 | FakeTensorMode.__torch_dispatch__:7254 | ProxyTorchDispatchMode.__torch_dispatch__:1787 | FakeTensor.__torch_dispatch__:1007 2024-06-26T08:42:36.7856041Z Dynamo produced 8 graphs covering 103 ops with 20 graph breaks (6 unique) 2024-06-26T08:42:40.4616441Z 2024-06-26T08:42:42.2206170Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:42:42.2206699Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:42:42.2207138Z cuda train basic_gnn_gcn 2024-06-26T08:42:44.7006201Z skipping cudagraphs due to ['incompatible ops'] 2024-06-26T08:42:45.2232022Z skipping cudagraphs due to ['incompatible ops'] 2024-06-26T08:42:45.5970726Z skipping cudagraphs due to ['incompatible ops'] 2024-06-26T08:42:45.8237248Z skipping cudagraphs due to ['incompatible ops'] 2024-06-26T08:42:46.3024022Z W0626 08:42:46.301000 139904630715008 torch/_logging/_internal.py:1040] [18/0] Profiler function will be ignored 2024-06-26T08:42:49.7953188Z pass 2024-06-26T08:42:49.7957403Z TIMING: entire_frame_compile:5.21031 inductor_compile:3.68523 backend_compile:4.29643 code_gen:2.62339 2024-06-26T08:42:49.7958720Z STATS: call_* op count: 99 | FakeTensorMode.__torch_dispatch__:4291 | FakeTensor.__torch_dispatch__:551 | ProxyTorchDispatchMode.__torch_dispatch__:1004 2024-06-26T08:42:49.7959834Z Dynamo produced 10 graphs covering 99 ops with 13 graph breaks (6 unique) 2024-06-26T08:42:53.4228365Z 2024-06-26T08:42:55.1188852Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:42:55.1189344Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:42:55.1189793Z cuda train basic_gnn_gin 2024-06-26T08:42:57.9944451Z skipping cudagraphs due to ['incompatible ops'] 2024-06-26T08:42:58.8122382Z W0626 08:42:58.811000 139999331123840 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:43:03.1998984Z pass 2024-06-26T08:43:03.2033292Z TIMING: entire_frame_compile:6.45583 code_gen:3.14844 inductor_compile:4.65812 backend_compile:5.35651 2024-06-26T08:43:03.2034576Z STATS: call_* op count: 113 | FakeTensorMode.__torch_dispatch__:6784 | FakeTensor.__torch_dispatch__:965 | ProxyTorchDispatchMode.__torch_dispatch__:1747 2024-06-26T08:43:03.2035687Z Dynamo produced 3 graphs covering 113 ops with 7 graph breaks (5 unique) 2024-06-26T08:43:06.8772443Z 2024-06-26T08:43:08.6079995Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:43:08.6080895Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:43:08.6081771Z cuda train basic_gnn_sage 2024-06-26T08:43:11.2598933Z skipping cudagraphs due to ['incompatible ops'] 2024-06-26T08:43:11.8634208Z W0626 08:43:11.862000 140050903945856 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:43:15.5376003Z pass 2024-06-26T08:43:15.5399866Z TIMING: entire_frame_compile:5.51446 code_gen:2.64474 inductor_compile:3.91732 backend_compile:4.58936 2024-06-26T08:43:15.5401321Z STATS: call_* op count: 107 | FakeTensorMode.__torch_dispatch__:5526 | FakeTensor.__torch_dispatch__:739 | ProxyTorchDispatchMode.__torch_dispatch__:1396 2024-06-26T08:43:15.5402445Z Dynamo produced 3 graphs covering 107 ops with 7 graph breaks (5 unique) 2024-06-26T08:43:19.1693981Z 2024-06-26T08:43:19.6506961Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:43:19.6507603Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:43:19.6508163Z cuda train dcgan 2024-06-26T08:43:23.9685505Z pass 2024-06-26T08:43:23.9703454Z TIMING: entire_frame_compile:2.16937 code_gen:0.9486 inductor_compile:1.85113 backend_compile:1.89942 2024-06-26T08:43:23.9704732Z STATS: call_* op count: 17 | FakeTensor.__torch_dispatch__:526 | FakeTensorMode.__torch_dispatch__:3162 | ProxyTorchDispatchMode.__torch_dispatch__:842 2024-06-26T08:43:23.9705825Z Dynamo produced 2 graphs covering 17 ops with 6 graph breaks (5 unique) 2024-06-26T08:43:27.4681384Z 2024-06-26T08:43:30.8782600Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:43:30.8783114Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:43:30.8783568Z cuda train demucs 2024-06-26T08:44:38.1469254Z pass 2024-06-26T08:44:38.1717911Z TIMING: entire_frame_compile:6.00329 inductor_compile:8.49005 backend_compile:5.07672 code_gen:6.1638 2024-06-26T08:44:38.1722342Z STATS: call_* op count: 88 | FakeTensorMode.__torch_dispatch__:9342 | FakeTensor.__torch_dispatch__:1630 | ProxyTorchDispatchMode.__torch_dispatch__:2363 2024-06-26T08:44:38.1724593Z Dynamo produced 6 graphs covering 88 ops with 9 graph breaks (6 unique) 2024-06-26T08:44:42.0294911Z 2024-06-26T08:44:44.1003325Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:44:44.1004263Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:44:44.1005158Z cuda train densenet121 2024-06-26T08:46:24.6547767Z pass 2024-06-26T08:46:24.6619686Z TIMING: entire_frame_compile:41.47845 code_gen:49.016 inductor_compile:68.35076 backend_compile:37.05639 2024-06-26T08:46:24.6621847Z STATS: call_* op count: 435 | FakeTensor.__torch_dispatch__:17439 | FakeTensorMode.__torch_dispatch__:100721 | ProxyTorchDispatchMode.__torch_dispatch__:30835 2024-06-26T08:46:24.6623022Z Dynamo produced 2 graphs covering 435 ops with 6 graph breaks (5 unique) 2024-06-26T08:46:31.3163435Z 2024-06-26T08:46:35.8113895Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:46:35.8116108Z loading model: 0it [00:04, ?it/s] 2024-06-26T08:46:35.8116653Z cuda train detectron2_maskrcnn_r_50_c4 2024-06-26T08:46:35.8125608Z Traceback (most recent call last): 2024-06-26T08:46:35.8126363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:46:35.8127151Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:46:35.8128008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in forward_and_backward_pass 2024-06-26T08:46:35.8128950Z pred = mod(*cloned_inputs) 2024-06-26T08:46:35.8130041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:46:35.8130934Z return self._call_impl(*args, **kwargs) 2024-06-26T08:46:35.8131862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:46:35.8132697Z return forward_call(*args, **kwargs) 2024-06-26T08:46:35.8133655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-06-26T08:46:35.8134771Z return self.inference(batched_inputs) 2024-06-26T08:46:35.8136369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference 2024-06-26T08:46:35.8138091Z results, _ = self.roi_heads(images, features, proposals, None) 2024-06-26T08:46:35.8139278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:46:35.8140173Z return self._call_impl(*args, **kwargs) 2024-06-26T08:46:35.8141106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:46:35.8141937Z return forward_call(*args, **kwargs) 2024-06-26T08:46:35.8142916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward 2024-06-26T08:46:35.8143828Z box_features = self._shared_roi_transform( 2024-06-26T08:46:35.8144912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform 2024-06-26T08:46:35.8145883Z x = self.pooler(features, boxes) 2024-06-26T08:46:35.8146827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:46:35.8147715Z return self._call_impl(*args, **kwargs) 2024-06-26T08:46:35.8148642Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:46:35.8149475Z return forward_call(*args, **kwargs) 2024-06-26T08:46:35.8150377Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward 2024-06-26T08:46:35.8151270Z return self.level_poolers[0](x[0], pooler_fmt_boxes) 2024-06-26T08:46:35.8152293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:46:35.8153181Z return self._call_impl(*args, **kwargs) 2024-06-26T08:46:35.8154089Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:46:35.8155150Z return forward_call(*args, **kwargs) 2024-06-26T08:46:35.8156052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward 2024-06-26T08:46:35.8156842Z return roi_align( 2024-06-26T08:46:35.8157751Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-06-26T08:46:35.8158875Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-06-26T08:46:35.8160060Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-06-26T08:46:35.8161142Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-06-26T08:46:35.8162360Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-06-26T08:46:35.8163237Z v1 = masked_index(y_low, x_low) 2024-06-26T08:46:35.8164136Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-06-26T08:46:35.8164938Z return input[ 2024-06-26T08:46:35.8168237Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 21890.87 GiB. GPU 0 has a total capacity of 21.98 GiB of which 16.54 GiB is free. Process 1407 has 5.42 GiB memory in use. Of the allocated memory 4.58 GiB is allocated by PyTorch, and 558.89 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-06-26T08:46:35.8171110Z 2024-06-26T08:46:35.8171409Z The above exception was the direct cause of the following exception: 2024-06-26T08:46:35.8171860Z 2024-06-26T08:46:35.8172016Z Traceback (most recent call last): 2024-06-26T08:46:35.8172666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:46:35.8173328Z ) = runner.load_model( 2024-06-26T08:46:35.8174001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:46:35.8174768Z self.validate_model(model, example_inputs) 2024-06-26T08:46:35.8175542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:46:35.8190579Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:46:35.8191305Z RuntimeError: Eager run failed 2024-06-26T08:46:35.8191668Z 2024-06-26T08:46:35.8191826Z eager_fail_to_run 2024-06-26T08:46:39.1479715Z 2024-06-26T08:46:47.3180192Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:46:47.3180702Z loading model: 0it [00:08, ?it/s] 2024-06-26T08:46:47.3181163Z cuda train dlrm 2024-06-26T08:46:55.4752820Z skipping cudagraphs due to sparsity not handled. Please file issue for sparse inference weights. Found from : 2024-06-26T08:46:55.4754394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in torch_dynamo_resume_in_forward_and_backward_pass_at_451 2024-06-26T08:46:55.4755521Z pred = mod(*cloned_inputs) 2024-06-26T08:46:55.4756630Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:46:55.4757554Z return forward_call(*args, **kwargs) 2024-06-26T08:46:55.4758519Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 355, in forward 2024-06-26T08:46:55.4759539Z return self.sequential_forward(dense_x, lS_o, lS_i) 2024-06-26T08:46:55.4760680Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 367, in sequential_forward 2024-06-26T08:46:55.4761719Z ly = self.apply_emb(lS_o, lS_i, self.emb_l) 2024-06-26T08:46:55.4762958Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 308, in apply_emb 2024-06-26T08:46:55.4764060Z V = E(sparse_index_group_batch, sparse_offset_group_batch) 2024-06-26T08:46:55.4764450Z 2024-06-26T08:46:55.9444961Z pass 2024-06-26T08:46:55.9854424Z TIMING: entire_frame_compile:5.92165 code_gen:1.28006 inductor_compile:1.9166 backend_compile:5.52493 2024-06-26T08:46:55.9855879Z STATS: call_* op count: 40 | FakeTensor.__torch_dispatch__:548 | FakeTensorMode.__torch_dispatch__:5142 | ProxyTorchDispatchMode.__torch_dispatch__:1507 2024-06-26T08:46:55.9856987Z Dynamo produced 2 graphs covering 40 ops with 6 graph breaks (5 unique) 2024-06-26T08:46:59.5967359Z 2024-06-26T08:47:00.9816223Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:47:00.9816680Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:47:00.9817132Z cuda train drq 2024-06-26T08:47:08.3332205Z W0626 08:47:08.332000 139704235790976 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:47:17.6495363Z pass 2024-06-26T08:47:17.6523723Z TIMING: entire_frame_compile:13.89553 code_gen:6.90467 inductor_compile:9.02582 backend_compile:12.13768 2024-06-26T08:47:17.6525034Z STATS: call_* op count: 137 | FakeTensorMode.__torch_dispatch__:12454 | FakeTensor.__torch_dispatch__:1952 | ProxyTorchDispatchMode.__torch_dispatch__:3300 2024-06-26T08:47:17.6526151Z Dynamo produced 5 graphs covering 137 ops with 7 graph breaks (5 unique) 2024-06-26T08:47:21.5480642Z 2024-06-26T08:47:23.1602760Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:47:23.1603237Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:47:23.1607217Z cuda train fastNLP_Bert 2024-06-26T08:47:29.6200640Z W0626 08:47:29.619000 140505015464576 torch/_dynamo/variables/tensor.py:714] [3/0] Graph break from `Tensor.item()`, consider setting: 2024-06-26T08:47:29.6202765Z W0626 08:47:29.619000 140505015464576 torch/_dynamo/variables/tensor.py:714] [3/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-26T08:47:29.6204358Z W0626 08:47:29.619000 140505015464576 torch/_dynamo/variables/tensor.py:714] [3/0] or: 2024-06-26T08:47:29.6205787Z W0626 08:47:29.619000 140505015464576 torch/_dynamo/variables/tensor.py:714] [3/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-26T08:47:29.6207858Z W0626 08:47:29.619000 140505015464576 torch/_dynamo/variables/tensor.py:714] [3/0] to include these operations in the captured graph. 2024-06-26T08:47:29.6209696Z W0626 08:47:29.619000 140505015464576 torch/_dynamo/variables/tensor.py:714] [3/0] 2024-06-26T08:48:05.0840169Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:48:05.0842313Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/modules/encoder/bert.py", line 512, in forward 2024-06-26T08:48:05.0843396Z embedding_output = self.embeddings(input_ids, token_type_ids) 2024-06-26T08:48:05.0844608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:48:05.0845557Z return forward_call(*args, **kwargs) 2024-06-26T08:48:05.0846579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/modules/encoder/bert.py", line 236, in forward 2024-06-26T08:48:05.0847573Z words_embeddings = self.word_embeddings(input_ids) 2024-06-26T08:48:05.0847971Z 2024-06-26T08:48:08.0488150Z pass 2024-06-26T08:48:08.1133813Z TIMING: entire_frame_compile:24.5865 inductor_compile:20.66734 backend_compile:21.62726 code_gen:12.2978 2024-06-26T08:48:08.1135127Z STATS: call_* op count: 448 | FakeTensorMode.__torch_dispatch__:53257 | ProxyTorchDispatchMode.__torch_dispatch__:19106 | FakeTensor.__torch_dispatch__:6560 2024-06-26T08:48:08.1136273Z Dynamo produced 7 graphs covering 448 ops with 10 graph breaks (6 unique) 2024-06-26T08:48:12.8453205Z 2024-06-26T08:48:13.6059393Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:48:13.6060203Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:48:13.6060660Z cuda train functorch_dp_cifar10 2024-06-26T08:48:34.0651960Z W0626 08:48:34.064000 140275233337984 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:48:53.3286779Z pass 2024-06-26T08:48:53.3346609Z TIMING: entire_frame_compile:26.71733 code_gen:16.17132 inductor_compile:24.57625 backend_compile:23.29687 2024-06-26T08:48:53.3347913Z STATS: call_* op count: 277 | FakeTensor.__torch_dispatch__:7229 | FakeTensorMode.__torch_dispatch__:35296 | ProxyTorchDispatchMode.__torch_dispatch__:7938 2024-06-26T08:48:53.3349021Z Dynamo produced 3 graphs covering 277 ops with 7 graph breaks (5 unique) 2024-06-26T08:48:58.1167459Z 2024-06-26T08:48:58.6318961Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:48:58.6319442Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:48:58.6319891Z cuda train functorch_maml_omniglot 2024-06-26T08:49:05.3969528Z W0626 08:49:05.395000 139653679149696 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:49:11.3490576Z pass 2024-06-26T08:49:11.3491455Z TIMING: entire_frame_compile:8.4753 code_gen:6.20075 inductor_compile:7.63913 backend_compile:7.67214 2024-06-26T08:49:11.3495314Z STATS: call_* op count: 78 | FakeTensor.__torch_dispatch__:1297 | FakeTensorMode.__torch_dispatch__:7066 | ProxyTorchDispatchMode.__torch_dispatch__:1433 2024-06-26T08:49:11.3496457Z Dynamo produced 3 graphs covering 78 ops with 7 graph breaks (5 unique) 2024-06-26T08:49:15.0627040Z 2024-06-26T08:49:16.2785246Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:49:16.2785846Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:49:16.2786292Z cuda train hf_Albert 2024-06-26T08:49:50.0088326Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:49:50.0091063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in torch_dynamo_resume_in_forward_and_backward_pass_at_451 2024-06-26T08:49:50.0092114Z pred = mod(*cloned_inputs) 2024-06-26T08:49:50.0093110Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:49:50.0093968Z return forward_call(*args, **kwargs) 2024-06-26T08:49:50.0095028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 986, in forward 2024-06-26T08:49:50.0096266Z outputs = self.albert( 2024-06-26T08:49:50.0097315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:49:50.0098742Z return forward_call(*args, **kwargs) 2024-06-26T08:49:50.0100071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 729, in forward 2024-06-26T08:49:50.0101078Z embedding_output = self.embeddings( 2024-06-26T08:49:50.0102036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:49:50.0102870Z return forward_call(*args, **kwargs) 2024-06-26T08:49:50.0103883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 250, in forward 2024-06-26T08:49:50.0104858Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T08:49:50.0105211Z 2024-06-26T08:49:50.0862876Z W0626 08:49:50.085000 140308251218560 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:50:00.6404775Z pass 2024-06-26T08:50:00.6556621Z TIMING: entire_frame_compile:25.81958 code_gen:13.25704 inductor_compile:22.19003 backend_compile:22.40276 2024-06-26T08:50:00.6558259Z STATS: call_* op count: 542 | FakeTensorMode.__torch_dispatch__:63913 | FakeTensor.__torch_dispatch__:8699 | ProxyTorchDispatchMode.__torch_dispatch__:21513 2024-06-26T08:50:00.6559720Z Dynamo produced 3 graphs covering 542 ops with 6 graph breaks (5 unique) 2024-06-26T08:50:05.3805312Z 2024-06-26T08:50:09.0742235Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:50:09.0742851Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:50:09.0743310Z cuda train hf_Bart 2024-06-26T08:50:52.1882205Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:50:52.1884573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in torch_dynamo_resume_in_forward_and_backward_pass_at_451 2024-06-26T08:50:52.1885613Z pred = mod(*cloned_inputs) 2024-06-26T08:50:52.1886613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:50:52.1887468Z return forward_call(*args, **kwargs) 2024-06-26T08:50:52.1888491Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 69, in forward 2024-06-26T08:50:52.1890042Z return self.model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) 2024-06-26T08:50:52.1891153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:50:52.1892047Z return forward_call(*args, **kwargs) 2024-06-26T08:50:52.1893064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1731, in forward 2024-06-26T08:50:52.1893967Z outputs = self.model( 2024-06-26T08:50:52.1894820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:50:52.1895706Z return forward_call(*args, **kwargs) 2024-06-26T08:50:52.1896704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1599, in forward 2024-06-26T08:50:52.1897917Z encoder_outputs = self.encoder( 2024-06-26T08:50:52.1898827Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:50:52.1899671Z return forward_call(*args, **kwargs) 2024-06-26T08:50:52.1900666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1152, in forward 2024-06-26T08:50:52.1901694Z inputs_embeds = self.embed_tokens(input_ids) * self.embed_scale 2024-06-26T08:50:52.1902114Z 2024-06-26T08:50:52.2767205Z W0626 08:50:52.275000 140169454563968 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:51:56.3194891Z pass 2024-06-26T08:51:56.3635152Z TIMING: entire_frame_compile:82.36094 code_gen:30.64564 inductor_compile:54.42719 backend_compile:69.84063 2024-06-26T08:51:56.3637758Z STATS: call_* op count: 1250 | FakeTensorMode.__torch_dispatch__:131917 | FakeTensor.__torch_dispatch__:22376 | ProxyTorchDispatchMode.__torch_dispatch__:32320 2024-06-26T08:51:56.3640039Z Dynamo produced 3 graphs covering 1250 ops with 6 graph breaks (5 unique) 2024-06-26T08:52:04.2581368Z 2024-06-26T08:52:06.7597755Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:52:06.7598314Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:52:06.7598761Z cuda train hf_Bert 2024-06-26T08:52:44.9548900Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:52:44.9550103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in torch_dynamo_resume_in_forward_and_backward_pass_at_451 2024-06-26T08:52:44.9551848Z pred = mod(*cloned_inputs) 2024-06-26T08:52:44.9553402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:52:44.9554290Z return forward_call(*args, **kwargs) 2024-06-26T08:52:44.9555302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1360, in forward 2024-06-26T08:52:44.9556419Z outputs = self.bert( 2024-06-26T08:52:44.9557495Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:52:44.9558531Z return forward_call(*args, **kwargs) 2024-06-26T08:52:44.9559527Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2024-06-26T08:52:44.9560560Z embedding_output = self.embeddings( 2024-06-26T08:52:44.9561488Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:52:44.9562321Z return forward_call(*args, **kwargs) 2024-06-26T08:52:44.9563290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2024-06-26T08:52:44.9564215Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T08:52:44.9564555Z 2024-06-26T08:52:45.0367371Z W0626 08:52:45.035000 139629383795328 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:53:33.6820447Z pass 2024-06-26T08:53:33.7222255Z TIMING: entire_frame_compile:67.17415 code_gen:22.19281 inductor_compile:40.82226 backend_compile:55.52898 2024-06-26T08:53:33.7224449Z STATS: call_* op count: 995 | FakeTensorMode.__torch_dispatch__:109769 | FakeTensor.__torch_dispatch__:17355 | ProxyTorchDispatchMode.__torch_dispatch__:27771 2024-06-26T08:53:33.7225592Z Dynamo produced 3 graphs covering 995 ops with 6 graph breaks (5 unique) 2024-06-26T08:53:40.6555304Z 2024-06-26T08:53:45.1563954Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:53:45.1564836Z loading model: 0it [00:04, ?it/s] 2024-06-26T08:53:45.1569818Z cuda train hf_Bert_large 2024-06-26T08:55:02.1095525Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:55:02.1097420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in torch_dynamo_resume_in_forward_and_backward_pass_at_451 2024-06-26T08:55:02.1098597Z pred = mod(*cloned_inputs) 2024-06-26T08:55:02.1099606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:55:02.1100464Z return forward_call(*args, **kwargs) 2024-06-26T08:55:02.1101484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1360, in forward 2024-06-26T08:55:02.1102606Z outputs = self.bert( 2024-06-26T08:55:02.1103692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:55:02.1104759Z return forward_call(*args, **kwargs) 2024-06-26T08:55:02.1105771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2024-06-26T08:55:02.1106955Z embedding_output = self.embeddings( 2024-06-26T08:55:02.1108080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:55:02.1108931Z return forward_call(*args, **kwargs) 2024-06-26T08:55:02.1109913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2024-06-26T08:55:02.1110846Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T08:55:02.1111194Z 2024-06-26T08:55:02.2321485Z W0626 08:55:02.231000 140378791465600 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:56:35.3972593Z pass 2024-06-26T08:56:35.5092308Z TIMING: entire_frame_compile:128.11433 code_gen:38.63358 inductor_compile:75.55773 backend_compile:104.19195 2024-06-26T08:56:35.5094986Z STATS: call_* op count: 1919 | FakeTensorMode.__torch_dispatch__:214253 | FakeTensor.__torch_dispatch__:33783 | ProxyTorchDispatchMode.__torch_dispatch__:54411 2024-06-26T08:56:35.5096171Z Dynamo produced 3 graphs covering 1919 ops with 6 graph breaks (5 unique) 2024-06-26T08:56:45.8649682Z 2024-06-26T08:56:49.1863927Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:56:49.1864404Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:56:49.1864851Z cuda train hf_BigBird 2024-06-26T08:57:00.4467384Z WARNING:common:fp64 golden ref were not generated for hf_BigBird. Setting accuracy check to cosine 2024-06-26T08:57:44.8117310Z ERROR:common: 2024-06-26T08:57:44.8118501Z Traceback (most recent call last): 2024-06-26T08:57:44.8119270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2627, in check_accuracy 2024-06-26T08:57:44.8120326Z new_result = optimized_model_iter_fn(model_copy, example_inputs) 2024-06-26T08:57:44.8121706Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 434, in _fn 2024-06-26T08:57:44.8122549Z return fn(*args, **kwargs) 2024-06-26T08:57:44.8123405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2359, in run_n_iterations 2024-06-26T08:57:44.8124299Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-06-26T08:57:44.8125583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 450, in forward_and_backward_pass 2024-06-26T08:57:44.8126548Z cloned_inputs = clone_inputs(inputs) 2024-06-26T08:57:44.8127661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 451, in torch_dynamo_resume_in_forward_and_backward_pass_at_450 2024-06-26T08:57:44.8128673Z self.optimizer_zero_grad(mod) 2024-06-26T08:57:44.8129722Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1121, in __call__ 2024-06-26T08:57:44.8130586Z return self._torchdynamo_orig_callable( 2024-06-26T08:57:44.8131522Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 948, in __call__ 2024-06-26T08:57:44.8132337Z result = self._inner_convert( 2024-06-26T08:57:44.8133257Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 472, in __call__ 2024-06-26T08:57:44.8134336Z return _compile( 2024-06-26T08:57:44.8135340Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 85, in wrapper_function 2024-06-26T08:57:44.8136306Z return StrobelightCompileTimeProfiler.profile_compile_time( 2024-06-26T08:57:44.8137808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_strobelight/compile_time_profiler.py", line 129, in profile_compile_time 2024-06-26T08:57:44.8138819Z return func(*args, **kwargs) 2024-06-26T08:57:44.8139431Z File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner 2024-06-26T08:57:44.8140066Z return func(*args, **kwds) 2024-06-26T08:57:44.8140931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 817, in _compile 2024-06-26T08:57:44.8141867Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-06-26T08:57:44.8142878Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 233, in time_wrapper 2024-06-26T08:57:44.8143674Z r = func(*args, **kwargs) 2024-06-26T08:57:44.8144561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 636, in compile_inner 2024-06-26T08:57:44.8145462Z out_code = transform_code_object(code, transform) 2024-06-26T08:57:44.8146560Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1270, in transform_code_object 2024-06-26T08:57:44.8147546Z transformations(instructions, code_options) 2024-06-26T08:57:44.8148458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 178, in _fn 2024-06-26T08:57:44.8149241Z return fn(*args, **kwargs) 2024-06-26T08:57:44.8150101Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 582, in transform 2024-06-26T08:57:44.8151099Z tracer.run() 2024-06-26T08:57:44.8151894Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2476, in run 2024-06-26T08:57:44.8152684Z super().run() 2024-06-26T08:57:44.8153468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 904, in run 2024-06-26T08:57:44.8154348Z while self.step(): 2024-06-26T08:57:44.8155170Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 816, in step 2024-06-26T08:57:44.8156034Z self.dispatch_table[inst.opcode](self, inst) 2024-06-26T08:57:44.8156985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 508, in wrapper 2024-06-26T08:57:44.8157906Z return handle_graph_break(self, inst, speculation.reason) 2024-06-26T08:57:44.8158981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 577, in handle_graph_break 2024-06-26T08:57:44.8159947Z self.output.compile_subgraph(self, reason=reason) 2024-06-26T08:57:44.8160960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1127, in compile_subgraph 2024-06-26T08:57:44.8161958Z self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root) 2024-06-26T08:57:44.8162754Z File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner 2024-06-26T08:57:44.8163386Z return func(*args, **kwds) 2024-06-26T08:57:44.8164341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1324, in compile_and_call_fx_graph 2024-06-26T08:57:44.8165324Z compiled_fn = self.call_user_compiler(gm) 2024-06-26T08:57:44.8166226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 233, in time_wrapper 2024-06-26T08:57:44.8167015Z r = func(*args, **kwargs) 2024-06-26T08:57:44.8167927Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1415, in call_user_compiler 2024-06-26T08:57:44.8169060Z raise BackendCompilerFailed(self.compiler_fn, e).with_traceback( 2024-06-26T08:57:44.8170154Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1396, in call_user_compiler 2024-06-26T08:57:44.8171102Z compiled_fn = compiler_fn(gm, self.example_inputs()) 2024-06-26T08:57:44.8172109Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 129, in __call__ 2024-06-26T08:57:44.8172993Z compiled_gm = compiler_fn(gm, example_inputs) 2024-06-26T08:57:44.8173856Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/__init__.py", line 2141, in __call__ 2024-06-26T08:57:44.8174782Z return compile_fx(model_, inputs_, config_patches=self.config) 2024-06-26T08:57:44.8175547Z File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner 2024-06-26T08:57:44.8176183Z return func(*args, **kwds) 2024-06-26T08:57:44.8177049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1550, in compile_fx 2024-06-26T08:57:44.8178052Z return aot_autograd( 2024-06-26T08:57:44.8178897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 69, in __call__ 2024-06-26T08:57:44.8179834Z cg = aot_module_simplified(gm, example_inputs, **self.kwargs) 2024-06-26T08:57:44.8180921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 990, in aot_module_simplified 2024-06-26T08:57:44.8181835Z compiled_fn = dispatch_and_compile() 2024-06-26T08:57:44.8182821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 979, in dispatch_and_compile 2024-06-26T08:57:44.8183762Z compiled_fn, _ = create_aot_dispatcher_function( 2024-06-26T08:57:44.8184696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 233, in time_wrapper 2024-06-26T08:57:44.8185581Z r = func(*args, **kwargs) 2024-06-26T08:57:44.8186569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 708, in create_aot_dispatcher_function 2024-06-26T08:57:44.8187525Z compiled_fn, fw_metadata = compiler_fn( 2024-06-26T08:57:44.8188746Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 315, in aot_dispatch_autograd 2024-06-26T08:57:44.8189822Z fw_module, bw_module = aot_config.partition_fn( 2024-06-26T08:57:44.8190805Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1482, in partition_fn 2024-06-26T08:57:44.8191655Z _recursive_joint_graph_passes(graph) 2024-06-26T08:57:44.8192674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 269, in _recursive_joint_graph_passes 2024-06-26T08:57:44.8193577Z joint_graph_passes(gm) 2024-06-26T08:57:44.8194530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/fx_passes/joint_graph.py", line 329, in joint_graph_passes 2024-06-26T08:57:44.8195482Z constant_fold_uniform_value(graph) 2024-06-26T08:57:44.8196114Z File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner 2024-06-26T08:57:44.8196747Z return func(*args, **kwds) 2024-06-26T08:57:44.8197749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/fx_passes/joint_graph.py", line 229, in constant_fold_uniform_value 2024-06-26T08:57:44.8198679Z cf.run() 2024-06-26T08:57:44.8199458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py", line 200, in run 2024-06-26T08:57:44.8200274Z return super().run(initial_env=env) 2024-06-26T08:57:44.8201103Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 146, in run 2024-06-26T08:57:44.8201874Z self.env[node] = self.run_node(node) 2024-06-26T08:57:44.8202802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/constant_folding.py", line 162, in run_node 2024-06-26T08:57:44.8203633Z out = super().run_node(node) 2024-06-26T08:57:44.8204463Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 203, in run_node 2024-06-26T08:57:44.8205364Z return getattr(self, n.op)(n.target, args, kwargs) 2024-06-26T08:57:44.8206324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 275, in call_function 2024-06-26T08:57:44.8207133Z return target(*args, **kwargs) 2024-06-26T08:57:44.8207902Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 670, in __call__ 2024-06-26T08:57:44.8208630Z return self_._op(*args, **kwargs) 2024-06-26T08:57:44.8209364Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised: 2024-06-26T08:57:44.8212628Z OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 6.44 MiB is free. Process 4834 has 21.96 GiB memory in use. Of the allocated memory 20.82 GiB is allocated by PyTorch, and 802.08 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-06-26T08:57:44.8215253Z 2024-06-26T08:57:44.8216189Z While executing %full_1658 : [num_users=0] = call_function[target=torch.ops.aten.full.default](args = ([4, 12, 16, 64, 64], 0), kwargs = {dtype: torch.float16, layout: torch.strided, device: cuda:0, pin_memory: False}) 2024-06-26T08:57:44.8217374Z Original traceback: 2024-06-26T08:57:44.8218450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in torch_dynamo_resume_in_forward_and_backward_pass_at_451 2024-06-26T08:57:44.8219508Z pred = mod(*cloned_inputs) 2024-06-26T08:57:44.8220512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2450, in forward 2024-06-26T08:57:44.8221429Z outputs = self.bert( 2024-06-26T08:57:44.8222460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2133, in forward 2024-06-26T08:57:44.8223375Z encoder_outputs = self.encoder( 2024-06-26T08:57:44.8224385Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1631, in forward 2024-06-26T08:57:44.8225358Z layer_outputs = layer_module( 2024-06-26T08:57:44.8226347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1488, in forward 2024-06-26T08:57:44.8227290Z self_attention_outputs = self.attention( 2024-06-26T08:57:44.8228332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1401, in forward 2024-06-26T08:57:44.8229236Z self_outputs = self.self( 2024-06-26T08:57:44.8230206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 472, in forward 2024-06-26T08:57:44.8231274Z context_layer, attention_probs = self.bigbird_block_sparse_attention( 2024-06-26T08:57:44.8232561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 826, in bigbird_block_sparse_attention 2024-06-26T08:57:44.8234077Z last_product = self.torch_bmm_nd_transpose(blocked_query_matrix[:, :, -1], key_layer, ndim=4) 2024-06-26T08:57:44.8234693Z 2024-06-26T08:57:44.8234698Z 2024-06-26T08:57:44.8235021Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information 2024-06-26T08:57:44.8235512Z 2024-06-26T08:57:44.8235526Z 2024-06-26T08:57:44.8235842Z You can suppress this exception and fall back to eager by setting: 2024-06-26T08:57:44.8236453Z import torch._dynamo 2024-06-26T08:57:44.8236881Z torch._dynamo.config.suppress_errors = True 2024-06-26T08:57:44.8237226Z 2024-06-26T08:57:44.8237526Z TorchDynamo optimized model failed to run because of following error 2024-06-26T08:57:44.8238099Z fail_to_run 2024-06-26T08:57:45.1736985Z TIMING: entire_frame_compile:0.02134 2024-06-26T08:57:45.1738301Z STATS: call_* op count: 3349 | FakeTensorMode.__torch_dispatch__:149414 | FakeTensor.__torch_dispatch__:3 | ProxyTorchDispatchMode.__torch_dispatch__:74544 2024-06-26T08:57:45.1739415Z Dynamo produced 0 graphs covering 3349 ops with 3 graph breaks (3 unique) 2024-06-26T08:57:49.5241522Z 2024-06-26T08:57:51.4162849Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:57:51.4163457Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:57:51.4164017Z cuda train hf_DistilBert 2024-06-26T08:58:12.8323026Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:58:12.8324222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in torch_dynamo_resume_in_forward_and_backward_pass_at_451 2024-06-26T08:58:12.8327378Z pred = mod(*cloned_inputs) 2024-06-26T08:58:12.8328531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:58:12.8329542Z return forward_call(*args, **kwargs) 2024-06-26T08:58:12.8330712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 905, in forward 2024-06-26T08:58:12.8331809Z dlbrt_output = self.distilbert( 2024-06-26T08:58:12.8332786Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:58:12.8333706Z return forward_call(*args, **kwargs) 2024-06-26T08:58:12.8334930Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2024-06-26T08:58:12.8336396Z embeddings = self.embeddings(input_ids, inputs_embeds) # (bs, seq_length, dim) 2024-06-26T08:58:12.8337999Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:58:12.8338910Z return forward_call(*args, **kwargs) 2024-06-26T08:58:12.8340178Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 141, in forward 2024-06-26T08:58:12.8341431Z input_embeds = self.word_embeddings(input_ids) # (bs, max_seq_length, dim) 2024-06-26T08:58:12.8341942Z 2024-06-26T08:58:12.9001202Z W0626 08:58:12.899000 140458793984640 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:58:40.0381703Z pass 2024-06-26T08:58:40.0629676Z TIMING: entire_frame_compile:36.29101 code_gen:14.40638 inductor_compile:24.85017 backend_compile:30.9183 2024-06-26T08:58:40.0632016Z STATS: call_* op count: 540 | FakeTensor.__torch_dispatch__:9366 | FakeTensorMode.__torch_dispatch__:57725 | ProxyTorchDispatchMode.__torch_dispatch__:14154 2024-06-26T08:58:40.0633251Z Dynamo produced 3 graphs covering 540 ops with 6 graph breaks (5 unique) 2024-06-26T08:58:45.3278226Z 2024-06-26T08:58:48.3151764Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:58:48.3152504Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:58:48.3153169Z cuda train hf_GPT2 2024-06-26T08:59:23.1477590Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:59:23.1479481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in torch_dynamo_resume_in_forward_and_backward_pass_at_451 2024-06-26T08:59:23.1480468Z pred = mod(*cloned_inputs) 2024-06-26T08:59:23.1481483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:59:23.1482351Z return forward_call(*args, **kwargs) 2024-06-26T08:59:23.1483353Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1074, in forward 2024-06-26T08:59:23.1484263Z transformer_outputs = self.transformer( 2024-06-26T08:59:23.1485376Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:59:23.1486386Z return forward_call(*args, **kwargs) 2024-06-26T08:59:23.1487655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 837, in forward 2024-06-26T08:59:23.1488622Z inputs_embeds = self.wte(input_ids) 2024-06-26T08:59:23.1489080Z 2024-06-26T08:59:23.2153454Z W0626 08:59:23.214000 139975070794368 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:59:59.8181736Z pass 2024-06-26T08:59:59.8951919Z TIMING: entire_frame_compile:54.05234 code_gen:20.21844 inductor_compile:36.76288 backend_compile:45.13574 2024-06-26T08:59:59.8954238Z STATS: call_* op count: 1088 | FakeTensorMode.__torch_dispatch__:86424 | FakeTensor.__torch_dispatch__:13666 | ProxyTorchDispatchMode.__torch_dispatch__:21914 2024-06-26T08:59:59.8955403Z Dynamo produced 3 graphs covering 1088 ops with 6 graph breaks (5 unique) 2024-06-26T09:00:06.1766220Z 2024-06-26T09:00:19.0491905Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:00:19.0493086Z loading model: 0it [00:12, ?it/s] 2024-06-26T09:00:19.0494118Z cuda train hf_GPT2_large 2024-06-26T09:00:19.0495088Z pass_due_to_skip 2024-06-26T09:00:19.5458028Z TIMING: 2024-06-26T09:00:19.5458368Z STATS: call_* op count: 0 2024-06-26T09:00:19.5461372Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T09:00:22.8212607Z 2024-06-26T09:00:24.2684514Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:00:24.2685042Z loading model: 0it [00:01, ?it/s] 2024-06-26T09:00:24.2685856Z cuda train hf_Reformer 2024-06-26T09:00:41.4698389Z skipping cudagraphs due to ['incompatible ops'] 2024-06-26T09:00:44.0507476Z skipping cudagraphs due to ['incompatible ops'] 2024-06-26T09:00:45.4249830Z skipping cudagraphs due to ['incompatible ops'] 2024-06-26T09:00:47.7923625Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:00:47.7926493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 2094, in torch_dynamo_resume_in_forward_at_2066 2024-06-26T09:00:47.7927918Z embedding_output = self.embeddings( 2024-06-26T09:00:47.7929347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 254, in forward 2024-06-26T09:00:47.7930434Z inputs_embeds = self.word_embeddings(input_ids) 2024-06-26T09:00:47.7931373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward 2024-06-26T09:00:47.7932170Z return F.embedding( 2024-06-26T09:00:47.7932398Z 2024-06-26T09:00:47.7986688Z W0626 09:00:47.798000 140413512622720 torch/_logging/_internal.py:1040] [32/0] Profiler function will be ignored 2024-06-26T09:01:15.3924628Z pass 2024-06-26T09:01:15.4043384Z TIMING: entire_frame_compile:34.30851 inductor_compile:21.58183 backend_compile:28.29221 code_gen:14.85353 2024-06-26T09:01:15.4044674Z STATS: call_* op count: 688 | FakeTensorMode.__torch_dispatch__:37325 | ProxyTorchDispatchMode.__torch_dispatch__:8312 | FakeTensor.__torch_dispatch__:4968 2024-06-26T09:01:15.4047769Z Dynamo produced 17 graphs covering 688 ops with 25 graph breaks (9 unique) 2024-06-26T09:01:20.0920720Z 2024-06-26T09:01:24.1789198Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:01:24.1789815Z loading model: 0it [00:04, ?it/s] 2024-06-26T09:01:24.1790379Z cuda train hf_T5_base 2024-06-26T09:01:27.3138060Z WARNING:common:fp64 golden ref were not generated for hf_T5_base. Setting accuracy check to cosine 2024-06-26T09:01:29.4200668Z ERROR:common: 2024-06-26T09:01:29.4201067Z Traceback (most recent call last): 2024-06-26T09:01:29.4201816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2564, in check_accuracy 2024-06-26T09:01:29.4202598Z correct_rerun_result = self.run_n_iterations( 2024-06-26T09:01:29.4204681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2359, in run_n_iterations 2024-06-26T09:01:29.4205540Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-06-26T09:01:29.4206462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 458, in forward_and_backward_pass 2024-06-26T09:01:29.4207300Z self.grad_scaler.scale(loss).backward() 2024-06-26T09:01:29.4208264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward 2024-06-26T09:01:29.4209166Z torch.autograd.backward( 2024-06-26T09:01:29.4210015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 288, in backward 2024-06-26T09:01:29.4210782Z _engine_run_backward( 2024-06-26T09:01:29.4211656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 768, in _engine_run_backward 2024-06-26T09:01:29.4212818Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-06-26T09:01:29.4216324Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 192.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 88.44 MiB is free. Process 6208 has 21.88 GiB memory in use. Of the allocated memory 20.95 GiB is allocated by PyTorch, and 595.70 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-06-26T09:01:29.5083025Z eager_2nd_run_OOM 2024-06-26T09:01:29.5116449Z TIMING: 2024-06-26T09:01:29.5116757Z STATS: call_* op count: 0 2024-06-26T09:01:29.5117298Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T09:01:32.7464377Z 2024-06-26T09:01:42.4021328Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:01:42.4021796Z loading model: 0it [00:09, ?it/s] 2024-06-26T09:01:42.4022584Z cuda train hf_T5_large 2024-06-26T09:01:42.4025211Z pass_due_to_skip 2024-06-26T09:01:42.8693520Z TIMING: 2024-06-26T09:01:42.8693965Z STATS: call_* op count: 0 2024-06-26T09:01:42.8709047Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T09:01:46.0991681Z 2024-06-26T09:01:47.1368456Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:01:47.1369087Z loading model: 0it [00:01, ?it/s] 2024-06-26T09:01:47.1369620Z cuda train hf_Whisper 2024-06-26T09:02:05.9800464Z W0626 09:02:05.979000 139772792488576 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T09:02:26.2196622Z pass 2024-06-26T09:02:26.2302798Z TIMING: entire_frame_compile:28.50229 code_gen:14.72206 inductor_compile:21.72717 backend_compile:24.8104 2024-06-26T09:02:26.2304689Z STATS: call_* op count: 347 | FakeTensor.__torch_dispatch__:6457 | FakeTensorMode.__torch_dispatch__:35991 | ProxyTorchDispatchMode.__torch_dispatch__:8630 2024-06-26T09:02:26.2305820Z Dynamo produced 3 graphs covering 347 ops with 6 graph breaks (5 unique) 2024-06-26T09:02:30.9317212Z 2024-06-26T09:02:31.2031460Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:02:31.2031953Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:02:31.2032404Z cuda train hf_distil_whisper 2024-06-26T09:02:31.2036806Z Traceback (most recent call last): 2024-06-26T09:02:31.2037591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T09:02:31.2038457Z ) = runner.load_model( 2024-06-26T09:02:31.2041221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 320, in load_model 2024-06-26T09:02:31.2042092Z benchmark = benchmark_cls( 2024-06-26T09:02:31.2043143Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-06-26T09:02:31.2044076Z obj = type.__call__(cls, *args, **kwargs) 2024-06-26T09:02:31.2045064Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper/__init__.py", line 12, in __init__ 2024-06-26T09:02:31.2046061Z raise NotImplementedError("Training is not implemented.") 2024-06-26T09:02:31.2046657Z NotImplementedError: Training is not implemented. 2024-06-26T09:02:31.2047006Z 2024-06-26T09:02:31.2047129Z model_fail_to_load 2024-06-26T09:02:34.3257726Z 2024-06-26T09:02:34.7090171Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:02:34.7090867Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:02:34.7091600Z cuda train lennard_jones 2024-06-26T09:02:39.5165825Z W0626 09:02:39.515000 139785333973632 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T09:02:43.1810268Z pass 2024-06-26T09:02:43.1810985Z TIMING: entire_frame_compile:6.52624 code_gen:2.11665 inductor_compile:3.00679 backend_compile:5.94304 2024-06-26T09:02:43.1812264Z STATS: call_* op count: 61 | FakeTensor.__torch_dispatch__:872 | FakeTensorMode.__torch_dispatch__:5191 | ProxyTorchDispatchMode.__torch_dispatch__:1238 2024-06-26T09:02:43.1813365Z Dynamo produced 3 graphs covering 61 ops with 7 graph breaks (5 unique) 2024-06-26T09:02:44.2212881Z accuracy pass_rate=78.79% 2024-06-26T09:02:44.2214148Z calls_captured gmean=0.00x mean=429.061x 2024-06-26T09:02:44.2217792Z unique_graphs gmean=0.00x mean=3.212x 2024-06-26T09:02:44.2222013Z graph_breaks gmean=0.00x mean=6.455x 2024-06-26T09:02:44.2225397Z unique_graph_breaks gmean=0.00x mean=4.273x 2024-06-26T09:02:44.2229249Z autograd_captures gmean=0.00x mean=0.000x 2024-06-26T09:02:44.2232960Z autograd_compiles gmean=0.00x mean=0.000x 2024-06-26T09:02:44.2236845Z cudagraph_skips gmean=0.00x mean=0.697x 2024-06-26T09:02:45.1094191Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy//inductor_torchbench_training.csv 2024-06-26T09:02:45.4355661Z torchrec_dlrm PASS 2024-06-26T09:02:45.4359912Z BERT_pytorch PASS 2024-06-26T09:02:45.4364260Z Background_Matting XFAIL 2024-06-26T09:02:45.4368647Z LearningToPaint PASS 2024-06-26T09:02:45.4373205Z Super_SloMo PASS 2024-06-26T09:02:45.4377849Z alexnet PASS 2024-06-26T09:02:45.4382211Z basic_gnn_edgecnn PASS 2024-06-26T09:02:45.4386524Z basic_gnn_gcn PASS 2024-06-26T09:02:45.4391164Z basic_gnn_gin PASS 2024-06-26T09:02:45.4395192Z basic_gnn_sage PASS 2024-06-26T09:02:45.4399519Z dcgan PASS 2024-06-26T09:02:45.4404031Z demucs PASS 2024-06-26T09:02:45.4408188Z densenet121 PASS 2024-06-26T09:02:45.4412848Z detectron2_maskrcnn_r_50_c4 XFAIL 2024-06-26T09:02:45.4417108Z dlrm PASS 2024-06-26T09:02:45.4421830Z drq PASS 2024-06-26T09:02:45.4426211Z fastNLP_Bert PASS 2024-06-26T09:02:45.4430390Z functorch_dp_cifar10 PASS 2024-06-26T09:02:45.4434721Z functorch_maml_omniglot PASS 2024-06-26T09:02:45.4439072Z hf_Albert PASS 2024-06-26T09:02:45.4443325Z hf_Bart PASS 2024-06-26T09:02:45.4447775Z hf_Bert PASS 2024-06-26T09:02:45.4452536Z hf_Bert_large PASS 2024-06-26T09:02:45.4456730Z hf_BigBird XFAIL 2024-06-26T09:02:45.4461114Z hf_DistilBert PASS 2024-06-26T09:02:45.4465570Z hf_GPT2 PASS 2024-06-26T09:02:45.4469796Z hf_GPT2_large XFAIL 2024-06-26T09:02:45.4474233Z hf_Reformer PASS 2024-06-26T09:02:45.4478509Z hf_T5_base XFAIL 2024-06-26T09:02:45.4482733Z hf_T5_large XFAIL 2024-06-26T09:02:45.4487007Z hf_Whisper PASS 2024-06-26T09:02:45.4491660Z hf_distil_whisper XFAIL 2024-06-26T09:02:45.4495725Z lennard_jones PASS 2024-06-26T09:02:45.4941025Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy//inductor_torchbench_training.csv 2024-06-26T09:02:45.8023221Z torchrec_dlrm PASS 2024-06-26T09:02:45.8029149Z BERT_pytorch PASS 2024-06-26T09:02:45.8032335Z Background_Matting PASS 2024-06-26T09:02:45.8036331Z LearningToPaint PASS 2024-06-26T09:02:45.8041042Z Super_SloMo PASS 2024-06-26T09:02:45.8045288Z alexnet PASS 2024-06-26T09:02:45.8049973Z basic_gnn_edgecnn PASS 2024-06-26T09:02:45.8054379Z basic_gnn_gcn PASS 2024-06-26T09:02:45.8059154Z basic_gnn_gin PASS 2024-06-26T09:02:45.8063375Z basic_gnn_sage PASS 2024-06-26T09:02:45.8067753Z dcgan PASS 2024-06-26T09:02:45.8072405Z demucs PASS 2024-06-26T09:02:45.8077032Z densenet121 PASS 2024-06-26T09:02:45.8081638Z detectron2_maskrcnn_r_50_c4 PASS 2024-06-26T09:02:45.8086191Z dlrm PASS 2024-06-26T09:02:45.8091034Z drq PASS 2024-06-26T09:02:45.8095700Z fastNLP_Bert PASS 2024-06-26T09:02:45.8100424Z functorch_dp_cifar10 PASS 2024-06-26T09:02:45.8104936Z functorch_maml_omniglot PASS 2024-06-26T09:02:45.8109500Z hf_Albert PASS 2024-06-26T09:02:45.8114160Z hf_Bart PASS 2024-06-26T09:02:45.8118889Z hf_Bert PASS 2024-06-26T09:02:45.8123354Z hf_Bert_large PASS 2024-06-26T09:02:45.8127893Z hf_BigBird PASS 2024-06-26T09:02:45.8132938Z hf_DistilBert PASS 2024-06-26T09:02:45.8137770Z hf_GPT2 PASS 2024-06-26T09:02:45.8142315Z hf_GPT2_large PASS 2024-06-26T09:02:45.8146911Z hf_Reformer PASS 2024-06-26T09:02:45.8151455Z hf_T5_base PASS 2024-06-26T09:02:45.8156062Z hf_T5_large PASS 2024-06-26T09:02:45.8160599Z hf_Whisper PASS 2024-06-26T09:02:45.8165157Z hf_distil_whisper PASS 2024-06-26T09:02:45.8169747Z lennard_jones PASS 2024-06-26T09:02:45.8623317Z + cleanup_workspace 2024-06-26T09:02:45.8624372Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-06-26T09:02:45.8625472Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-06-26T09:02:45.8626405Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-06-26T09:02:45.8627054Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-06-26T09:02:45.8627863Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-06-26T09:02:45.8628737Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-06-26T09:02:45.8629431Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-06-26T09:02:46.5456472Z ##[group]Run cat test/**/*_toprint.log || true 2024-06-26T09:02:46.5457001Z cat test/**/*_toprint.log || true 2024-06-26T09:02:46.5465798Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:02:46.5466292Z env: 2024-06-26T09:02:46.5466562Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:02:46.5467008Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:02:46.5467720Z DOCKER_CONTAINER_ID: 604b66743ea1a58a12a4e607c457b971331d736afd449b38895678368be0fe52 2024-06-26T09:02:46.5468403Z ##[endgroup] 2024-06-26T09:02:46.5530687Z cat: test/**/*_toprint.log: No such file or directory 2024-06-26T09:02:46.5557453Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-06-26T09:02:46.5557946Z kill "$MONITOR_SCRIPT_PID" 2024-06-26T09:02:46.5564747Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:02:46.5565249Z env: 2024-06-26T09:02:46.5565519Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:02:46.5565982Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:02:46.5566720Z DOCKER_CONTAINER_ID: 604b66743ea1a58a12a4e607c457b971331d736afd449b38895678368be0fe52 2024-06-26T09:02:46.5567359Z MONITOR_SCRIPT_PID: 12738 2024-06-26T09:02:46.5567709Z ##[endgroup] 2024-06-26T09:02:46.5754627Z Prepare all required actions 2024-06-26T09:02:46.5755079Z Getting action download info 2024-06-26T09:02:46.7523046Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-06-26T09:02:46.8970181Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-06-26T09:02:46.8970653Z with: 2024-06-26T09:02:46.8971190Z file-suffix: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688911389 2024-06-26T09:02:46.8971847Z s3-bucket: gha-artifacts 2024-06-26T09:02:46.8972185Z env: 2024-06-26T09:02:46.8972455Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:02:46.8972916Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:02:46.8973639Z DOCKER_CONTAINER_ID: 604b66743ea1a58a12a4e607c457b971331d736afd449b38895678368be0fe52 2024-06-26T09:02:46.8974417Z ##[endgroup] 2024-06-26T09:02:46.8999124Z ##[group]Run # Remove any previous test jsons if they exist 2024-06-26T09:02:46.8999773Z # Remove any previous test jsons if they exist 2024-06-26T09:02:46.9000288Z rm -f test-jsons-*.zip 2024-06-26T09:02:46.9000822Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-06-26T09:02:46.9008657Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:02:46.9009299Z env: 2024-06-26T09:02:46.9009583Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:02:46.9010036Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:02:46.9010755Z DOCKER_CONTAINER_ID: 604b66743ea1a58a12a4e607c457b971331d736afd449b38895678368be0fe52 2024-06-26T09:02:46.9011623Z FILE_SUFFIX: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688911389 2024-06-26T09:02:46.9012260Z ##[endgroup] 2024-06-26T09:02:46.9187466Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-06-26T09:02:46.9216696Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-06-26T09:02:46.9217735Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-06-26T09:02:46.9223592Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-06-26T09:02:46.9248634Z ##[group]Run # Remove any previous test reports if they exist 2024-06-26T09:02:46.9249463Z # Remove any previous test reports if they exist 2024-06-26T09:02:46.9250000Z rm -f test-reports-*.zip 2024-06-26T09:02:46.9250601Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-06-26T09:02:46.9258680Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:02:46.9259190Z env: 2024-06-26T09:02:46.9259481Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:02:46.9259949Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:02:46.9260673Z DOCKER_CONTAINER_ID: 604b66743ea1a58a12a4e607c457b971331d736afd449b38895678368be0fe52 2024-06-26T09:02:46.9261578Z FILE_SUFFIX: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688911389 2024-06-26T09:02:46.9262230Z ##[endgroup] 2024-06-26T09:02:46.9427708Z adding: test/test-reports/inference_torchbench.csv (deflated 73%) 2024-06-26T09:02:46.9432228Z adding: test/test-reports/inference_torchbench_graph_breaks.csv (deflated 92%) 2024-06-26T09:02:46.9434141Z adding: test/test-reports/inference_torchbench_graph_break_deduped.csv (deflated 79%) 2024-06-26T09:02:46.9435015Z adding: test/test-reports/training_torchbench.csv (deflated 63%) 2024-06-26T09:02:46.9440838Z adding: test/test-reports/training_torchbench_graph_breaks.csv (deflated 96%) 2024-06-26T09:02:46.9441893Z adding: test/test-reports/training_torchbench_graph_break_deduped.csv (deflated 78%) 2024-06-26T09:02:46.9469209Z ##[group]Run # Remove any previous usage logs if they exist 2024-06-26T09:02:46.9469847Z # Remove any previous usage logs if they exist 2024-06-26T09:02:46.9470349Z rm -f logs-*.zip 2024-06-26T09:02:46.9471036Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-06-26T09:02:46.9471957Z # so check to see if the file exists first 2024-06-26T09:02:46.9472475Z if [ -f 'usage_log.txt' ]; then 2024-06-26T09:02:46.9473024Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-06-26T09:02:46.9473513Z fi 2024-06-26T09:02:46.9473889Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-06-26T09:02:46.9474467Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-06-26T09:02:46.9474967Z fi 2024-06-26T09:02:46.9482389Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:02:46.9482892Z env: 2024-06-26T09:02:46.9483166Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:02:46.9483634Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:02:46.9484367Z DOCKER_CONTAINER_ID: 604b66743ea1a58a12a4e607c457b971331d736afd449b38895678368be0fe52 2024-06-26T09:02:46.9485386Z FILE_SUFFIX: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688911389 2024-06-26T09:02:46.9486023Z ##[endgroup] 2024-06-26T09:02:46.9570599Z adding: usage_log.txt (deflated 92%) 2024-06-26T09:02:46.9639206Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-06-26T09:02:46.9639945Z # Remove any previous debugging artifacts if they exist 2024-06-26T09:02:46.9640514Z rm -f debug-*.zip 2024-06-26T09:02:46.9640922Z if [ -d 'test/debug' ]; then 2024-06-26T09:02:46.9641430Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-06-26T09:02:46.9641925Z fi 2024-06-26T09:02:46.9649362Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:02:46.9649864Z env: 2024-06-26T09:02:46.9650141Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:02:46.9650604Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:02:46.9651335Z DOCKER_CONTAINER_ID: 604b66743ea1a58a12a4e607c457b971331d736afd449b38895678368be0fe52 2024-06-26T09:02:46.9652236Z FILE_SUFFIX: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688911389 2024-06-26T09:02:46.9652877Z ##[endgroup] 2024-06-26T09:02:46.9729707Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T09:02:46.9730154Z with: 2024-06-26T09:02:46.9730458Z s3-bucket: gha-artifacts 2024-06-26T09:02:46.9730892Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T09:02:46.9731367Z retention-days: 14 2024-06-26T09:02:46.9731701Z if-no-files-found: warn 2024-06-26T09:02:46.9732063Z path: test-jsons-*.zip 2024-06-26T09:02:46.9732409Z name: artifact 2024-06-26T09:02:46.9732709Z region: us-east-1 2024-06-26T09:02:46.9733016Z env: 2024-06-26T09:02:46.9733305Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:02:46.9733772Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:02:46.9734502Z DOCKER_CONTAINER_ID: 604b66743ea1a58a12a4e607c457b971331d736afd449b38895678368be0fe52 2024-06-26T09:02:46.9735143Z ##[endgroup] 2024-06-26T09:02:47.2850625Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T09:02:47.2851232Z With the provided path, there will be 1 file uploaded 2024-06-26T09:02:47.2851856Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T09:02:47.2877713Z Starting upload of test-jsons-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688911389.zip 2024-06-26T09:02:47.4347422Z Finished upload of test-jsons-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688911389.zip 2024-06-26T09:02:47.4500315Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T09:02:47.4500878Z with: 2024-06-26T09:02:47.4501238Z s3-bucket: gha-artifacts 2024-06-26T09:02:47.4501764Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T09:02:47.4502350Z retention-days: 14 2024-06-26T09:02:47.4502766Z if-no-files-found: error 2024-06-26T09:02:47.4503228Z path: test-reports-*.zip 2024-06-26T09:02:47.4503668Z name: artifact 2024-06-26T09:02:47.4504053Z region: us-east-1 2024-06-26T09:02:47.4504439Z env: 2024-06-26T09:02:47.4504789Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:02:47.4505356Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:02:47.4506426Z DOCKER_CONTAINER_ID: 604b66743ea1a58a12a4e607c457b971331d736afd449b38895678368be0fe52 2024-06-26T09:02:47.4507225Z ##[endgroup] 2024-06-26T09:02:47.7633211Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T09:02:47.7633838Z With the provided path, there will be 1 file uploaded 2024-06-26T09:02:47.7634462Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T09:02:47.7660674Z Starting upload of test-reports-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688911389.zip 2024-06-26T09:02:47.9001494Z Finished upload of test-reports-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688911389.zip 2024-06-26T09:02:47.9161375Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T09:02:47.9161819Z with: 2024-06-26T09:02:47.9162088Z s3-bucket: gha-artifacts 2024-06-26T09:02:47.9162673Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T09:02:47.9163173Z retention-days: 14 2024-06-26T09:02:47.9163515Z if-no-files-found: ignore 2024-06-26T09:02:47.9163875Z path: logs-*.zip 2024-06-26T09:02:47.9164254Z name: artifact 2024-06-26T09:02:47.9164612Z region: us-east-1 2024-06-26T09:02:47.9164902Z env: 2024-06-26T09:02:47.9165176Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:02:47.9165634Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:02:47.9166356Z DOCKER_CONTAINER_ID: 604b66743ea1a58a12a4e607c457b971331d736afd449b38895678368be0fe52 2024-06-26T09:02:47.9166974Z ##[endgroup] 2024-06-26T09:02:48.2269669Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T09:02:48.2270285Z With the provided path, there will be 1 file uploaded 2024-06-26T09:02:48.2270889Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T09:02:48.2296465Z Starting upload of logs-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688911389.zip 2024-06-26T09:02:48.4419125Z Finished upload of logs-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688911389.zip 2024-06-26T09:02:48.4560875Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T09:02:48.4561315Z with: 2024-06-26T09:02:48.4561605Z s3-bucket: gha-artifacts 2024-06-26T09:02:48.4562035Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T09:02:48.4562508Z retention-days: 14 2024-06-26T09:02:48.4562847Z if-no-files-found: ignore 2024-06-26T09:02:48.4563208Z path: debug-*.zip 2024-06-26T09:02:48.4563521Z name: artifact 2024-06-26T09:02:48.4563825Z region: us-east-1 2024-06-26T09:02:48.4564127Z env: 2024-06-26T09:02:48.4564409Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:02:48.4564866Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:02:48.4565586Z DOCKER_CONTAINER_ID: 604b66743ea1a58a12a4e607c457b971331d736afd449b38895678368be0fe52 2024-06-26T09:02:48.4566215Z ##[endgroup] 2024-06-26T09:02:48.7635789Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-06-26T09:02:48.7774523Z ##[group]Run # shellcheck disable=SC2156 2024-06-26T09:02:48.7775007Z # shellcheck disable=SC2156 2024-06-26T09:02:48.7775793Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-06-26T09:02:48.7784127Z shell: /usr/bin/bash -e {0} 2024-06-26T09:02:48.7784479Z env: 2024-06-26T09:02:48.7784797Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:02:48.7785271Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:02:48.7785991Z DOCKER_CONTAINER_ID: 604b66743ea1a58a12a4e607c457b971331d736afd449b38895678368be0fe52 2024-06-26T09:02:48.7786617Z ##[endgroup] 2024-06-26T09:02:48.9444309Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-06-26T09:02:48.9444868Z with: 2024-06-26T09:02:48.9445131Z env: 2024-06-26T09:02:48.9445409Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:02:48.9445862Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:02:48.9446607Z DOCKER_CONTAINER_ID: 604b66743ea1a58a12a4e607c457b971331d736afd449b38895678368be0fe52 2024-06-26T09:02:48.9447233Z ##[endgroup] 2024-06-26T09:02:48.9465954Z ##[group]Run set -eou pipefail 2024-06-26T09:02:48.9466521Z set -eou pipefail 2024-06-26T09:02:48.9466994Z  2024-06-26T09:02:48.9467513Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-06-26T09:02:48.9468149Z for _ in $(seq 1440); do 2024-06-26T09:02:48.9468631Z  # Break if no ssh session exists anymore 2024-06-26T09:02:48.9469138Z  if [ "$(who)" = "" ]; then 2024-06-26T09:02:48.9469549Z  break 2024-06-26T09:02:48.9469899Z  fi 2024-06-26T09:02:48.9470198Z  echo "." 2024-06-26T09:02:48.9470530Z  sleep 5 2024-06-26T09:02:48.9470853Z done 2024-06-26T09:02:48.9478309Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:02:48.9478942Z env: 2024-06-26T09:02:48.9479227Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:02:48.9479682Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:02:48.9480401Z DOCKER_CONTAINER_ID: 604b66743ea1a58a12a4e607c457b971331d736afd449b38895678368be0fe52 2024-06-26T09:02:48.9481021Z ##[endgroup] 2024-06-26T09:02:48.9505981Z Holding runner for 2 hours until all ssh sessions have logged out 2024-06-26T09:02:48.9541083Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T09:02:48.9541839Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T09:02:48.9542428Z # shellcheck disable=SC2046 2024-06-26T09:02:48.9542902Z docker stop $(docker ps -q) || true 2024-06-26T09:02:48.9543385Z # Prune all of the docker images 2024-06-26T09:02:48.9543850Z docker system prune -af 2024-06-26T09:02:48.9550773Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:02:48.9551271Z env: 2024-06-26T09:02:48.9551558Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:02:48.9552021Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:02:48.9552754Z DOCKER_CONTAINER_ID: 604b66743ea1a58a12a4e607c457b971331d736afd449b38895678368be0fe52 2024-06-26T09:02:48.9553391Z ##[endgroup] 2024-06-26T09:02:49.3427742Z 604b66743ea1 2024-06-26T09:02:51.6127957Z Deleted Containers: 2024-06-26T09:02:51.6128496Z 604b66743ea1a58a12a4e607c457b971331d736afd449b38895678368be0fe52 2024-06-26T09:02:51.6129051Z 2024-06-26T09:02:55.5347611Z Deleted Images: 2024-06-26T09:02:55.5349078Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T09:02:55.5351350Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T09:02:55.5352877Z deleted: sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd 2024-06-26T09:02:55.5353717Z deleted: sha256:754ead7012f0b7691885a9660f02e80be6cee8c5665486fb995c1fb06adb6e92 2024-06-26T09:02:55.5354548Z deleted: sha256:fbe4496d8830e11be86d8f222522e7956f32f85bfc06b05ea53961f34a58af74 2024-06-26T09:02:55.5355381Z deleted: sha256:1c83814aa1a1a7e0cc998e3479bdab8e8766b975560cbf00690acf39ed899a80 2024-06-26T09:02:55.5356222Z deleted: sha256:e9ea8e61942c7f8ae4aab97b8d102b01d6bb4b6cf2a605808cd34130a17d8a0f 2024-06-26T09:02:55.5357107Z deleted: sha256:acc5fc32588cd95e082a6ce35e48d0f303ae8f96e3ab65c4e375b85e1f3c4a92 2024-06-26T09:02:55.5357934Z deleted: sha256:a93e53400904d29e4185d3be942ea34f19572ea438962788ae4e9b3d9bf4c498 2024-06-26T09:02:55.5358939Z deleted: sha256:c97092b88b5e4c77c68bbc4b51afab64e38719af1ff58013ec49a696cdd40a6e 2024-06-26T09:02:55.5359878Z deleted: sha256:b132889c90d67ffa53655846603e3fcc2372c23f04dbdbc54690a39d304eb89b 2024-06-26T09:02:55.5360785Z deleted: sha256:915d54081ed607710624abe3f2490114c745858831f4fc13396bd3b4b9a121bc 2024-06-26T09:02:55.5361690Z deleted: sha256:bfa8d446c498c87f535aef102695564de1b22c755e6c63944ae2f5073a2b31ec 2024-06-26T09:02:55.5362611Z deleted: sha256:e71460fb363c29e70bbe357d680aaa7ae694f094ef2f2713609158ad90611744 2024-06-26T09:02:55.5363524Z deleted: sha256:84079bb1b4ee1d0119ae3d3e701de460a1324912297c675425c9a36d30fcdb0f 2024-06-26T09:02:55.5364437Z deleted: sha256:d86ac3ca148d4758cb03b493fdd49109d8f1a2c7c85c4b755493c3c4529553e5 2024-06-26T09:02:55.5365348Z deleted: sha256:532b3ab97c44a673a0d5f62ee46fa6a83248a84de8877754a612355be56ed0a3 2024-06-26T09:02:55.5366263Z deleted: sha256:617e2a35f25ae24500c88c0c9b8475f0687fbcbed3d200013b4c99c6d8941aff 2024-06-26T09:02:55.5367191Z deleted: sha256:b88120bcd5a26a660c09c5df13176e0ef098fb2c8cdb57d74ab0796a3d1194d6 2024-06-26T09:02:55.5368104Z deleted: sha256:c016b2e7764735243205fb76fa9a273bb3dd4bf05532288e3a613ce022131ac3 2024-06-26T09:02:55.5369136Z deleted: sha256:bcaca462c390deddd2995de2d245a7495e7d2e1e94f0edad30a573c83d0588fd 2024-06-26T09:02:55.5370096Z deleted: sha256:9e260eba003d5c25d7b410e5c4c3093845d9e46611896840901c689d7fbf8ee6 2024-06-26T09:02:55.5370896Z deleted: sha256:3e62b86f8b911a9b59c62d55f063e71683ae6a01c22418db5d27930712adaad6 2024-06-26T09:02:55.5371705Z deleted: sha256:905920ccb1d88449b8dde686c34a4b2a8b6f59e78ac44916b189ea3593b9fbbb 2024-06-26T09:02:55.5372501Z deleted: sha256:961201237e03acd5061406238dd7a3c1343ac888d7583212cd0ab5ae1b4f0540 2024-06-26T09:02:55.5373318Z deleted: sha256:70047a2aac62159fa28fb4dbd4567ec2ffbab68fe046967e09acb0eb3e5cb9cb 2024-06-26T09:02:55.5374269Z deleted: sha256:1ec680ad781436ae65fa987f5fa1ed032ecd5c82a6e35eea6707de501f62c0f4 2024-06-26T09:02:55.5375267Z deleted: sha256:ef859623a50e85abf6e6c1dbd6188aa765a66cc7b2bd618a1cbe4825b7b2bd91 2024-06-26T09:02:55.5376105Z deleted: sha256:404ade5cacff59773b93ab438bfa64cd3a171d07f6f168bdc4a5d4bfa0b43526 2024-06-26T09:02:55.5376941Z deleted: sha256:5b27de6ee3d9f2f18217bf18dec725f7b4954c0e48c85d35752634c22d62209b 2024-06-26T09:02:55.5378078Z deleted: sha256:498e39c575b3fa069acaac7e46b91b4e54ae3a05f2ada5c4347091f346f25d4c 2024-06-26T09:02:55.5378881Z deleted: sha256:1405919d2694d65669b1f4c842282021afe89208ce86570e96464903733da2d0 2024-06-26T09:02:55.5379715Z deleted: sha256:d5b41e5d70896c0472fc3e6da0516a17d89cd6635dc75010ecddc31ce9690fad 2024-06-26T09:02:55.5380522Z deleted: sha256:18e9379b62b9d0bb40293d28a652eafaee66a499f66f0776bf14e37a513101b7 2024-06-26T09:02:55.5381343Z deleted: sha256:a781f80ed00ff2926a699fd01a309bb0b4fb7d5655eb652c205bb9c7f811035f 2024-06-26T09:02:55.5382156Z deleted: sha256:c0430973b7cf759075330ee29c79bfd1312c91e6033d9dd883b4b1c29743a171 2024-06-26T09:02:55.5382981Z deleted: sha256:e733b3d66efb9342e5a283376c0e1ddffa47a42fecc26e61ecaf0a3472885d97 2024-06-26T09:02:55.5383929Z deleted: sha256:8929c0b2047f74b84c82c39eff9077a92ae1a828cde00c9ab78fb3e0ed5e0283 2024-06-26T09:02:55.5385072Z deleted: sha256:f5b56155fb58a0a60fe6391d3581cf91610ed69d4f0a321257c19944da296a52 2024-06-26T09:02:55.5386206Z deleted: sha256:b54d2cb355238d1aa91e58df93064447b80f53706d1ca62823acc834f5ac54fe 2024-06-26T09:02:55.5387024Z deleted: sha256:e7311285af9eeddb51aa5629aa98386e89f1456ff32843b6aee6443247edf66c 2024-06-26T09:02:55.5387828Z deleted: sha256:67657ceec347da8dc8085d5731633efeac115ba677fa4b7c5966c4260f7abab0 2024-06-26T09:02:55.5388645Z deleted: sha256:b496e50409dbf414930fc9b4437aa0aa4ce27bc7840e050ae459aaffcf9aca86 2024-06-26T09:02:55.5389457Z deleted: sha256:8f5276170cc4a60d36ad7b964a502306b29546254cedcbcb101ea9123497d2c0 2024-06-26T09:02:55.5390253Z deleted: sha256:53d524d1564745e96a7e515462171b73184d9f1ff5454b32fcfbbca39d361467 2024-06-26T09:02:55.5391067Z deleted: sha256:2acbadb00d69b6aca575f04baba4ea26ae2b5c2d85f09945004ad93a78a8eb75 2024-06-26T09:02:55.5392030Z deleted: sha256:02665efeff878aeb603c1edb2e21cb700c171c025d64f5aae1b2b20b12f3964a 2024-06-26T09:02:55.5392879Z deleted: sha256:19bd23e9a51dfe883a8b8d2ff6d66ff1b3476b8912856406bb79b81762aa6d4b 2024-06-26T09:02:55.5393696Z deleted: sha256:73aa8637bf9299d6298dabd858480c274a62117388f2ca1d5e9eb15c6d1e4155 2024-06-26T09:02:55.5394504Z deleted: sha256:5f8968406305f348b1caebc0e035187d6e9965ace7b9d5f2f21114d37c9fe2e1 2024-06-26T09:02:55.5395304Z deleted: sha256:d827104914db0c503cf5c81f13e2456bf4efce42d4cf1f8d5d70838207278bee 2024-06-26T09:02:55.5396120Z deleted: sha256:867b219724f2f1aaf56f42e3e9f03a5d64bdca8a6a72407efd10328aeea5d510 2024-06-26T09:02:55.5397001Z deleted: sha256:b9ff2a7cceb9237b26bcb17728edf8a558eec5a76f63626e84f80541dfc0f1d4 2024-06-26T09:02:55.5397828Z deleted: sha256:be73f509106819007ce2b0f5a1301ac56ae583cb6c1fb69bee93efa0840368f1 2024-06-26T09:02:55.5398650Z deleted: sha256:2a11c07c915c3fdbb4e73096dea73ea950c970d50fe1a5764060bd62b7b7cf2c 2024-06-26T09:02:55.5399477Z deleted: sha256:5e9989622e2abc1c72f8184343507b45d75e4ff971edb68b96cbcd75ae484ff5 2024-06-26T09:02:55.5400313Z deleted: sha256:05ce642bd9cf5a9205bff107df28b1d6cef7d5f6ab3ea944d28c032ec61270df 2024-06-26T09:02:55.5401127Z deleted: sha256:017702617f6c07b73f2088678f4710b794cfe897a213a5cf122a0e2519eae0ac 2024-06-26T09:02:55.5402012Z deleted: sha256:44b3b37bb1ca39dc5cf356de41a23a5c12f605a1992813e408087fb2b0ad6d1c 2024-06-26T09:02:55.5402836Z deleted: sha256:8c387982e9f27db357a1d149dc149cabc0bd04668ad6956387a5072682dada67 2024-06-26T09:02:55.5403655Z deleted: sha256:10d415dca4e3fbb88a747b40317f3c6333c041ba8f73855f67fb9fe981fae5ee 2024-06-26T09:02:55.5404473Z deleted: sha256:6c65469bf0f53dcd625faefe363b8f3136c08388a6cf64d9158e8576e4ee89b1 2024-06-26T09:02:55.5405299Z deleted: sha256:027db5d8b76deeefbd8c34b485ab7ff14ed2a3e6800425c1ce0d77c2e9fdd4ab 2024-06-26T09:02:55.5406123Z deleted: sha256:24941234ad28cf5c8174d7cf18326c10b4e17f2c2137672ea0322df461597dd7 2024-06-26T09:02:55.5406947Z deleted: sha256:240ea2eebfa8519d6c2d88f4c99e66ce4e038d000f93ce8d03726c01d81180ef 2024-06-26T09:02:55.5407765Z deleted: sha256:09e823cb391a69f6a195e3ca8a41624b5f8b34eee14c6eb36277de814a91966e 2024-06-26T09:02:55.5408557Z deleted: sha256:730479d695169a6593bf0ac16b377491dce65175a6d158e012422367d04577fd 2024-06-26T09:02:55.5409470Z deleted: sha256:b0a30f9f7bd5459ff662453b9346a7ba8ea8db1191528345c1953eab4aa2f54e 2024-06-26T09:02:55.5410313Z deleted: sha256:75cefa87effaa3ad361c74642839923101f0f9982f814f3c9e2ed4ad52f2a008 2024-06-26T09:02:55.5411124Z deleted: sha256:6f97db87ba9512e615e871c6982000602982e1e5a5be334a56e7292251e84e00 2024-06-26T09:02:55.5411935Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-06-26T09:02:55.5412752Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-06-26T09:02:55.5413552Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-06-26T09:02:55.5414364Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-06-26T09:02:55.5415187Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-06-26T09:02:55.5415993Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-06-26T09:02:55.5416825Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-06-26T09:02:55.5417897Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-06-26T09:02:55.5418743Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-06-26T09:02:55.5419569Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-06-26T09:02:55.5420388Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-06-26T09:02:55.5420890Z 2024-06-26T09:02:55.5421047Z Total reclaimed space: 40.92GB 2024-06-26T09:02:55.5483224Z Post job cleanup. 2024-06-26T09:02:55.5545670Z Post job cleanup. 2024-06-26T09:02:55.6351219Z [command]/usr/bin/git version 2024-06-26T09:02:55.6384747Z git version 2.40.1 2024-06-26T09:02:55.6414261Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/aca18e7e-cce6-4b5a-b9f2-1a71281cb2b0/.gitconfig' 2024-06-26T09:02:55.6425190Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/aca18e7e-cce6-4b5a-b9f2-1a71281cb2b0' before making global git config changes 2024-06-26T09:02:55.6426462Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T09:02:55.6429632Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T09:02:55.6455731Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T09:02:55.6478195Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T09:02:55.6694630Z Entering 'android/libs/fbjni' 2024-06-26T09:02:55.6730002Z Entering 'third_party/FP16' 2024-06-26T09:02:55.6767262Z Entering 'third_party/FXdiv' 2024-06-26T09:02:55.6804225Z Entering 'third_party/NNPACK' 2024-06-26T09:02:55.6839811Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T09:02:55.6877016Z Entering 'third_party/XNNPACK' 2024-06-26T09:02:55.6924477Z Entering 'third_party/benchmark' 2024-06-26T09:02:55.6959493Z Entering 'third_party/cpp-httplib' 2024-06-26T09:02:55.6996463Z Entering 'third_party/cpuinfo' 2024-06-26T09:02:55.7033223Z Entering 'third_party/cudnn_frontend' 2024-06-26T09:02:55.7068552Z Entering 'third_party/cutlass' 2024-06-26T09:02:55.7111161Z Entering 'third_party/eigen' 2024-06-26T09:02:55.7147522Z Entering 'third_party/fbgemm' 2024-06-26T09:02:55.7182570Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T09:02:55.7217911Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T09:02:55.7252464Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T09:02:55.7292154Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T09:02:55.7327240Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T09:02:55.7363215Z Entering 'third_party/flatbuffers' 2024-06-26T09:02:55.7399535Z Entering 'third_party/fmt' 2024-06-26T09:02:55.7435096Z Entering 'third_party/foxi' 2024-06-26T09:02:55.7468494Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T09:02:55.7502466Z Entering 'third_party/gloo' 2024-06-26T09:02:55.7534638Z Entering 'third_party/googletest' 2024-06-26T09:02:55.7571019Z Entering 'third_party/ideep' 2024-06-26T09:02:55.7606862Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T09:02:55.7644068Z Entering 'third_party/ittapi' 2024-06-26T09:02:55.7678355Z Entering 'third_party/kineto' 2024-06-26T09:02:55.7714000Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T09:02:55.7748133Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T09:02:55.7783364Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T09:02:55.7815571Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T09:02:55.7851769Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T09:02:55.7884309Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T09:02:55.7919236Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T09:02:55.7951906Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T09:02:55.7986146Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T09:02:55.8020200Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T09:02:55.8054967Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T09:02:55.8089165Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T09:02:55.8125950Z Entering 'third_party/mimalloc' 2024-06-26T09:02:55.8158911Z Entering 'third_party/nccl/nccl' 2024-06-26T09:02:55.8191828Z Entering 'third_party/nlohmann' 2024-06-26T09:02:55.8227328Z Entering 'third_party/onnx' 2024-06-26T09:02:55.8272813Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T09:02:55.8307447Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T09:02:55.8343261Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T09:02:55.8377781Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T09:02:55.8410969Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T09:02:55.8443991Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T09:02:55.8477335Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T09:02:55.8512501Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T09:02:55.8546266Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T09:02:55.8579899Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T09:02:55.8613080Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T09:02:55.8649795Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T09:02:55.8685757Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T09:02:55.8736813Z Entering 'third_party/pocketfft' 2024-06-26T09:02:55.8773150Z Entering 'third_party/protobuf' 2024-06-26T09:02:55.8810373Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T09:02:55.8844800Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T09:02:55.8879255Z Entering 'third_party/psimd' 2024-06-26T09:02:55.8915139Z Entering 'third_party/pthreadpool' 2024-06-26T09:02:55.8949254Z Entering 'third_party/pybind11' 2024-06-26T09:02:55.8984253Z Entering 'third_party/python-peachpy' 2024-06-26T09:02:55.9019504Z Entering 'third_party/sleef' 2024-06-26T09:02:55.9054419Z Entering 'third_party/tensorpipe' 2024-06-26T09:02:55.9089406Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T09:02:55.9124374Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T09:02:55.9157615Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T09:02:55.9190950Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T09:02:55.9222900Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T09:02:55.9273307Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T09:02:55.9293898Z http.https://github.com/.extraheader 2024-06-26T09:02:55.9302530Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-06-26T09:02:55.9330448Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T09:02:55.9536479Z Entering 'android/libs/fbjni' 2024-06-26T09:02:55.9558102Z http.https://github.com/.extraheader 2024-06-26T09:02:55.9583597Z Entering 'third_party/FP16' 2024-06-26T09:02:55.9605330Z http.https://github.com/.extraheader 2024-06-26T09:02:55.9629005Z Entering 'third_party/FXdiv' 2024-06-26T09:02:55.9650233Z http.https://github.com/.extraheader 2024-06-26T09:02:55.9674413Z Entering 'third_party/NNPACK' 2024-06-26T09:02:55.9696074Z http.https://github.com/.extraheader 2024-06-26T09:02:55.9720212Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T09:02:55.9742364Z http.https://github.com/.extraheader 2024-06-26T09:02:55.9763781Z Entering 'third_party/XNNPACK' 2024-06-26T09:02:55.9786504Z http.https://github.com/.extraheader 2024-06-26T09:02:55.9821378Z Entering 'third_party/benchmark' 2024-06-26T09:02:55.9843440Z http.https://github.com/.extraheader 2024-06-26T09:02:55.9866999Z Entering 'third_party/cpp-httplib' 2024-06-26T09:02:55.9888463Z http.https://github.com/.extraheader 2024-06-26T09:02:55.9911757Z Entering 'third_party/cpuinfo' 2024-06-26T09:02:55.9933757Z http.https://github.com/.extraheader 2024-06-26T09:02:55.9956640Z Entering 'third_party/cudnn_frontend' 2024-06-26T09:02:55.9976552Z http.https://github.com/.extraheader 2024-06-26T09:02:55.9999326Z Entering 'third_party/cutlass' 2024-06-26T09:02:56.0020619Z http.https://github.com/.extraheader 2024-06-26T09:02:56.0050423Z Entering 'third_party/eigen' 2024-06-26T09:02:56.0071804Z http.https://github.com/.extraheader 2024-06-26T09:02:56.0096685Z Entering 'third_party/fbgemm' 2024-06-26T09:02:56.0118680Z http.https://github.com/.extraheader 2024-06-26T09:02:56.0142121Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T09:02:56.0163736Z http.https://github.com/.extraheader 2024-06-26T09:02:56.0186977Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T09:02:56.0207506Z http.https://github.com/.extraheader 2024-06-26T09:02:56.0230930Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T09:02:56.0252218Z http.https://github.com/.extraheader 2024-06-26T09:02:56.0279504Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T09:02:56.0300758Z http.https://github.com/.extraheader 2024-06-26T09:02:56.0323306Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T09:02:56.0344822Z http.https://github.com/.extraheader 2024-06-26T09:02:56.0369336Z Entering 'third_party/flatbuffers' 2024-06-26T09:02:56.0391924Z http.https://github.com/.extraheader 2024-06-26T09:02:56.0418220Z Entering 'third_party/fmt' 2024-06-26T09:02:56.0440394Z http.https://github.com/.extraheader 2024-06-26T09:02:56.0462946Z Entering 'third_party/foxi' 2024-06-26T09:02:56.0484930Z http.https://github.com/.extraheader 2024-06-26T09:02:56.0507937Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T09:02:56.0528974Z http.https://github.com/.extraheader 2024-06-26T09:02:56.0552750Z Entering 'third_party/gloo' 2024-06-26T09:02:56.0573418Z http.https://github.com/.extraheader 2024-06-26T09:02:56.0595607Z Entering 'third_party/googletest' 2024-06-26T09:02:56.0617081Z http.https://github.com/.extraheader 2024-06-26T09:02:56.0641907Z Entering 'third_party/ideep' 2024-06-26T09:02:56.0663042Z http.https://github.com/.extraheader 2024-06-26T09:02:56.0685859Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T09:02:56.0706955Z http.https://github.com/.extraheader 2024-06-26T09:02:56.0735523Z Entering 'third_party/ittapi' 2024-06-26T09:02:56.0759356Z http.https://github.com/.extraheader 2024-06-26T09:02:56.0783025Z Entering 'third_party/kineto' 2024-06-26T09:02:56.0803955Z http.https://github.com/.extraheader 2024-06-26T09:02:56.0827566Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T09:02:56.0849489Z http.https://github.com/.extraheader 2024-06-26T09:02:56.0873869Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T09:02:56.0894901Z http.https://github.com/.extraheader 2024-06-26T09:02:56.0918905Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T09:02:56.0940096Z http.https://github.com/.extraheader 2024-06-26T09:02:56.0963152Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T09:02:56.0985504Z http.https://github.com/.extraheader 2024-06-26T09:02:56.1009227Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T09:02:56.1030240Z http.https://github.com/.extraheader 2024-06-26T09:02:56.1050829Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T09:02:56.1071608Z http.https://github.com/.extraheader 2024-06-26T09:02:56.1095841Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T09:02:56.1116871Z http.https://github.com/.extraheader 2024-06-26T09:02:56.1141229Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T09:02:56.1162634Z http.https://github.com/.extraheader 2024-06-26T09:02:56.1187834Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T09:02:56.1208362Z http.https://github.com/.extraheader 2024-06-26T09:02:56.1232741Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T09:02:56.1253991Z http.https://github.com/.extraheader 2024-06-26T09:02:56.1277947Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T09:02:56.1300128Z http.https://github.com/.extraheader 2024-06-26T09:02:56.1322742Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T09:02:56.1344447Z http.https://github.com/.extraheader 2024-06-26T09:02:56.1368793Z Entering 'third_party/mimalloc' 2024-06-26T09:02:56.1391359Z http.https://github.com/.extraheader 2024-06-26T09:02:56.1414845Z Entering 'third_party/nccl/nccl' 2024-06-26T09:02:56.1436653Z http.https://github.com/.extraheader 2024-06-26T09:02:56.1461316Z Entering 'third_party/nlohmann' 2024-06-26T09:02:56.1482165Z http.https://github.com/.extraheader 2024-06-26T09:02:56.1507420Z Entering 'third_party/onnx' 2024-06-26T09:02:56.1529074Z http.https://github.com/.extraheader 2024-06-26T09:02:56.1566061Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T09:02:56.1588746Z http.https://github.com/.extraheader 2024-06-26T09:02:56.1612127Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T09:02:56.1633932Z http.https://github.com/.extraheader 2024-06-26T09:02:56.1660199Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T09:02:56.1683102Z http.https://github.com/.extraheader 2024-06-26T09:02:56.1707600Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T09:02:56.1728625Z http.https://github.com/.extraheader 2024-06-26T09:02:56.1752391Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T09:02:56.1773206Z http.https://github.com/.extraheader 2024-06-26T09:02:56.1795820Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T09:02:56.1817004Z http.https://github.com/.extraheader 2024-06-26T09:02:56.1841177Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T09:02:56.1863293Z http.https://github.com/.extraheader 2024-06-26T09:02:56.1887168Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T09:02:56.1908915Z http.https://github.com/.extraheader 2024-06-26T09:02:56.1932650Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T09:02:56.1954718Z http.https://github.com/.extraheader 2024-06-26T09:02:56.1977093Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T09:02:56.1998475Z http.https://github.com/.extraheader 2024-06-26T09:02:56.2022064Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T09:02:56.2042154Z http.https://github.com/.extraheader 2024-06-26T09:02:56.2065622Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T09:02:56.2086711Z http.https://github.com/.extraheader 2024-06-26T09:02:56.2112240Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T09:02:56.2133527Z http.https://github.com/.extraheader 2024-06-26T09:02:56.2172548Z Entering 'third_party/pocketfft' 2024-06-26T09:02:56.2196152Z http.https://github.com/.extraheader 2024-06-26T09:02:56.2218596Z Entering 'third_party/protobuf' 2024-06-26T09:02:56.2239964Z http.https://github.com/.extraheader 2024-06-26T09:02:56.2264969Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T09:02:56.2284336Z http.https://github.com/.extraheader 2024-06-26T09:02:56.2306014Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T09:02:56.2326889Z http.https://github.com/.extraheader 2024-06-26T09:02:56.2350710Z Entering 'third_party/psimd' 2024-06-26T09:02:56.2371038Z http.https://github.com/.extraheader 2024-06-26T09:02:56.2392748Z Entering 'third_party/pthreadpool' 2024-06-26T09:02:56.2414421Z http.https://github.com/.extraheader 2024-06-26T09:02:56.2439133Z Entering 'third_party/pybind11' 2024-06-26T09:02:56.2460895Z http.https://github.com/.extraheader 2024-06-26T09:02:56.2483471Z Entering 'third_party/python-peachpy' 2024-06-26T09:02:56.2505285Z http.https://github.com/.extraheader 2024-06-26T09:02:56.2528855Z Entering 'third_party/sleef' 2024-06-26T09:02:56.2550194Z http.https://github.com/.extraheader 2024-06-26T09:02:56.2573199Z Entering 'third_party/tensorpipe' 2024-06-26T09:02:56.2594334Z http.https://github.com/.extraheader 2024-06-26T09:02:56.2616562Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T09:02:56.2637940Z http.https://github.com/.extraheader 2024-06-26T09:02:56.2660230Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T09:02:56.2680802Z http.https://github.com/.extraheader 2024-06-26T09:02:56.2705183Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T09:02:56.2726532Z http.https://github.com/.extraheader 2024-06-26T09:02:56.2749800Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T09:02:56.2771199Z http.https://github.com/.extraheader 2024-06-26T09:02:56.2794562Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T09:02:56.2815440Z http.https://github.com/.extraheader 2024-06-26T09:02:56.2915431Z A job completed hook has been configured by the self-hosted runner administrator 2024-06-26T09:02:56.2932993Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T09:02:56.2940084Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:02:56.2940739Z ##[endgroup] 2024-06-26T09:02:56.9609342Z Cleaning up orphan processes