2024-06-26T07:02:26.3874377Z Current runner version: '2.317.0' 2024-06-26T07:02:26.3880840Z Runner name: 'i-02844b9639d793963' 2024-06-26T07:02:26.3881740Z Runner group name: 'Default' 2024-06-26T07:02:26.3882731Z Machine name: 'ip-10-0-62-2' 2024-06-26T07:02:26.3888043Z ##[group]GITHUB_TOKEN Permissions 2024-06-26T07:02:26.3890054Z Actions: read 2024-06-26T07:02:26.3890698Z Attestations: read 2024-06-26T07:02:26.3891266Z Checks: read 2024-06-26T07:02:26.3891809Z Contents: read 2024-06-26T07:02:26.3892447Z Deployments: read 2024-06-26T07:02:26.3893006Z Discussions: read 2024-06-26T07:02:26.3893585Z Issues: read 2024-06-26T07:02:26.3894177Z Metadata: read 2024-06-26T07:02:26.3894755Z Packages: read 2024-06-26T07:02:26.3895332Z Pages: read 2024-06-26T07:02:26.3895908Z PullRequests: read 2024-06-26T07:02:26.3896517Z RepositoryProjects: read 2024-06-26T07:02:26.3897166Z SecurityEvents: read 2024-06-26T07:02:26.3897792Z Statuses: read 2024-06-26T07:02:26.3898340Z ##[endgroup] 2024-06-26T07:02:26.3901710Z Secret source: Actions 2024-06-26T07:02:26.3902536Z Prepare workflow directory 2024-06-26T07:02:26.7526710Z Prepare all required actions 2024-06-26T07:02:26.7717493Z Getting action download info 2024-06-26T07:02:26.9550313Z Download action repository 'pytorch/test-infra@main' (SHA:43a2ce341cc31288e9a38b65ce600a7f43021bd5) 2024-06-26T07:02:27.2481607Z Download action repository 'pytorch/pytorch@main' (SHA:c422a9549d2b320b47a9391f6cbcf592b2876c88) 2024-06-26T07:02:30.0219304Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-06-26T07:02:30.1557046Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-06-26T07:02:30.4014092Z Getting action download info 2024-06-26T07:02:30.5804513Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-06-26T07:02:30.7452242Z Getting action download info 2024-06-26T07:02:30.8785115Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-06-26T07:02:31.0268956Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/129470 (b8c4c54d347aa776934c60784e35936878ef18dc) 2024-06-26T07:02:31.0271208Z ##[group] Inputs 2024-06-26T07:02:31.0271678Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:02:31.0279015Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:02:31.0286955Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:31.0288142Z sync-tag: 2024-06-26T07:02:31.0289163Z timeout-minutes: 240 2024-06-26T07:02:31.0289511Z use-gha: 2024-06-26T07:02:31.0289798Z dashboard-tag: 2024-06-26T07:02:31.0290115Z s3-bucket: gha-artifacts 2024-06-26T07:02:31.0290481Z aws-role-to-assume: 2024-06-26T07:02:31.0290818Z ##[endgroup] 2024-06-26T07:02:31.0291894Z Complete job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:02:31.0855423Z A job started hook has been configured by the self-hosted runner administrator 2024-06-26T07:02:31.0995711Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T07:02:31.1006316Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:31.1006842Z ##[endgroup] 2024-06-26T07:02:32.1370995Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-06-26T07:02:32.1371562Z with: 2024-06-26T07:02:32.1372266Z github-secret: *** 2024-06-26T07:02:32.1373198Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-06-26T07:02:32.1374184Z activate-with-label: false 2024-06-26T07:02:32.1374548Z label: with-ssh 2024-06-26T07:02:32.1374882Z remove-existing-keys: true 2024-06-26T07:02:32.1375292Z fail-silently: true 2024-06-26T07:02:32.1375632Z env: 2024-06-26T07:02:32.1375911Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:32.1376261Z ##[endgroup] 2024-06-26T07:02:32.2242444Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-06-26T07:02:32.2243575Z ciflow reference detected, attempting to extract PR number 2024-06-26T07:02:32.4636463Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-06-26T07:02:32.5340222Z No SSH keys found for user pytorch-bot[bot] 2024-06-26T07:02:32.5341361Z Grabbing public ssh keys from https://github.com/leslie-fang-intel.keys 2024-06-26T07:02:32.5962139Z No SSH keys found for user leslie-fang-intel 2024-06-26T07:02:32.6066477Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-06-26T07:02:32.6067066Z with: 2024-06-26T07:02:32.6067406Z submodules: recursive 2024-06-26T07:02:32.6067832Z fetch-depth: 0 2024-06-26T07:02:32.6068177Z env: 2024-06-26T07:02:32.6068475Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:32.6068887Z ##[endgroup] 2024-06-26T07:02:32.6289637Z ##[group]Run retry () { 2024-06-26T07:02:32.6290035Z retry () { 2024-06-26T07:02:32.6290679Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-06-26T07:02:32.6291272Z } 2024-06-26T07:02:32.6291592Z echo "${GITHUB_WORKSPACE}" 2024-06-26T07:02:32.6292017Z if [ -z "${NO_SUDO}" ]; then 2024-06-26T07:02:32.6292520Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:02:32.6292983Z else 2024-06-26T07:02:32.6293325Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:02:32.6293972Z fi 2024-06-26T07:02:32.6294321Z mkdir "${GITHUB_WORKSPACE}" 2024-06-26T07:02:32.6302319Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:32.6302833Z env: 2024-06-26T07:02:32.6303119Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:32.6303468Z NO_SUDO: 2024-06-26T07:02:32.6303755Z ##[endgroup] 2024-06-26T07:02:32.6325984Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:02:34.8938349Z ##[group]Run malfet/checkout@silent-checkout 2024-06-26T07:02:34.8938866Z with: 2024-06-26T07:02:34.8939265Z ref: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:02:34.8939790Z fetch-depth: 0 2024-06-26T07:02:34.8940185Z submodules: recursive 2024-06-26T07:02:34.8940602Z quiet-checkout: true 2024-06-26T07:02:34.8941162Z repository: pytorch/pytorch 2024-06-26T07:02:34.8941719Z token: *** 2024-06-26T07:02:34.8942093Z ssh-strict: true 2024-06-26T07:02:34.8942494Z persist-credentials: true 2024-06-26T07:02:34.8942931Z clean: true 2024-06-26T07:02:34.8943339Z sparse-checkout-cone-mode: true 2024-06-26T07:02:34.8943807Z lfs: false 2024-06-26T07:02:34.8944182Z set-safe-directory: true 2024-06-26T07:02:34.8944599Z env: 2024-06-26T07:02:34.8944949Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:34.8945363Z ##[endgroup] 2024-06-26T07:02:34.9875741Z Syncing repository: pytorch/pytorch 2024-06-26T07:02:34.9877412Z ##[group]Getting Git version info 2024-06-26T07:02:34.9878244Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:02:34.9879211Z [command]/usr/bin/git version 2024-06-26T07:02:34.9879659Z git version 2.40.1 2024-06-26T07:02:34.9881298Z ##[endgroup] 2024-06-26T07:02:34.9892681Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/306ec7f2-e770-47b5-b6b1-3514dcc691b9' before making global git config changes 2024-06-26T07:02:34.9894044Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T07:02:34.9896562Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:02:34.9931333Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:02:34.9935485Z ##[group]Initializing the repository 2024-06-26T07:02:34.9938494Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:02:34.9958864Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-06-26T07:02:34.9960104Z hint: is subject to change. To configure the initial branch name to use in all 2024-06-26T07:02:34.9961033Z hint: of your new repositories, which will suppress this warning, call: 2024-06-26T07:02:34.9961906Z hint: 2024-06-26T07:02:34.9962638Z hint: git config --global init.defaultBranch 2024-06-26T07:02:34.9963205Z hint: 2024-06-26T07:02:34.9963796Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-06-26T07:02:34.9964902Z hint: 'development'. The just-created branch can be renamed via this command: 2024-06-26T07:02:34.9965873Z hint: 2024-06-26T07:02:34.9966279Z hint: git branch -m 2024-06-26T07:02:34.9967159Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-06-26T07:02:34.9969068Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-06-26T07:02:34.9989682Z ##[endgroup] 2024-06-26T07:02:34.9990524Z ##[group]Disabling automatic garbage collection 2024-06-26T07:02:34.9992606Z [command]/usr/bin/git config --local gc.auto 0 2024-06-26T07:02:35.0012941Z ##[endgroup] 2024-06-26T07:02:35.0013596Z ##[group]Setting up auth 2024-06-26T07:02:35.0019115Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T07:02:35.0040550Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T07:02:35.0236653Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T07:02:35.0259022Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T07:02:35.0464558Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:02:35.0507380Z ##[endgroup] 2024-06-26T07:02:35.0508116Z ##[group]Fetching the repository 2024-06-26T07:02:35.0514034Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-06-26T07:02:37.5359279Z remote: Enumerating objects: 987619 2024-06-26T07:02:37.5360254Z remote: Enumerating objects: 990209, done. 2024-06-26T07:02:37.5363211Z remote: Counting objects: 0% (1/2590) 2024-06-26T07:02:37.5363869Z remote: Counting objects: 1% (26/2590) 2024-06-26T07:02:37.5365039Z remote: Counting objects: 2% (52/2590) 2024-06-26T07:02:37.5366573Z remote: Counting objects: 3% (78/2590) 2024-06-26T07:02:37.5367900Z remote: Counting objects: 4% (104/2590) 2024-06-26T07:02:37.5369194Z remote: Counting objects: 5% (130/2590) 2024-06-26T07:02:37.5370238Z remote: Counting objects: 6% (156/2590) 2024-06-26T07:02:37.5371315Z remote: Counting objects: 7% (182/2590) 2024-06-26T07:02:37.5372281Z remote: Counting objects: 8% (208/2590) 2024-06-26T07:02:37.5373154Z remote: Counting objects: 9% (234/2590) 2024-06-26T07:02:37.5374211Z remote: Counting objects: 10% (259/2590) 2024-06-26T07:02:37.5374934Z remote: Counting objects: 11% (285/2590) 2024-06-26T07:02:37.5375807Z remote: Counting objects: 12% (311/2590) 2024-06-26T07:02:37.5376404Z remote: Counting objects: 13% (337/2590) 2024-06-26T07:02:37.5377433Z remote: Counting objects: 14% (363/2590) 2024-06-26T07:02:37.5378303Z remote: Counting objects: 15% (389/2590) 2024-06-26T07:02:37.5379298Z remote: Counting objects: 16% (415/2590) 2024-06-26T07:02:37.5380035Z remote: Counting objects: 17% (441/2590) 2024-06-26T07:02:37.5380730Z remote: Counting objects: 18% (467/2590) 2024-06-26T07:02:37.5381326Z remote: Counting objects: 19% (493/2590) 2024-06-26T07:02:37.5381918Z remote: Counting objects: 20% (518/2590) 2024-06-26T07:02:37.5390430Z remote: Counting objects: 21% (544/2590) 2024-06-26T07:02:37.5391357Z remote: Counting objects: 22% (570/2590) 2024-06-26T07:02:37.5392145Z remote: Counting objects: 23% (596/2590) 2024-06-26T07:02:37.5393077Z remote: Counting objects: 24% (622/2590) 2024-06-26T07:02:37.5393732Z remote: Counting objects: 25% (648/2590) 2024-06-26T07:02:37.5394791Z remote: Counting objects: 26% (674/2590) 2024-06-26T07:02:37.5395605Z remote: Counting objects: 27% (700/2590) 2024-06-26T07:02:37.5396481Z remote: Counting objects: 28% (726/2590) 2024-06-26T07:02:37.5397426Z remote: Counting objects: 29% (752/2590) 2024-06-26T07:02:37.5398072Z remote: Counting objects: 30% (777/2590) 2024-06-26T07:02:37.5399142Z remote: Counting objects: 31% (803/2590) 2024-06-26T07:02:37.5399791Z remote: Counting objects: 32% (829/2590) 2024-06-26T07:02:37.5400986Z remote: Counting objects: 33% (855/2590) 2024-06-26T07:02:37.5401971Z remote: Counting objects: 34% (881/2590) 2024-06-26T07:02:37.5403000Z remote: Counting objects: 35% (907/2590) 2024-06-26T07:02:37.5403994Z remote: Counting objects: 36% (933/2590) 2024-06-26T07:02:37.5404870Z remote: Counting objects: 37% (959/2590) 2024-06-26T07:02:37.5405667Z remote: Counting objects: 38% (985/2590) 2024-06-26T07:02:37.5406513Z remote: Counting objects: 39% (1011/2590) 2024-06-26T07:02:37.5407419Z remote: Counting objects: 40% (1036/2590) 2024-06-26T07:02:37.5408440Z remote: Counting objects: 41% (1062/2590) 2024-06-26T07:02:37.5409362Z remote: Counting objects: 42% (1088/2590) 2024-06-26T07:02:37.5410171Z remote: Counting objects: 43% (1114/2590) 2024-06-26T07:02:37.5411323Z remote: Counting objects: 44% (1140/2590) 2024-06-26T07:02:37.5412003Z remote: Counting objects: 45% (1166/2590) 2024-06-26T07:02:37.5413183Z remote: Counting objects: 46% (1192/2590) 2024-06-26T07:02:37.5414108Z remote: Counting objects: 47% (1218/2590) 2024-06-26T07:02:37.5415127Z remote: Counting objects: 48% (1244/2590) 2024-06-26T07:02:37.5415873Z remote: Counting objects: 49% (1270/2590) 2024-06-26T07:02:37.5416743Z remote: Counting objects: 50% (1295/2590) 2024-06-26T07:02:37.5417954Z remote: Counting objects: 51% (1321/2590) 2024-06-26T07:02:37.5418609Z remote: Counting objects: 52% (1347/2590) 2024-06-26T07:02:37.5419206Z remote: Counting objects: 53% (1373/2590) 2024-06-26T07:02:37.5419806Z remote: Counting objects: 54% (1399/2590) 2024-06-26T07:02:37.5420403Z remote: Counting objects: 55% (1425/2590) 2024-06-26T07:02:37.5421206Z remote: Counting objects: 56% (1451/2590) 2024-06-26T07:02:37.5421972Z remote: Counting objects: 57% (1477/2590) 2024-06-26T07:02:37.5422582Z remote: Counting objects: 58% (1503/2590) 2024-06-26T07:02:37.5423482Z remote: Counting objects: 59% (1529/2590) 2024-06-26T07:02:37.5424213Z remote: Counting objects: 60% (1554/2590) 2024-06-26T07:02:37.5424814Z remote: Counting objects: 61% (1580/2590) 2024-06-26T07:02:37.5425426Z remote: Counting objects: 62% (1606/2590) 2024-06-26T07:02:37.5426057Z remote: Counting objects: 63% (1632/2590) 2024-06-26T07:02:37.5426665Z remote: Counting objects: 64% (1658/2590) 2024-06-26T07:02:37.5427325Z remote: Counting objects: 65% (1684/2590) 2024-06-26T07:02:37.5427931Z remote: Counting objects: 66% (1710/2590) 2024-06-26T07:02:37.5428539Z remote: Counting objects: 67% (1736/2590) 2024-06-26T07:02:37.5429149Z remote: Counting objects: 68% (1762/2590) 2024-06-26T07:02:37.5429753Z remote: Counting objects: 69% (1788/2590) 2024-06-26T07:02:37.5430485Z remote: Counting objects: 70% (1813/2590) 2024-06-26T07:02:37.5431084Z remote: Counting objects: 71% (1839/2590) 2024-06-26T07:02:37.5431683Z remote: Counting objects: 72% (1865/2590) 2024-06-26T07:02:37.5432398Z remote: Counting objects: 73% (1891/2590) 2024-06-26T07:02:37.5432994Z remote: Counting objects: 74% (1917/2590) 2024-06-26T07:02:37.5433591Z remote: Counting objects: 75% (1943/2590) 2024-06-26T07:02:37.5434202Z remote: Counting objects: 76% (1969/2590) 2024-06-26T07:02:37.5434820Z remote: Counting objects: 77% (1995/2590) 2024-06-26T07:02:37.5435425Z remote: Counting objects: 78% (2021/2590) 2024-06-26T07:02:37.5436130Z remote: Counting objects: 79% (2047/2590) 2024-06-26T07:02:37.5436745Z remote: Counting objects: 80% (2072/2590) 2024-06-26T07:02:37.5437389Z remote: Counting objects: 81% (2098/2590) 2024-06-26T07:02:37.5437981Z remote: Counting objects: 82% (2124/2590) 2024-06-26T07:02:37.5438580Z remote: Counting objects: 83% (2150/2590) 2024-06-26T07:02:37.5439181Z remote: Counting objects: 84% (2176/2590) 2024-06-26T07:02:37.5439778Z remote: Counting objects: 85% (2202/2590) 2024-06-26T07:02:37.5440369Z remote: Counting objects: 86% (2228/2590) 2024-06-26T07:02:37.5440970Z remote: Counting objects: 87% (2254/2590) 2024-06-26T07:02:37.5441575Z remote: Counting objects: 88% (2280/2590) 2024-06-26T07:02:37.5442185Z remote: Counting objects: 89% (2306/2590) 2024-06-26T07:02:37.5442782Z remote: Counting objects: 90% (2331/2590) 2024-06-26T07:02:37.5443386Z remote: Counting objects: 91% (2357/2590) 2024-06-26T07:02:37.5443999Z remote: Counting objects: 92% (2383/2590) 2024-06-26T07:02:37.5444604Z remote: Counting objects: 93% (2409/2590) 2024-06-26T07:02:37.5445202Z remote: Counting objects: 94% (2435/2590) 2024-06-26T07:02:37.5445805Z remote: Counting objects: 95% (2461/2590) 2024-06-26T07:02:37.5446409Z remote: Counting objects: 96% (2487/2590) 2024-06-26T07:02:37.5447015Z remote: Counting objects: 97% (2513/2590) 2024-06-26T07:02:37.5447609Z remote: Counting objects: 98% (2539/2590) 2024-06-26T07:02:37.5448212Z remote: Counting objects: 99% (2565/2590) 2024-06-26T07:02:37.5448813Z remote: Counting objects: 100% (2590/2590) 2024-06-26T07:02:37.5449452Z remote: Counting objects: 100% (2590/2590), done. 2024-06-26T07:02:37.5586652Z remote: Compressing objects: 0% (1/1068) 2024-06-26T07:02:37.6050223Z remote: Compressing objects: 1% (11/1068) 2024-06-26T07:02:37.6474466Z remote: Compressing objects: 2% (22/1068) 2024-06-26T07:02:37.6852569Z remote: Compressing objects: 3% (33/1068) 2024-06-26T07:02:37.7073734Z remote: Compressing objects: 4% (43/1068) 2024-06-26T07:02:37.7336736Z remote: Compressing objects: 5% (54/1068) 2024-06-26T07:02:37.7529369Z remote: Compressing objects: 6% (65/1068) 2024-06-26T07:02:37.7908949Z remote: Compressing objects: 7% (75/1068) 2024-06-26T07:02:37.8746353Z remote: Compressing objects: 8% (86/1068) 2024-06-26T07:02:37.9519081Z remote: Compressing objects: 9% (97/1068) 2024-06-26T07:02:37.9759637Z remote: Compressing objects: 10% (107/1068) 2024-06-26T07:02:37.9939090Z remote: Compressing objects: 11% (118/1068) 2024-06-26T07:02:38.0038455Z remote: Compressing objects: 12% (129/1068) 2024-06-26T07:02:38.0039091Z remote: Compressing objects: 13% (139/1068) 2024-06-26T07:02:38.0159385Z remote: Compressing objects: 14% (150/1068) 2024-06-26T07:02:38.0160034Z remote: Compressing objects: 15% (161/1068) 2024-06-26T07:02:38.0228924Z remote: Compressing objects: 16% (171/1068) 2024-06-26T07:02:38.0229553Z remote: Compressing objects: 17% (182/1068) 2024-06-26T07:02:38.0230301Z remote: Compressing objects: 18% (193/1068) 2024-06-26T07:02:38.0240318Z remote: Compressing objects: 19% (203/1068) 2024-06-26T07:02:38.0260569Z remote: Compressing objects: 20% (214/1068) 2024-06-26T07:02:38.0274937Z remote: Compressing objects: 21% (225/1068) 2024-06-26T07:02:38.0291023Z remote: Compressing objects: 22% (235/1068) 2024-06-26T07:02:38.0313623Z remote: Compressing objects: 23% (246/1068) 2024-06-26T07:02:38.0323160Z remote: Compressing objects: 24% (257/1068) 2024-06-26T07:02:38.0332094Z remote: Compressing objects: 25% (267/1068) 2024-06-26T07:02:38.0341260Z remote: Compressing objects: 26% (278/1068) 2024-06-26T07:02:38.0346175Z remote: Compressing objects: 27% (289/1068) 2024-06-26T07:02:38.0350485Z remote: Compressing objects: 28% (300/1068) 2024-06-26T07:02:38.0358469Z remote: Compressing objects: 29% (310/1068) 2024-06-26T07:02:38.0365008Z remote: Compressing objects: 30% (321/1068) 2024-06-26T07:02:38.0370719Z remote: Compressing objects: 31% (332/1068) 2024-06-26T07:02:38.0377582Z remote: Compressing objects: 32% (342/1068) 2024-06-26T07:02:38.0386860Z remote: Compressing objects: 33% (353/1068) 2024-06-26T07:02:38.0399193Z remote: Compressing objects: 34% (364/1068) 2024-06-26T07:02:38.0414478Z remote: Compressing objects: 35% (374/1068) 2024-06-26T07:02:38.0431988Z remote: Compressing objects: 36% (385/1068) 2024-06-26T07:02:38.0446466Z remote: Compressing objects: 37% (396/1068) 2024-06-26T07:02:38.0462089Z remote: Compressing objects: 38% (406/1068) 2024-06-26T07:02:38.0476135Z remote: Compressing objects: 39% (417/1068) 2024-06-26T07:02:38.0491757Z remote: Compressing objects: 40% (428/1068) 2024-06-26T07:02:38.0511239Z remote: Compressing objects: 41% (438/1068) 2024-06-26T07:02:38.0526299Z remote: Compressing objects: 42% (449/1068) 2024-06-26T07:02:38.0537918Z remote: Compressing objects: 43% (460/1068) 2024-06-26T07:02:38.0542937Z remote: Compressing objects: 44% (470/1068) 2024-06-26T07:02:38.0547637Z remote: Compressing objects: 45% (481/1068) 2024-06-26T07:02:38.0555267Z remote: Compressing objects: 46% (492/1068) 2024-06-26T07:02:38.0586197Z remote: Compressing objects: 47% (502/1068) 2024-06-26T07:02:38.0597370Z remote: Compressing objects: 48% (513/1068) 2024-06-26T07:02:38.0601357Z remote: Compressing objects: 49% (524/1068) 2024-06-26T07:02:38.0608492Z remote: Compressing objects: 50% (534/1068) 2024-06-26T07:02:38.0618339Z remote: Compressing objects: 51% (545/1068) 2024-06-26T07:02:38.0630234Z remote: Compressing objects: 52% (556/1068) 2024-06-26T07:02:38.0646306Z remote: Compressing objects: 53% (567/1068) 2024-06-26T07:02:38.0662101Z remote: Compressing objects: 54% (577/1068) 2024-06-26T07:02:38.0671268Z remote: Compressing objects: 55% (588/1068) 2024-06-26T07:02:38.0680218Z remote: Compressing objects: 56% (599/1068) 2024-06-26T07:02:38.0695039Z remote: Compressing objects: 57% (609/1068) 2024-06-26T07:02:38.0706722Z remote: Compressing objects: 58% (620/1068) 2024-06-26T07:02:38.0715706Z remote: Compressing objects: 59% (631/1068) 2024-06-26T07:02:38.0728326Z remote: Compressing objects: 60% (641/1068) 2024-06-26T07:02:38.0739672Z remote: Compressing objects: 61% (652/1068) 2024-06-26T07:02:38.0747022Z remote: Compressing objects: 62% (663/1068) 2024-06-26T07:02:38.0755525Z remote: Compressing objects: 63% (673/1068) 2024-06-26T07:02:38.0763838Z remote: Compressing objects: 64% (684/1068) 2024-06-26T07:02:38.0770538Z remote: Compressing objects: 65% (695/1068) 2024-06-26T07:02:38.0773002Z remote: Compressing objects: 66% (705/1068) 2024-06-26T07:02:38.0783370Z remote: Compressing objects: 67% (716/1068) 2024-06-26T07:02:38.0785820Z remote: Compressing objects: 68% (727/1068) 2024-06-26T07:02:38.0789707Z remote: Compressing objects: 69% (737/1068) 2024-06-26T07:02:38.0793113Z remote: Compressing objects: 70% (748/1068) 2024-06-26T07:02:38.0796031Z remote: Compressing objects: 71% (759/1068) 2024-06-26T07:02:38.0798571Z remote: Compressing objects: 72% (769/1068) 2024-06-26T07:02:38.0799756Z remote: Compressing objects: 73% (780/1068) 2024-06-26T07:02:38.0800743Z remote: Compressing objects: 74% (791/1068) 2024-06-26T07:02:38.0801513Z remote: Compressing objects: 75% (801/1068) 2024-06-26T07:02:38.0802141Z remote: Compressing objects: 76% (812/1068) 2024-06-26T07:02:38.0802885Z remote: Compressing objects: 77% (823/1068) 2024-06-26T07:02:38.0803515Z remote: Compressing objects: 78% (834/1068) 2024-06-26T07:02:38.0806204Z remote: Compressing objects: 79% (844/1068) 2024-06-26T07:02:38.0806858Z remote: Compressing objects: 80% (855/1068) 2024-06-26T07:02:38.0809283Z remote: Compressing objects: 81% (866/1068) 2024-06-26T07:02:38.0812208Z remote: Compressing objects: 82% (876/1068) 2024-06-26T07:02:38.0813033Z remote: Compressing objects: 83% (887/1068) 2024-06-26T07:02:38.0815052Z remote: Compressing objects: 84% (898/1068) 2024-06-26T07:02:38.0815921Z remote: Compressing objects: 85% (908/1068) 2024-06-26T07:02:38.0818072Z remote: Compressing objects: 86% (919/1068) 2024-06-26T07:02:38.0820257Z remote: Compressing objects: 87% (930/1068) 2024-06-26T07:02:38.0821119Z remote: Compressing objects: 88% (940/1068) 2024-06-26T07:02:38.0823342Z remote: Compressing objects: 89% (951/1068) 2024-06-26T07:02:38.0824206Z remote: Compressing objects: 90% (962/1068) 2024-06-26T07:02:38.0824939Z remote: Compressing objects: 91% (972/1068) 2024-06-26T07:02:38.0825762Z remote: Compressing objects: 92% (983/1068) 2024-06-26T07:02:38.0827290Z remote: Compressing objects: 93% (994/1068) 2024-06-26T07:02:38.0828123Z remote: Compressing objects: 94% (1004/1068) 2024-06-26T07:02:38.0830105Z remote: Compressing objects: 95% (1015/1068) 2024-06-26T07:02:38.0830753Z remote: Compressing objects: 96% (1026/1068) 2024-06-26T07:02:38.0832219Z remote: Compressing objects: 97% (1036/1068) 2024-06-26T07:02:38.0833350Z remote: Compressing objects: 98% (1047/1068) 2024-06-26T07:02:38.0834285Z remote: Compressing objects: 99% (1058/1068) 2024-06-26T07:02:38.0835092Z remote: Compressing objects: 100% (1068/1068) 2024-06-26T07:02:38.0835752Z remote: Compressing objects: 100% (1068/1068), done. 2024-06-26T07:02:58.3610245Z remote: Total 990209 (delta 2055), reused 1861 (delta 1521), pack-reused 987619 2024-06-26T07:03:21.9178295Z [command]/usr/bin/git rev-parse --verify --quiet b8c4c54d347aa776934c60784e35936878ef18dc^{object} 2024-06-26T07:03:21.9197759Z b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:03:21.9201801Z ##[endgroup] 2024-06-26T07:03:21.9202357Z ##[group]Determining the checkout info 2024-06-26T07:03:21.9204680Z ##[endgroup] 2024-06-26T07:03:21.9205243Z ##[group]Checking out the ref 2024-06-26T07:03:21.9206668Z [command]/usr/bin/git checkout --quiet --force b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:03:23.1409260Z ##[endgroup] 2024-06-26T07:03:23.1409867Z ##[group]Setting up auth for fetching submodules 2024-06-26T07:03:23.1414217Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:03:23.1459800Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-06-26T07:03:23.1485760Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-06-26T07:03:23.1510071Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-06-26T07:03:23.1532463Z ##[endgroup] 2024-06-26T07:03:23.1533012Z ##[group]Fetching submodules 2024-06-26T07:03:23.1536325Z [command]/usr/bin/git submodule sync --recursive 2024-06-26T07:03:23.1741944Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-06-26T07:03:23.1951701Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-06-26T07:03:23.1953495Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-06-26T07:03:23.1955058Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-06-26T07:03:23.1957612Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-06-26T07:03:23.1959374Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-06-26T07:03:23.1961036Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-06-26T07:03:23.1962640Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-06-26T07:03:23.1964571Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-06-26T07:03:23.1966853Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-06-26T07:03:23.1969240Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-06-26T07:03:23.1971453Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-06-26T07:03:23.1973870Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-06-26T07:03:23.1976275Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-06-26T07:03:23.1978943Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-06-26T07:03:23.1981398Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-06-26T07:03:23.1984124Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-06-26T07:03:23.1987059Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:03:23.1989808Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-06-26T07:03:23.1993016Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-06-26T07:03:23.1996445Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-06-26T07:03:23.1999359Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-06-26T07:03:23.2002454Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-06-26T07:03:23.2005748Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-06-26T07:03:23.2008894Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-06-26T07:03:23.2012209Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-06-26T07:03:23.2015460Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-06-26T07:03:23.2019129Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-06-26T07:03:23.2022369Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-06-26T07:03:23.2025871Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-06-26T07:03:23.2029446Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-06-26T07:03:23.2033789Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-06-26T07:03:23.2037503Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-06-26T07:03:23.2041223Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-06-26T07:03:23.2044905Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-06-26T07:03:23.2048902Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-06-26T07:03:23.2067691Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-06-26T07:03:23.4364532Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-06-26T07:03:23.5987893Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-06-26T07:03:23.7780742Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-06-26T07:03:24.0505711Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-06-26T07:03:26.0673399Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-06-26T07:03:35.7498510Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-06-26T07:03:36.1180769Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-06-26T07:03:36.7250895Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-06-26T07:03:37.2969393Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-06-26T07:03:38.4803881Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-06-26T07:03:40.1554085Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-06-26T07:03:46.0692981Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-06-26T07:03:47.3700387Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-06-26T07:03:49.0139859Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-06-26T07:03:50.2336202Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-06-26T07:03:50.4226179Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-06-26T07:03:50.7990167Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-06-26T07:03:51.1381772Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-06-26T07:03:52.1308244Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-06-26T07:03:52.4640415Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-06-26T07:03:52.7257782Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-06-26T07:03:54.1992585Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-06-26T07:03:54.9367971Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-06-26T07:03:55.6285259Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-06-26T07:04:01.2138502Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-06-26T07:04:03.5393910Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-06-26T07:04:09.1071877Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-06-26T07:04:09.2894681Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-06-26T07:04:17.6772547Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-06-26T07:04:17.8251222Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-06-26T07:04:17.9914566Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-06-26T07:04:19.0096306Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-06-26T07:04:19.2714631Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-06-26T07:04:19.8429424Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-06-26T07:04:20.2983450Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-06-26T07:04:20.3066060Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-06-26T07:04:20.3126911Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-06-26T07:04:20.3312803Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-06-26T07:04:20.3622587Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-06-26T07:04:21.1408460Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-06-26T07:04:21.1585822Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-06-26T07:04:21.1943940Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-06-26T07:04:21.2787277Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-06-26T07:04:21.3060851Z Submodule path 'third_party/cudnn_frontend': checked out 'aa3abd4bc689d6412979c7f55f9cd132848c9c6a' 2024-06-26T07:04:21.7351994Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-06-26T07:04:21.9472934Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-06-26T07:04:22.0107597Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-06-26T07:04:22.0119351Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:04:22.0122265Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:04:22.0123751Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:04:22.0125290Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:04:22.0126973Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:04:22.0144386Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-06-26T07:04:22.9003172Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-06-26T07:04:23.5760300Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-06-26T07:04:25.3593154Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-06-26T07:04:26.4049261Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-06-26T07:04:26.7306881Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-06-26T07:04:26.8147077Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-06-26T07:04:27.1575829Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-06-26T07:04:27.2122068Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-06-26T07:04:27.2214128Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-06-26T07:04:27.3125291Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-06-26T07:04:27.3411521Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-06-26T07:04:27.3479445Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-06-26T07:04:27.3787692Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-06-26T07:04:27.3986630Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-06-26T07:04:27.4350612Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:04:27.4450623Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-06-26T07:04:27.4461600Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-06-26T07:04:27.4479012Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-06-26T07:04:39.0854570Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-06-26T07:04:39.0995179Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-06-26T07:04:39.1771823Z Submodule path 'third_party/kineto': checked out '8681ff11e1fa54da39023076c5c43eddd87b7a8a' 2024-06-26T07:04:39.1782744Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:04:39.1784634Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:04:39.1786613Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:04:39.1804544Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-06-26T07:04:39.6763921Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-06-26T07:04:40.9608611Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-06-26T07:04:41.9718975Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-06-26T07:04:41.9729216Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:41.9731014Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:41.9732794Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:41.9734789Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:41.9764213Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:41.9766339Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:41.9768205Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:41.9769953Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:41.9771605Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-06-26T07:04:42.9928769Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-06-26T07:04:43.3560673Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-06-26T07:04:44.5827036Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-06-26T07:04:44.8416094Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-06-26T07:04:45.2702613Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-06-26T07:04:46.2436148Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-06-26T07:04:51.7466055Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-06-26T07:04:52.0771179Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-06-26T07:04:52.0909213Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-06-26T07:04:52.1200896Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-06-26T07:04:52.1298530Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-06-26T07:04:52.1308023Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:04:52.1326104Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-06-26T07:04:52.3924159Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-06-26T07:04:52.4064228Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-06-26T07:04:52.4401329Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-06-26T07:04:52.5208892Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-06-26T07:04:52.5328279Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-06-26T07:04:52.5628684Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-06-26T07:04:52.6123309Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-06-26T07:04:52.6428788Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-06-26T07:04:52.6627526Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-06-26T07:04:52.7482545Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-06-26T07:04:53.0244199Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-06-26T07:04:53.0268743Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-06-26T07:04:53.0270579Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-06-26T07:04:53.0287112Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-06-26T07:04:53.4401553Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-06-26T07:04:54.5822763Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-06-26T07:04:54.6096593Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-06-26T07:04:54.6619687Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-06-26T07:04:54.6633387Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:04:54.6639099Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:04:54.6641054Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:04:54.6643335Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:04:54.6645569Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:04:54.6647946Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:04:54.6649844Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:04:54.6651457Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:04:54.6665671Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-06-26T07:04:55.1450704Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-06-26T07:04:56.2271840Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-06-26T07:04:56.5160537Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-06-26T07:05:02.2129378Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-06-26T07:05:02.5855225Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-06-26T07:05:02.8191166Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-06-26T07:05:03.1182319Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-06-26T07:05:10.5210578Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-06-26T07:05:10.5542744Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-06-26T07:05:10.5668644Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-06-26T07:05:10.6503114Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-06-26T07:05:10.6596877Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-06-26T07:05:10.6703636Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-06-26T07:05:10.6816823Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-06-26T07:05:10.6827951Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:05:10.6830043Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:05:10.6847826Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-06-26T07:05:12.4973070Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-06-26T07:05:13.7156364Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-06-26T07:05:13.7522563Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:05:14.1145307Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-06-26T07:05:14.1228253Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-06-26T07:05:14.3386343Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-06-26T07:05:14.3400325Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:05:14.3402339Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-06-26T07:05:14.3421372Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-06-26T07:05:14.7220182Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-06-26T07:05:15.7066852Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-06-26T07:05:15.7692449Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-06-26T07:05:15.7755432Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-06-26T07:05:15.7844713Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-06-26T07:05:15.8122635Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-06-26T07:05:15.8342952Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-06-26T07:05:15.8674052Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-06-26T07:05:15.8876907Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-06-26T07:05:15.8887834Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:05:15.8889815Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:05:15.8892380Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:05:15.8894841Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:05:15.8911672Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-06-26T07:05:16.8383609Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-06-26T07:05:17.0500071Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-06-26T07:05:18.0672766Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-06-26T07:05:19.0622930Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-06-26T07:05:19.0739198Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-06-26T07:05:19.1330140Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-06-26T07:05:19.1562530Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-06-26T07:05:19.1570919Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:05:19.1587472Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-06-26T07:05:19.3811827Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-06-26T07:05:19.3838835Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-06-26T07:05:19.4044841Z Entering 'android/libs/fbjni' 2024-06-26T07:05:19.4073781Z Entering 'third_party/FP16' 2024-06-26T07:05:19.4101751Z Entering 'third_party/FXdiv' 2024-06-26T07:05:19.4130505Z Entering 'third_party/NNPACK' 2024-06-26T07:05:19.4161124Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:05:19.4189259Z Entering 'third_party/XNNPACK' 2024-06-26T07:05:19.4228010Z Entering 'third_party/benchmark' 2024-06-26T07:05:19.4258468Z Entering 'third_party/cpp-httplib' 2024-06-26T07:05:19.4286151Z Entering 'third_party/cpuinfo' 2024-06-26T07:05:19.4314656Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:05:19.4342789Z Entering 'third_party/cutlass' 2024-06-26T07:05:19.4378487Z Entering 'third_party/eigen' 2024-06-26T07:05:19.4409737Z Entering 'third_party/fbgemm' 2024-06-26T07:05:19.4440722Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:05:19.4469363Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:05:19.4498546Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:05:19.4529411Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:05:19.4557744Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:05:19.4584524Z Entering 'third_party/flatbuffers' 2024-06-26T07:05:19.4615833Z Entering 'third_party/fmt' 2024-06-26T07:05:19.4645724Z Entering 'third_party/foxi' 2024-06-26T07:05:19.4673185Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:05:19.4701869Z Entering 'third_party/gloo' 2024-06-26T07:05:19.4729588Z Entering 'third_party/googletest' 2024-06-26T07:05:19.4755885Z Entering 'third_party/ideep' 2024-06-26T07:05:19.4783167Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:05:19.4814643Z Entering 'third_party/ittapi' 2024-06-26T07:05:19.4840476Z Entering 'third_party/kineto' 2024-06-26T07:05:19.4868467Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:05:19.4898354Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:05:19.4928046Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:05:19.4957225Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:05:19.4982726Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:05:19.5010989Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:05:19.5037777Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:05:19.5063992Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:05:19.5091538Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:05:19.5117801Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:05:19.5144271Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:05:19.5171879Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:05:19.5198816Z Entering 'third_party/mimalloc' 2024-06-26T07:05:19.5227802Z Entering 'third_party/nccl/nccl' 2024-06-26T07:05:19.5257432Z Entering 'third_party/nlohmann' 2024-06-26T07:05:19.5285254Z Entering 'third_party/onnx' 2024-06-26T07:05:19.5326627Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:05:19.5354295Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:05:19.5384753Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:05:19.5413848Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:05:19.5440909Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:05:19.5466727Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:05:19.5493812Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:05:19.5520407Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:05:19.5547116Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:05:19.5574351Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:05:19.5598872Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:05:19.5629459Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:05:19.5659934Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:05:19.5704009Z Entering 'third_party/pocketfft' 2024-06-26T07:05:19.5732774Z Entering 'third_party/protobuf' 2024-06-26T07:05:19.5765206Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:05:19.5794545Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:05:19.5824956Z Entering 'third_party/psimd' 2024-06-26T07:05:19.5854534Z Entering 'third_party/pthreadpool' 2024-06-26T07:05:19.5881266Z Entering 'third_party/pybind11' 2024-06-26T07:05:19.5910108Z Entering 'third_party/python-peachpy' 2024-06-26T07:05:19.5939884Z Entering 'third_party/sleef' 2024-06-26T07:05:19.5968006Z Entering 'third_party/tensorpipe' 2024-06-26T07:05:19.5994978Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:05:19.6022010Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:05:19.6048696Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:05:19.6077571Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:05:19.6104387Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:05:19.6145635Z ##[endgroup] 2024-06-26T07:05:19.6148284Z ##[group]Persisting credentials for submodules 2024-06-26T07:05:19.6151310Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-06-26T07:05:19.6363452Z Entering 'android/libs/fbjni' 2024-06-26T07:05:19.6396332Z Entering 'third_party/FP16' 2024-06-26T07:05:19.6434286Z Entering 'third_party/FXdiv' 2024-06-26T07:05:19.6467494Z Entering 'third_party/NNPACK' 2024-06-26T07:05:19.6502747Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:05:19.6539520Z Entering 'third_party/XNNPACK' 2024-06-26T07:05:19.6585826Z Entering 'third_party/benchmark' 2024-06-26T07:05:19.6622267Z Entering 'third_party/cpp-httplib' 2024-06-26T07:05:19.6658099Z Entering 'third_party/cpuinfo' 2024-06-26T07:05:19.6693907Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:05:19.6729743Z Entering 'third_party/cutlass' 2024-06-26T07:05:19.6769377Z Entering 'third_party/eigen' 2024-06-26T07:05:19.6806168Z Entering 'third_party/fbgemm' 2024-06-26T07:05:19.6840687Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:05:19.6875342Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:05:19.6909139Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:05:19.6949054Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:05:19.6983958Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:05:19.7019878Z Entering 'third_party/flatbuffers' 2024-06-26T07:05:19.7056741Z Entering 'third_party/fmt' 2024-06-26T07:05:19.7093182Z Entering 'third_party/foxi' 2024-06-26T07:05:19.7128214Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:05:19.7164379Z Entering 'third_party/gloo' 2024-06-26T07:05:19.7200037Z Entering 'third_party/googletest' 2024-06-26T07:05:19.7234471Z Entering 'third_party/ideep' 2024-06-26T07:05:19.7269694Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:05:19.7310208Z Entering 'third_party/ittapi' 2024-06-26T07:05:19.7346948Z Entering 'third_party/kineto' 2024-06-26T07:05:19.7383683Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:05:19.7420055Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:05:19.7454695Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:05:19.7488983Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:05:19.7524653Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:05:19.7559391Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:05:19.7595616Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:05:19.7630768Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:05:19.7665046Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:05:19.7701331Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:05:19.7735326Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:05:19.7770801Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:05:19.7807004Z Entering 'third_party/mimalloc' 2024-06-26T07:05:19.7839924Z Entering 'third_party/nccl/nccl' 2024-06-26T07:05:19.7876202Z Entering 'third_party/nlohmann' 2024-06-26T07:05:19.7910978Z Entering 'third_party/onnx' 2024-06-26T07:05:19.7959706Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:05:19.7993287Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:05:19.8028820Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:05:19.8062314Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:05:19.8097319Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:05:19.8131670Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:05:19.8164698Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:05:19.8199744Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:05:19.8234072Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:05:19.8266271Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:05:19.8300660Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:05:19.8335077Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:05:19.8369887Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:05:19.8420673Z Entering 'third_party/pocketfft' 2024-06-26T07:05:19.8457341Z Entering 'third_party/protobuf' 2024-06-26T07:05:19.8493317Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:05:19.8526707Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:05:19.8561851Z Entering 'third_party/psimd' 2024-06-26T07:05:19.8598683Z Entering 'third_party/pthreadpool' 2024-06-26T07:05:19.8632142Z Entering 'third_party/pybind11' 2024-06-26T07:05:19.8667406Z Entering 'third_party/python-peachpy' 2024-06-26T07:05:19.8702992Z Entering 'third_party/sleef' 2024-06-26T07:05:19.8740300Z Entering 'third_party/tensorpipe' 2024-06-26T07:05:19.8776388Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:05:19.8809884Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:05:19.8843197Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:05:19.8875251Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:05:19.8907912Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:05:19.8956315Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-06-26T07:05:19.9159471Z Entering 'android/libs/fbjni' 2024-06-26T07:05:19.9191324Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-06-26T07:05:19.9205906Z Entering 'third_party/FP16' 2024-06-26T07:05:19.9239294Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-06-26T07:05:19.9252385Z Entering 'third_party/FXdiv' 2024-06-26T07:05:19.9283658Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-06-26T07:05:19.9298062Z Entering 'third_party/NNPACK' 2024-06-26T07:05:19.9331396Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-06-26T07:05:19.9343058Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:05:19.9374709Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-06-26T07:05:19.9389214Z Entering 'third_party/XNNPACK' 2024-06-26T07:05:19.9421703Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-06-26T07:05:19.9445927Z Entering 'third_party/benchmark' 2024-06-26T07:05:19.9478656Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:05:19.9492151Z Entering 'third_party/cpp-httplib' 2024-06-26T07:05:19.9523649Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-06-26T07:05:19.9538608Z Entering 'third_party/cpuinfo' 2024-06-26T07:05:19.9571839Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:05:19.9585752Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:05:19.9619038Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-06-26T07:05:19.9632725Z Entering 'third_party/cutlass' 2024-06-26T07:05:19.9664766Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:05:19.9684359Z Entering 'third_party/eigen' 2024-06-26T07:05:19.9717076Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-06-26T07:05:19.9731758Z Entering 'third_party/fbgemm' 2024-06-26T07:05:19.9764128Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-06-26T07:05:19.9777483Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:05:19.9809171Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-06-26T07:05:19.9822121Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:05:19.9854171Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:05:19.9867231Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:05:19.9899421Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:05:19.9916119Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:05:19.9947858Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:05:19.9961066Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:05:19.9991992Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-06-26T07:05:20.0005421Z Entering 'third_party/flatbuffers' 2024-06-26T07:05:20.0037136Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-06-26T07:05:20.0052907Z Entering 'third_party/fmt' 2024-06-26T07:05:20.0085131Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:05:20.0098786Z Entering 'third_party/foxi' 2024-06-26T07:05:20.0130686Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-06-26T07:05:20.0144012Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:05:20.0176141Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-06-26T07:05:20.0189026Z Entering 'third_party/gloo' 2024-06-26T07:05:20.0223772Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-06-26T07:05:20.0237976Z Entering 'third_party/googletest' 2024-06-26T07:05:20.0268885Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:05:20.0281663Z Entering 'third_party/ideep' 2024-06-26T07:05:20.0312192Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-06-26T07:05:20.0324483Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:05:20.0356262Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-06-26T07:05:20.0375003Z Entering 'third_party/ittapi' 2024-06-26T07:05:20.0407854Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-06-26T07:05:20.0422224Z Entering 'third_party/kineto' 2024-06-26T07:05:20.0455112Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-06-26T07:05:20.0466281Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:05:20.0498592Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-06-26T07:05:20.0510070Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:05:20.0543072Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-06-26T07:05:20.0557339Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:05:20.0589411Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-06-26T07:05:20.0602719Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:05:20.0635449Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:05:20.0648867Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:05:20.0680264Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-06-26T07:05:20.0691387Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:05:20.0723233Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-06-26T07:05:20.0737489Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:05:20.0770296Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-06-26T07:05:20.0783044Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:05:20.0816499Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:05:20.0828891Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:05:20.0860302Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-06-26T07:05:20.0872932Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:05:20.0903933Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-06-26T07:05:20.0918809Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:05:20.0948483Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-06-26T07:05:20.0960390Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:05:20.0991041Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-06-26T07:05:20.1005606Z Entering 'third_party/mimalloc' 2024-06-26T07:05:20.1038725Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-06-26T07:05:20.1052926Z Entering 'third_party/nccl/nccl' 2024-06-26T07:05:20.1084795Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-06-26T07:05:20.1098489Z Entering 'third_party/nlohmann' 2024-06-26T07:05:20.1130262Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-06-26T07:05:20.1145211Z Entering 'third_party/onnx' 2024-06-26T07:05:20.1177982Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-06-26T07:05:20.1202225Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:05:20.1235115Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:05:20.1248262Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:05:20.1278624Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:05:20.1293603Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:05:20.1325148Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-06-26T07:05:20.1337560Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:05:20.1370306Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:05:20.1383898Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:05:20.1417368Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:05:20.1429174Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:05:20.1461961Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-06-26T07:05:20.1475152Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:05:20.1507358Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-06-26T07:05:20.1520883Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:05:20.1551723Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-06-26T07:05:20.1564607Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:05:20.1599954Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-06-26T07:05:20.1612285Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:05:20.1643458Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-06-26T07:05:20.1655908Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:05:20.1687426Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-06-26T07:05:20.1700715Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:05:20.1732646Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-06-26T07:05:20.1747229Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:05:20.1779840Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-06-26T07:05:20.1807191Z Entering 'third_party/pocketfft' 2024-06-26T07:05:20.1839460Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-06-26T07:05:20.1853691Z Entering 'third_party/protobuf' 2024-06-26T07:05:20.1887110Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-06-26T07:05:20.1900692Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:05:20.1933272Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:05:20.1946019Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:05:20.1979092Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:05:20.1993245Z Entering 'third_party/psimd' 2024-06-26T07:05:20.2025576Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-06-26T07:05:20.2037685Z Entering 'third_party/pthreadpool' 2024-06-26T07:05:20.2069535Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-06-26T07:05:20.2082805Z Entering 'third_party/pybind11' 2024-06-26T07:05:20.2115658Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:05:20.2128551Z Entering 'third_party/python-peachpy' 2024-06-26T07:05:20.2160949Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-06-26T07:05:20.2174352Z Entering 'third_party/sleef' 2024-06-26T07:05:20.2206591Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-06-26T07:05:20.2218516Z Entering 'third_party/tensorpipe' 2024-06-26T07:05:20.2249548Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-06-26T07:05:20.2263382Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:05:20.2294714Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:05:20.2307645Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:05:20.2340786Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-06-26T07:05:20.2351880Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:05:20.2384854Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-06-26T07:05:20.2397805Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:05:20.2429293Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:05:20.2440355Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:05:20.2471572Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-06-26T07:05:20.3363951Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-06-26T07:05:20.3567988Z Entering 'android/libs/fbjni' 2024-06-26T07:05:20.3594485Z Entering 'third_party/FP16' 2024-06-26T07:05:20.3619474Z Entering 'third_party/FXdiv' 2024-06-26T07:05:20.3648791Z Entering 'third_party/NNPACK' 2024-06-26T07:05:20.3675489Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:05:20.3704452Z Entering 'third_party/XNNPACK' 2024-06-26T07:05:20.3742639Z Entering 'third_party/benchmark' 2024-06-26T07:05:20.3772519Z Entering 'third_party/cpp-httplib' 2024-06-26T07:05:20.3802092Z Entering 'third_party/cpuinfo' 2024-06-26T07:05:20.3831002Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:05:20.3860167Z Entering 'third_party/cutlass' 2024-06-26T07:05:20.3892964Z Entering 'third_party/eigen' 2024-06-26T07:05:20.3922214Z Entering 'third_party/fbgemm' 2024-06-26T07:05:20.3951582Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:05:20.3979038Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:05:20.4005704Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:05:20.4038096Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:05:20.4067954Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:05:20.4098091Z Entering 'third_party/flatbuffers' 2024-06-26T07:05:20.4129464Z Entering 'third_party/fmt' 2024-06-26T07:05:20.4157662Z Entering 'third_party/foxi' 2024-06-26T07:05:20.4185648Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:05:20.4213633Z Entering 'third_party/gloo' 2024-06-26T07:05:20.4244045Z Entering 'third_party/googletest' 2024-06-26T07:05:20.4271901Z Entering 'third_party/ideep' 2024-06-26T07:05:20.4300429Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:05:20.4332709Z Entering 'third_party/ittapi' 2024-06-26T07:05:20.4359109Z Entering 'third_party/kineto' 2024-06-26T07:05:20.4386171Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:05:20.4414475Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:05:20.4443571Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:05:20.4469851Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:05:20.4496588Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:05:20.4524694Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:05:20.4550782Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:05:20.4577822Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:05:20.4607113Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:05:20.4635319Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:05:20.4663912Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:05:20.4693442Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:05:20.4720066Z Entering 'third_party/mimalloc' 2024-06-26T07:05:20.4747691Z Entering 'third_party/nccl/nccl' 2024-06-26T07:05:20.4777885Z Entering 'third_party/nlohmann' 2024-06-26T07:05:20.4806367Z Entering 'third_party/onnx' 2024-06-26T07:05:20.4843834Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:05:20.4875073Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:05:20.4907049Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:05:20.4938689Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:05:20.4966420Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:05:20.4992046Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:05:20.5018802Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:05:20.5048788Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:05:20.5077921Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:05:20.5104996Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:05:20.5131539Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:05:20.5161748Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:05:20.5190566Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:05:20.5236045Z Entering 'third_party/pocketfft' 2024-06-26T07:05:20.5265083Z Entering 'third_party/protobuf' 2024-06-26T07:05:20.5296979Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:05:20.5324635Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:05:20.5352181Z Entering 'third_party/psimd' 2024-06-26T07:05:20.5380952Z Entering 'third_party/pthreadpool' 2024-06-26T07:05:20.5408369Z Entering 'third_party/pybind11' 2024-06-26T07:05:20.5438363Z Entering 'third_party/python-peachpy' 2024-06-26T07:05:20.5465766Z Entering 'third_party/sleef' 2024-06-26T07:05:20.5495340Z Entering 'third_party/tensorpipe' 2024-06-26T07:05:20.5523492Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:05:20.5551507Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:05:20.5576801Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:05:20.5605059Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:05:20.5630131Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:05:20.5673447Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-06-26T07:05:20.5879153Z Entering 'android/libs/fbjni' 2024-06-26T07:05:20.5908515Z Entering 'third_party/FP16' 2024-06-26T07:05:20.5934878Z Entering 'third_party/FXdiv' 2024-06-26T07:05:20.5965271Z Entering 'third_party/NNPACK' 2024-06-26T07:05:20.5993136Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:05:20.6021858Z Entering 'third_party/XNNPACK' 2024-06-26T07:05:20.6060195Z Entering 'third_party/benchmark' 2024-06-26T07:05:20.6088379Z Entering 'third_party/cpp-httplib' 2024-06-26T07:05:20.6116934Z Entering 'third_party/cpuinfo' 2024-06-26T07:05:20.6144348Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:05:20.6174808Z Entering 'third_party/cutlass' 2024-06-26T07:05:20.6209427Z Entering 'third_party/eigen' 2024-06-26T07:05:20.6240063Z Entering 'third_party/fbgemm' 2024-06-26T07:05:20.6266342Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:05:20.6294166Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:05:20.6323073Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:05:20.6356468Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:05:20.6384536Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:05:20.6413205Z Entering 'third_party/flatbuffers' 2024-06-26T07:05:20.6442272Z Entering 'third_party/fmt' 2024-06-26T07:05:20.6469950Z Entering 'third_party/foxi' 2024-06-26T07:05:20.6498345Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:05:20.6526318Z Entering 'third_party/gloo' 2024-06-26T07:05:20.6552634Z Entering 'third_party/googletest' 2024-06-26T07:05:20.6582714Z Entering 'third_party/ideep' 2024-06-26T07:05:20.6608797Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:05:20.6642998Z Entering 'third_party/ittapi' 2024-06-26T07:05:20.6669451Z Entering 'third_party/kineto' 2024-06-26T07:05:20.6699594Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:05:20.6726606Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:05:20.6756501Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:05:20.6782440Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:05:20.6809693Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:05:20.6837382Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:05:20.6866122Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:05:20.6892751Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:05:20.6921943Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:05:20.6948831Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:05:20.6979667Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:05:20.7008768Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:05:20.7035697Z Entering 'third_party/mimalloc' 2024-06-26T07:05:20.7064193Z Entering 'third_party/nccl/nccl' 2024-06-26T07:05:20.7093362Z Entering 'third_party/nlohmann' 2024-06-26T07:05:20.7123302Z Entering 'third_party/onnx' 2024-06-26T07:05:20.7165350Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:05:20.7194339Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:05:20.7222551Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:05:20.7255366Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:05:20.7280974Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:05:20.7308819Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:05:20.7337445Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:05:20.7363851Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:05:20.7391205Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:05:20.7419854Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:05:20.7447765Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:05:20.7475534Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:05:20.7503824Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:05:20.7548806Z Entering 'third_party/pocketfft' 2024-06-26T07:05:20.7579020Z Entering 'third_party/protobuf' 2024-06-26T07:05:20.7608591Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:05:20.7637739Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:05:20.7666149Z Entering 'third_party/psimd' 2024-06-26T07:05:20.7696896Z Entering 'third_party/pthreadpool' 2024-06-26T07:05:20.7727712Z Entering 'third_party/pybind11' 2024-06-26T07:05:20.7754785Z Entering 'third_party/python-peachpy' 2024-06-26T07:05:20.7782880Z Entering 'third_party/sleef' 2024-06-26T07:05:20.7810931Z Entering 'third_party/tensorpipe' 2024-06-26T07:05:20.7839748Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:05:20.7868447Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:05:20.7897692Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:05:20.7924224Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:05:20.7951350Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:05:20.7991221Z ##[endgroup] 2024-06-26T07:05:20.8020195Z [command]/usr/bin/git log -1 --format='%H' 2024-06-26T07:05:20.8037480Z 'b8c4c54d347aa776934c60784e35936878ef18dc' 2024-06-26T07:05:20.8206808Z Prepare all required actions 2024-06-26T07:05:20.8207257Z Getting action download info 2024-06-26T07:05:20.9674401Z ##[group]Run ./.github/actions/setup-linux 2024-06-26T07:05:20.9675063Z env: 2024-06-26T07:05:20.9675341Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:20.9675689Z ##[endgroup] 2024-06-26T07:05:20.9735271Z ##[group]Run set -euo pipefail 2024-06-26T07:05:20.9735710Z set -euo pipefail 2024-06-26T07:05:20.9736100Z function get_ec2_metadata() { 2024-06-26T07:05:20.9736636Z  # Pulled from instance metadata endpoint for EC2 2024-06-26T07:05:20.9737526Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-06-26T07:05:20.9738298Z  category=$1 2024-06-26T07:05:20.9738821Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-06-26T07:05:20.9739458Z  runner_name_str=i-02844b9639d793963 2024-06-26T07:05:20.9739954Z  if [[ -f /.inarc ]]; then 2024-06-26T07:05:20.9740449Z  echo "ARC Runner, no info on ec2 metadata" 2024-06-26T07:05:20.9741019Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-06-26T07:05:20.9741729Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-06-26T07:05:20.9742337Z  else 2024-06-26T07:05:20.9742826Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-06-26T07:05:20.9743403Z  fi 2024-06-26T07:05:20.9743691Z } 2024-06-26T07:05:20.9744051Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-06-26T07:05:20.9744656Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-06-26T07:05:20.9745339Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-06-26T07:05:20.9745910Z echo "system info $(uname -a)" 2024-06-26T07:05:20.9754003Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:20.9754496Z env: 2024-06-26T07:05:20.9754781Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:20.9755125Z ##[endgroup] 2024-06-26T07:05:20.9825235Z ami-id: ami-0ce0c36d7a00b20e2 2024-06-26T07:05:20.9875583Z instance-id: i-02844b9639d793963 2024-06-26T07:05:20.9923514Z instance-type: g5.4xlarge 2024-06-26T07:05:20.9929726Z system info Linux ip-10-0-62-2.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-06-26T07:05:20.9949012Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:05:20.9950055Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:05:20.9958199Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:20.9958695Z env: 2024-06-26T07:05:20.9958972Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:20.9959320Z ##[endgroup] 2024-06-26T07:05:21.0020237Z ##[group]Run if systemctl is-active --quiet docker; then 2024-06-26T07:05:21.0020853Z if systemctl is-active --quiet docker; then 2024-06-26T07:05:21.0021401Z  echo "Docker daemon is running..."; 2024-06-26T07:05:21.0021991Z else 2024-06-26T07:05:21.0022483Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-06-26T07:05:21.0023068Z fi 2024-06-26T07:05:21.0030673Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:21.0031175Z env: 2024-06-26T07:05:21.0031445Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:21.0031794Z ##[endgroup] 2024-06-26T07:05:21.0066495Z Docker daemon is running... 2024-06-26T07:05:21.0110480Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:05:21.0111039Z with: 2024-06-26T07:05:21.0111306Z shell: bash 2024-06-26T07:05:21.0111593Z timeout_minutes: 5 2024-06-26T07:05:21.0111912Z max_attempts: 3 2024-06-26T07:05:21.0112225Z retry_wait_seconds: 30 2024-06-26T07:05:21.0114001Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-06-26T07:05:21.0115462Z polling_interval_seconds: 1 2024-06-26T07:05:21.0115838Z warning_on_retry: true 2024-06-26T07:05:21.0116190Z continue_on_error: false 2024-06-26T07:05:21.0116522Z env: 2024-06-26T07:05:21.0116782Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:21.0117136Z AWS_RETRY_MODE: standard 2024-06-26T07:05:21.0117483Z AWS_MAX_ATTEMPTS: 5 2024-06-26T07:05:21.0117813Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:05:21.0118182Z ##[endgroup] 2024-06-26T07:05:21.8554853Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:05:21.8555710Z Configure a credential helper to remove this warning. See 2024-06-26T07:05:21.8556615Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:05:21.8557132Z 2024-06-26T07:05:21.8557285Z Login Succeeded 2024-06-26T07:05:22.0609099Z Command completed after 1 attempt(s). 2024-06-26T07:05:22.0650166Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:05:22.0650905Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:05:22.0651571Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:05:22.0659305Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:22.0659805Z env: 2024-06-26T07:05:22.0660086Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:22.0660437Z ##[endgroup] 2024-06-26T07:05:22.0719350Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:05:22.0720125Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:05:22.0720720Z # shellcheck disable=SC2046 2024-06-26T07:05:22.0721180Z docker stop $(docker ps -q) || true 2024-06-26T07:05:22.0721668Z # Prune all of the docker images 2024-06-26T07:05:22.0722124Z docker system prune -af 2024-06-26T07:05:22.0729938Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:22.0730453Z env: 2024-06-26T07:05:22.0730722Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:22.0731073Z ##[endgroup] 2024-06-26T07:05:22.0976132Z "docker stop" requires at least 1 argument. 2024-06-26T07:05:22.0976761Z See 'docker stop --help'. 2024-06-26T07:05:22.0976997Z 2024-06-26T07:05:22.0977222Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-06-26T07:05:22.0977603Z 2024-06-26T07:05:22.0977751Z Stop one or more running containers 2024-06-26T07:05:22.1122313Z Total reclaimed space: 0B 2024-06-26T07:05:22.1150504Z ##[group]Run set +e 2024-06-26T07:05:22.1150858Z set +e 2024-06-26T07:05:22.1151159Z set -x 2024-06-26T07:05:22.1151451Z  2024-06-26T07:05:22.1151768Z PT_DOMAIN=download.pytorch.org 2024-06-26T07:05:22.1152870Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-06-26T07:05:22.1153970Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-06-26T07:05:22.1154880Z # one is returned at random 2024-06-26T07:05:22.1155425Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-06-26T07:05:22.1155935Z  2024-06-26T07:05:22.1156252Z if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:05:22.1156876Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-06-26T07:05:22.1157642Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-06-26T07:05:22.1158194Z  2024-06-26T07:05:22.1158519Z  if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:05:22.1159079Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-06-26T07:05:22.1159588Z  exit 1 2024-06-26T07:05:22.1159892Z  fi 2024-06-26T07:05:22.1160180Z fi 2024-06-26T07:05:22.1160565Z  2024-06-26T07:05:22.1160929Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-06-26T07:05:22.1161451Z  # Clean up any old records first 2024-06-26T07:05:22.1161969Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-06-26T07:05:22.1162425Z fi 2024-06-26T07:05:22.1162692Z  2024-06-26T07:05:22.1163123Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-06-26T07:05:22.1163674Z cat /etc/hosts 2024-06-26T07:05:22.1171422Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:22.1171918Z env: 2024-06-26T07:05:22.1172193Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:22.1172545Z ##[endgroup] 2024-06-26T07:05:22.1195932Z + PT_DOMAIN=download.pytorch.org 2024-06-26T07:05:22.1199659Z ++ dig -4 +short download.pytorch.org 2024-06-26T07:05:22.1200104Z ++ tail -n1 2024-06-26T07:05:22.1659434Z + RESOLVED_IP=18.160.10.22 2024-06-26T07:05:22.1659876Z + '[' -z 18.160.10.22 ']' 2024-06-26T07:05:22.1660385Z + grep -r download.pytorch.org /etc/hosts 2024-06-26T07:05:22.1666636Z 18.160.18.34 download.pytorch.org 2024-06-26T07:05:22.1667329Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-06-26T07:05:22.1763087Z + echo '18.160.10.22 download.pytorch.org' 2024-06-26T07:05:22.1763774Z + sudo tee -a /etc/hosts 2024-06-26T07:05:22.2144682Z 18.160.10.22 download.pytorch.org 2024-06-26T07:05:22.2158126Z + cat /etc/hosts 2024-06-26T07:05:22.2162373Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-06-26T07:05:22.2175333Z ::1 localhost6 localhost6.localdomain6 2024-06-26T07:05:22.2175851Z 18.160.10.22 download.pytorch.org 2024-06-26T07:05:22.2386997Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-06-26T07:05:22.2387603Z with: 2024-06-26T07:05:22.2388703Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:22.2390048Z docker-build-dir: .ci/docker 2024-06-26T07:05:22.2390430Z working-directory: . 2024-06-26T07:05:22.2390912Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:22.2391441Z force-push: false 2024-06-26T07:05:22.2391734Z env: 2024-06-26T07:05:22.2391996Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:22.2392524Z ##[endgroup] 2024-06-26T07:05:22.2409398Z ##[group]Run set -ex 2024-06-26T07:05:22.2409943Z set -ex 2024-06-26T07:05:22.2410238Z  2024-06-26T07:05:22.2410802Z # If the docker build directory or the build script doesn't exist, the action will 2024-06-26T07:05:22.2411806Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-06-26T07:05:22.2412618Z # job could then download the pre-built image as usual 2024-06-26T07:05:22.2413494Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-06-26T07:05:22.2414194Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:22.2414890Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:22.2415589Z  2024-06-26T07:05:22.2416105Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-06-26T07:05:22.2416731Z  exit 0 2024-06-26T07:05:22.2417033Z else 2024-06-26T07:05:22.2417407Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:22.2417854Z fi 2024-06-26T07:05:22.2418133Z  2024-06-26T07:05:22.2418611Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-06-26T07:05:22.2419462Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-06-26T07:05:22.2420236Z  # use it as it is, but first let's extract the tag 2024-06-26T07:05:22.2420951Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-06-26T07:05:22.2421675Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:22.2422379Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:22.2422943Z else 2024-06-26T07:05:22.2423383Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-06-26T07:05:22.2424050Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:22.2424998Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:22.2425739Z fi 2024-06-26T07:05:22.2433687Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:22.2434190Z env: 2024-06-26T07:05:22.2434463Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:22.2434811Z REPO_NAME: pytorch 2024-06-26T07:05:22.2435946Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:22.2437158Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:05:22.2437657Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:22.2438180Z ##[endgroup] 2024-06-26T07:05:22.2457768Z + [[ ! -d .ci/docker ]] 2024-06-26T07:05:22.2458202Z + [[ ! -f .ci/docker/build.sh ]] 2024-06-26T07:05:22.2458586Z + echo skip=false 2024-06-26T07:05:22.2460570Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-06-26T07:05:22.2463344Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:22.2464548Z ++ awk -F '[:,]' '{print $2}' 2024-06-26T07:05:22.2470818Z + DOCKER_TAG=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:22.2471466Z + echo docker-tag=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:22.2473355Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:22.2499609Z ##[group]Run set +e 2024-06-26T07:05:22.2499975Z set +e 2024-06-26T07:05:22.2500276Z set -x 2024-06-26T07:05:22.2500559Z  2024-06-26T07:05:22.2500843Z login() { 2024-06-26T07:05:22.2501518Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:05:22.2502246Z } 2024-06-26T07:05:22.2502519Z  2024-06-26T07:05:22.2502794Z retry () { 2024-06-26T07:05:22.2503186Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:05:22.2503645Z } 2024-06-26T07:05:22.2503914Z  2024-06-26T07:05:22.2504234Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:05:22.2504659Z  2024-06-26T07:05:22.2505176Z # Check if image already exists, if it does then skip building it 2024-06-26T07:05:22.2506008Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-06-26T07:05:22.2506512Z  exit 0 2024-06-26T07:05:22.2506817Z fi 2024-06-26T07:05:22.2507091Z  2024-06-26T07:05:22.2507602Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-06-26T07:05:22.2508447Z # be empty. The default action would be to continue rebuild the image 2024-06-26T07:05:22.2509202Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-06-26T07:05:22.2510034Z  # if we're on the base branch then use the parent commit 2024-06-26T07:05:22.2510644Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-06-26T07:05:22.2511086Z else 2024-06-26T07:05:22.2511558Z  # otherwise we're on a PR, so use the most recent base commit 2024-06-26T07:05:22.2512251Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-06-26T07:05:22.2513004Z fi 2024-06-26T07:05:22.2513287Z  2024-06-26T07:05:22.2513607Z if [[ -z "${MERGE_BASE}" ]]; then 2024-06-26T07:05:22.2514105Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:22.2514568Z  2024-06-26T07:05:22.2515222Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-06-26T07:05:22.2515981Z  exit 0 2024-06-26T07:05:22.2516274Z fi 2024-06-26T07:05:22.2516556Z  2024-06-26T07:05:22.2516995Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-06-26T07:05:22.2517968Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-06-26T07:05:22.2518766Z  exit 1 2024-06-26T07:05:22.2519073Z fi 2024-06-26T07:05:22.2519350Z  2024-06-26T07:05:22.2519840Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-06-26T07:05:22.2520788Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-06-26T07:05:22.2521658Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-06-26T07:05:22.2522633Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-06-26T07:05:22.2523718Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-06-26T07:05:22.2524471Z fi 2024-06-26T07:05:22.2524750Z  2024-06-26T07:05:22.2525107Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:22.2532820Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:22.2533327Z env: 2024-06-26T07:05:22.2533604Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:22.2533967Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:05:22.2534435Z BASE_REVISION: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:05:22.2535777Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:22.2537034Z DOCKER_TAG: 91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:22.2537634Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:22.2538156Z ##[endgroup] 2024-06-26T07:05:22.2557459Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:22.2558117Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:22.2559127Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:05:22.2559901Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:22.6565971Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:05:22.6566782Z Configure a credential helper to remove this warning. See 2024-06-26T07:05:22.6567772Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:05:22.6568290Z 2024-06-26T07:05:22.6568412Z Login Succeeded 2024-06-26T07:05:22.6577883Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:22.8473550Z { 2024-06-26T07:05:22.8473945Z "schemaVersion": 2, 2024-06-26T07:05:22.8474633Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-06-26T07:05:22.8475404Z "config": { 2024-06-26T07:05:22.8475945Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-06-26T07:05:22.8476603Z "size": 48681, 2024-06-26T07:05:22.8477178Z "digest": "sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd" 2024-06-26T07:05:22.8477850Z }, 2024-06-26T07:05:22.8478106Z "layers": [ 2024-06-26T07:05:22.8478397Z { 2024-06-26T07:05:22.8478829Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8479495Z "size": 28580681, 2024-06-26T07:05:22.8480280Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-06-26T07:05:22.8481128Z }, 2024-06-26T07:05:22.8481448Z { 2024-06-26T07:05:22.8482033Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8482768Z "size": 7943451, 2024-06-26T07:05:22.8483493Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-06-26T07:05:22.8484314Z }, 2024-06-26T07:05:22.8484834Z { 2024-06-26T07:05:22.8485317Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8485878Z "size": 55728572, 2024-06-26T07:05:22.8486434Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-06-26T07:05:22.8487091Z }, 2024-06-26T07:05:22.8487387Z { 2024-06-26T07:05:22.8487839Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8488406Z "size": 186, 2024-06-26T07:05:22.8488949Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-06-26T07:05:22.8489571Z }, 2024-06-26T07:05:22.8489816Z { 2024-06-26T07:05:22.8490247Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8490797Z "size": 6886, 2024-06-26T07:05:22.8491330Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-06-26T07:05:22.8491949Z }, 2024-06-26T07:05:22.8492375Z { 2024-06-26T07:05:22.8492809Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8493355Z "size": 1329001756, 2024-06-26T07:05:22.8493934Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-06-26T07:05:22.8494566Z }, 2024-06-26T07:05:22.8494806Z { 2024-06-26T07:05:22.8495287Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8495845Z "size": 62501, 2024-06-26T07:05:22.8496396Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-06-26T07:05:22.8497008Z }, 2024-06-26T07:05:22.8497253Z { 2024-06-26T07:05:22.8497681Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8498228Z "size": 1684, 2024-06-26T07:05:22.8498749Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-06-26T07:05:22.8499351Z }, 2024-06-26T07:05:22.8499595Z { 2024-06-26T07:05:22.8500017Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8500564Z "size": 1523, 2024-06-26T07:05:22.8501111Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-06-26T07:05:22.8501742Z }, 2024-06-26T07:05:22.8501979Z { 2024-06-26T07:05:22.8502409Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8502960Z "size": 2528295403, 2024-06-26T07:05:22.8503526Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-06-26T07:05:22.8504230Z }, 2024-06-26T07:05:22.8504478Z { 2024-06-26T07:05:22.8504905Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8505455Z "size": 86016, 2024-06-26T07:05:22.8505996Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-06-26T07:05:22.8506619Z }, 2024-06-26T07:05:22.8506860Z { 2024-06-26T07:05:22.8507285Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8507832Z "size": 1823, 2024-06-26T07:05:22.8508361Z "digest": "sha256:309b872da805fcdf74137a887464876fa0615dc8b24199307119eb65fc2bfb49" 2024-06-26T07:05:22.8508964Z }, 2024-06-26T07:05:22.8509200Z { 2024-06-26T07:05:22.8509628Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8510359Z "size": 246737542, 2024-06-26T07:05:22.8510910Z "digest": "sha256:e82932cb002c06a782077b8e5965f8733eb8563406a23cab260d9d8c4a994f7d" 2024-06-26T07:05:22.8511518Z }, 2024-06-26T07:05:22.8511762Z { 2024-06-26T07:05:22.8512186Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8512929Z "size": 545, 2024-06-26T07:05:22.8513576Z "digest": "sha256:ca9dd73e3e9378225ebdcfd12199ef44649e34fe55003cfbc8afe2e8a4527c5b" 2024-06-26T07:05:22.8514206Z }, 2024-06-26T07:05:22.8514452Z { 2024-06-26T07:05:22.8514869Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8515422Z "size": 1283, 2024-06-26T07:05:22.8515952Z "digest": "sha256:16d76a79d5d6616e228537785ec83a869f68a402520bbc6e826201621ca6cb98" 2024-06-26T07:05:22.8516553Z }, 2024-06-26T07:05:22.8516791Z { 2024-06-26T07:05:22.8517213Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8517760Z "size": 485, 2024-06-26T07:05:22.8518301Z "digest": "sha256:4bdf41c9ecbe1651e0986ccbea4663a0556af27b7145a679cfa45c35dc96b891" 2024-06-26T07:05:22.8518921Z }, 2024-06-26T07:05:22.8519164Z { 2024-06-26T07:05:22.8519591Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8520133Z "size": 91710937, 2024-06-26T07:05:22.8520697Z "digest": "sha256:f4863801e30cdbcd1d4e6cb1cbbcde7dbcaf6ae317ca80429295741ce60ac31d" 2024-06-26T07:05:22.8521314Z }, 2024-06-26T07:05:22.8521558Z { 2024-06-26T07:05:22.8521974Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8522515Z "size": 3233, 2024-06-26T07:05:22.8523148Z "digest": "sha256:d37dd9ed8948a0d02e1198bbedadbca36477a0980dd553196385aab0c025581d" 2024-06-26T07:05:22.8523765Z }, 2024-06-26T07:05:22.8524004Z { 2024-06-26T07:05:22.8524455Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8525024Z "size": 1905, 2024-06-26T07:05:22.8525566Z "digest": "sha256:cf2f34f225dbed17c1a9a8b9ea4ad030aba797774a716dc0a82fe212324aa751" 2024-06-26T07:05:22.8526187Z }, 2024-06-26T07:05:22.8526437Z { 2024-06-26T07:05:22.8526862Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8527402Z "size": 702, 2024-06-26T07:05:22.8527941Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:05:22.8528561Z }, 2024-06-26T07:05:22.8528803Z { 2024-06-26T07:05:22.8529220Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8529776Z "size": 2784789002, 2024-06-26T07:05:22.8530344Z "digest": "sha256:3d914950f8addfc4d657a2365c76e1143eb80b710cd3f6e9ae05640217fc6347" 2024-06-26T07:05:22.8530954Z }, 2024-06-26T07:05:22.8531191Z { 2024-06-26T07:05:22.8531621Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8532166Z "size": 381, 2024-06-26T07:05:22.8532696Z "digest": "sha256:96d0e2032184c57e62a652aaffcf0241f0d0c7a23e894ca0ad192c4a64a36583" 2024-06-26T07:05:22.8533315Z }, 2024-06-26T07:05:22.8533557Z { 2024-06-26T07:05:22.8534088Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8534680Z "size": 11879, 2024-06-26T07:05:22.8535216Z "digest": "sha256:06e12ce59790cb8b091e5d619c2a62a28329008ee184a8b9a562e3db12471f30" 2024-06-26T07:05:22.8535822Z }, 2024-06-26T07:05:22.8536070Z { 2024-06-26T07:05:22.8536487Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8537034Z "size": 803, 2024-06-26T07:05:22.8537583Z "digest": "sha256:ed712f63cde7070bdec2a6179c4c35e752c6991ddbc69fe9f2724a9903250ceb" 2024-06-26T07:05:22.8538200Z }, 2024-06-26T07:05:22.8538438Z { 2024-06-26T07:05:22.8538862Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8539403Z "size": 106, 2024-06-26T07:05:22.8539917Z "digest": "sha256:a97a71ef8a4e66234dfc2098834f8395ac5630240e60164d1a251fdb0115ba85" 2024-06-26T07:05:22.8540513Z }, 2024-06-26T07:05:22.8540757Z { 2024-06-26T07:05:22.8541178Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8541722Z "size": 503, 2024-06-26T07:05:22.8542248Z "digest": "sha256:25cd0c482ad612f936b35b4f220c580e16be982226aac8801a6a849a6bfdc221" 2024-06-26T07:05:22.8542859Z }, 2024-06-26T07:05:22.8543102Z { 2024-06-26T07:05:22.8543517Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8544066Z "size": 121477177, 2024-06-26T07:05:22.8544619Z "digest": "sha256:9bf94bc456584365fda21d0ed83b7cc690b08a71727b71c4c1e667e15028a386" 2024-06-26T07:05:22.8545276Z }, 2024-06-26T07:05:22.8545522Z { 2024-06-26T07:05:22.8545945Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8546488Z "size": 109, 2024-06-26T07:05:22.8547011Z "digest": "sha256:de45a14c9b227e4b0c9d45d42223b50a71c03522ddd9e2ca0d9517835206c823" 2024-06-26T07:05:22.8547627Z }, 2024-06-26T07:05:22.8547876Z { 2024-06-26T07:05:22.8548299Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8548845Z "size": 491, 2024-06-26T07:05:22.8549372Z "digest": "sha256:e99241e77209b0b35a0fb59a61c0821fc161190178af7e4af45e6eeaa043375e" 2024-06-26T07:05:22.8550070Z }, 2024-06-26T07:05:22.8550307Z { 2024-06-26T07:05:22.8550729Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8551268Z "size": 291, 2024-06-26T07:05:22.8551810Z "digest": "sha256:35e8f4e2a42f9df0c969dedd2199cf71719cf5ecfa676dfa014bbdeb4fdb295f" 2024-06-26T07:05:22.8552518Z }, 2024-06-26T07:05:22.8552847Z { 2024-06-26T07:05:22.8553277Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8553818Z "size": 101, 2024-06-26T07:05:22.8554338Z "digest": "sha256:a823ffa4c541394b52953391dc0a6867e26c37fe4a71fa022a2cc3f07f0ba70d" 2024-06-26T07:05:22.8554995Z }, 2024-06-26T07:05:22.8555241Z { 2024-06-26T07:05:22.8555664Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8556201Z "size": 1473, 2024-06-26T07:05:22.8556732Z "digest": "sha256:44ba272eae2064603605a6238b3f35cc6578859d660068e702a047a9cd8ab2fd" 2024-06-26T07:05:22.8557324Z }, 2024-06-26T07:05:22.8557559Z { 2024-06-26T07:05:22.8557981Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8558529Z "size": 446710781, 2024-06-26T07:05:22.8559078Z "digest": "sha256:da02d278e5240c285bc855ce4e42cd6121d5b903787f5c14f03e575ad1268650" 2024-06-26T07:05:22.8559673Z }, 2024-06-26T07:05:22.8559917Z { 2024-06-26T07:05:22.8560343Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8560884Z "size": 161, 2024-06-26T07:05:22.8561392Z "digest": "sha256:73a0ce681c295135456e190578733273276b38b4dd5babbb39298f384d43b1c6" 2024-06-26T07:05:22.8561989Z }, 2024-06-26T07:05:22.8562234Z { 2024-06-26T07:05:22.8562651Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8563196Z "size": 564, 2024-06-26T07:05:22.8563733Z "digest": "sha256:49737bd5ae709dbf7956e21fc8b05ba009e9e6c9b130b0126a73bd3d3f5efc78" 2024-06-26T07:05:22.8564429Z }, 2024-06-26T07:05:22.8564668Z { 2024-06-26T07:05:22.8565145Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8565686Z "size": 35874600, 2024-06-26T07:05:22.8566231Z "digest": "sha256:9cd4c745f8e4f9d5c9bb8050923575afad21f5b7852975a4a1981bc9eff282c5" 2024-06-26T07:05:22.8566830Z }, 2024-06-26T07:05:22.8567078Z { 2024-06-26T07:05:22.8567505Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8568050Z "size": 104, 2024-06-26T07:05:22.8568557Z "digest": "sha256:b2950a0365345d4c17a48627c905fd8930b3b3154c727f0526eb9ca547102bb7" 2024-06-26T07:05:22.8569146Z }, 2024-06-26T07:05:22.8569392Z { 2024-06-26T07:05:22.8569809Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8570352Z "size": 425, 2024-06-26T07:05:22.8570871Z "digest": "sha256:25e19d81d9fc081962ff4cbc09e098f4035419822712c19a410f5282300dfb9e" 2024-06-26T07:05:22.8571470Z }, 2024-06-26T07:05:22.8571705Z { 2024-06-26T07:05:22.8572124Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8572668Z "size": 20262084, 2024-06-26T07:05:22.8573226Z "digest": "sha256:60156b40fcbec2c25f0170b0a82fa8cd9303e0daae8bb1811fd0fc6562e88e8e" 2024-06-26T07:05:22.8573829Z }, 2024-06-26T07:05:22.8574077Z { 2024-06-26T07:05:22.8574502Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8575053Z "size": 439, 2024-06-26T07:05:22.8575618Z "digest": "sha256:7a868239227fd19b30a19927bbc8f058d247403fe5b343c55f4b4b8987918911" 2024-06-26T07:05:22.8576214Z }, 2024-06-26T07:05:22.8576458Z { 2024-06-26T07:05:22.8576874Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8577422Z "size": 702, 2024-06-26T07:05:22.8577957Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:05:22.8578582Z }, 2024-06-26T07:05:22.8578816Z { 2024-06-26T07:05:22.8579241Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8579784Z "size": 143, 2024-06-26T07:05:22.8580307Z "digest": "sha256:83911d7c7ecf11b5932bb7449975c52e7f737a5b867691935e1ccb85ceb25b52" 2024-06-26T07:05:22.8580900Z }, 2024-06-26T07:05:22.8581152Z { 2024-06-26T07:05:22.8581575Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8582115Z "size": 135, 2024-06-26T07:05:22.8582719Z "digest": "sha256:b3c5d66c679f58c71febdd502889bd8cd3d36e5001754cdf1e30eabe21135cce" 2024-06-26T07:05:22.8583343Z }, 2024-06-26T07:05:22.8583588Z { 2024-06-26T07:05:22.8584006Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8584579Z "size": 4911495247, 2024-06-26T07:05:22.8585155Z "digest": "sha256:84b2c7a899820f6acdb22148c5e8346d8b26c33269fd01842c9573515251e1e1" 2024-06-26T07:05:22.8585758Z }, 2024-06-26T07:05:22.8586004Z { 2024-06-26T07:05:22.8586427Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8586971Z "size": 190, 2024-06-26T07:05:22.8587517Z "digest": "sha256:c1068d03a811d8dead5da9e82fdc8869205fc4ee9cb92d4f7a4acfbb5ba83950" 2024-06-26T07:05:22.8588127Z }, 2024-06-26T07:05:22.8588370Z { 2024-06-26T07:05:22.8588793Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8589327Z "size": 565, 2024-06-26T07:05:22.8589858Z "digest": "sha256:671b4a301d81074fca17d36851f81801552c4ecb2f90ff3412aa9ab6a1a91371" 2024-06-26T07:05:22.8590522Z }, 2024-06-26T07:05:22.8590768Z { 2024-06-26T07:05:22.8591183Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8591728Z "size": 43163095, 2024-06-26T07:05:22.8592369Z "digest": "sha256:67760cf5ec46ab58d3ca98637fd9b26cb47a67446bfd0db176fe5e023606aa0d" 2024-06-26T07:05:22.8592986Z }, 2024-06-26T07:05:22.8593223Z { 2024-06-26T07:05:22.8593721Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8594267Z "size": 106, 2024-06-26T07:05:22.8594796Z "digest": "sha256:ddc9740bcea2b301e99b50525059aea8d9c7121a1c8eab566561a8968438a8c2" 2024-06-26T07:05:22.8595411Z }, 2024-06-26T07:05:22.8595656Z { 2024-06-26T07:05:22.8596079Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8596616Z "size": 1213, 2024-06-26T07:05:22.8597177Z "digest": "sha256:fb2e977685609fcc01de170ba7f50a573e73e6520e47253717e92a28952c5ee8" 2024-06-26T07:05:22.8597786Z }, 2024-06-26T07:05:22.8598029Z { 2024-06-26T07:05:22.8598443Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8598987Z "size": 702, 2024-06-26T07:05:22.8599523Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:05:22.8600131Z }, 2024-06-26T07:05:22.8600369Z { 2024-06-26T07:05:22.8600790Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8601336Z "size": 138, 2024-06-26T07:05:22.8601866Z "digest": "sha256:b599c5a8c7cb461cbd7b3c98b99960b8eb04506e05e79db1cc0cfadb14a47e91" 2024-06-26T07:05:22.8602486Z }, 2024-06-26T07:05:22.8602732Z { 2024-06-26T07:05:22.8603158Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8603698Z "size": 120, 2024-06-26T07:05:22.8604240Z "digest": "sha256:e08e88c1366827455ee7ddbf92b6ae615ceb3cbd3fb11cd03ddd31bd3fa94a9c" 2024-06-26T07:05:22.8604863Z }, 2024-06-26T07:05:22.8605109Z { 2024-06-26T07:05:22.8605531Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8606087Z "size": 1904157495, 2024-06-26T07:05:22.8606653Z "digest": "sha256:12ba7431ff38de8cf6af3052a2a52e79f4e712b8c99f5bafde3c20e63b1f570f" 2024-06-26T07:05:22.8607273Z }, 2024-06-26T07:05:22.8607515Z { 2024-06-26T07:05:22.8607944Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8608502Z "size": 172, 2024-06-26T07:05:22.8609032Z "digest": "sha256:cf7c2cb896cba6a594028cc0fd246e127a2f4481d48905a3aa12c5f5872332d4" 2024-06-26T07:05:22.8609653Z }, 2024-06-26T07:05:22.8609897Z { 2024-06-26T07:05:22.8610320Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8610862Z "size": 908, 2024-06-26T07:05:22.8611409Z "digest": "sha256:3a778ddfac2af5b3a11f49ddf2f3f1ad1731f74ba691f434ab69966d1e2a189f" 2024-06-26T07:05:22.8612030Z }, 2024-06-26T07:05:22.8612355Z { 2024-06-26T07:05:22.8612775Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8613324Z "size": 702, 2024-06-26T07:05:22.8613867Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:05:22.8614482Z }, 2024-06-26T07:05:22.8614735Z { 2024-06-26T07:05:22.8615207Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8615756Z "size": 134, 2024-06-26T07:05:22.8616287Z "digest": "sha256:236c8f0245f4b40616a4322eaf1e1b3e178d5990db1fcaba39622ea4e79c06a1" 2024-06-26T07:05:22.8616897Z }, 2024-06-26T07:05:22.8617145Z { 2024-06-26T07:05:22.8617565Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8618104Z "size": 32, 2024-06-26T07:05:22.8618638Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:05:22.8619254Z }, 2024-06-26T07:05:22.8619491Z { 2024-06-26T07:05:22.8619921Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8620468Z "size": 156, 2024-06-26T07:05:22.8621012Z "digest": "sha256:dda1ce302b9a9df0c8f1145d2e3eaa03c96909bb45a792451f1fcb555b06738d" 2024-06-26T07:05:22.8621629Z }, 2024-06-26T07:05:22.8621872Z { 2024-06-26T07:05:22.8622303Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8622852Z "size": 1841, 2024-06-26T07:05:22.8623382Z "digest": "sha256:e797fe86caef19319e03f8dca92cf96838ab3bb3644f4447c6dfc04230118afa" 2024-06-26T07:05:22.8624063Z }, 2024-06-26T07:05:22.8624308Z { 2024-06-26T07:05:22.8624734Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8625320Z "size": 7529784, 2024-06-26T07:05:22.8625866Z "digest": "sha256:967190076fc0ab2bb2fc0d74e58edd3d662a537d702ff3f8b6e876230154a16f" 2024-06-26T07:05:22.8626475Z }, 2024-06-26T07:05:22.8626711Z { 2024-06-26T07:05:22.8627139Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8627682Z "size": 165, 2024-06-26T07:05:22.8628214Z "digest": "sha256:4fb0a83f06855534ef82b80dcf7c685303edb2bcbd446196066dc2ad2dd71281" 2024-06-26T07:05:22.8628821Z }, 2024-06-26T07:05:22.8629066Z { 2024-06-26T07:05:22.8629492Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8630103Z "size": 7944, 2024-06-26T07:05:22.8630634Z "digest": "sha256:68af2ec4e30a33f26947ece9f3a31b7d3b04c2bca5865d4b48618457832a1da2" 2024-06-26T07:05:22.8631258Z }, 2024-06-26T07:05:22.8631501Z { 2024-06-26T07:05:22.8631916Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8632558Z "size": 8070, 2024-06-26T07:05:22.8633087Z "digest": "sha256:cc91933e985a044c6df53caa15488d2285896f5e991f38ec068c253ceec53f14" 2024-06-26T07:05:22.8633687Z }, 2024-06-26T07:05:22.8633922Z { 2024-06-26T07:05:22.8634344Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8634891Z "size": 303, 2024-06-26T07:05:22.8635422Z "digest": "sha256:9f3187916e7dcad33cc572f8f0e949d2f507b887f99cfd6acefd14826b44960d" 2024-06-26T07:05:22.8636021Z }, 2024-06-26T07:05:22.8636265Z { 2024-06-26T07:05:22.8636687Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8637230Z "size": 7629071, 2024-06-26T07:05:22.8637742Z "digest": "sha256:52608399485738d10d779254946c9e1d536010dd45593c69bf1eb9f62f85dc24" 2024-06-26T07:05:22.8638327Z }, 2024-06-26T07:05:22.8638570Z { 2024-06-26T07:05:22.8638990Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8639529Z "size": 108, 2024-06-26T07:05:22.8640063Z "digest": "sha256:05a6f6dcc1c00bac6660923a22adc7d7c34416718f8c89daf9b3351d9365f694" 2024-06-26T07:05:22.8640672Z }, 2024-06-26T07:05:22.8640911Z { 2024-06-26T07:05:22.8641335Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8641875Z "size": 54145775, 2024-06-26T07:05:22.8642523Z "digest": "sha256:6bc9dc8315b09b27c976bfba9700767ba412d72fd0528b18bbe8f8e62005ec22" 2024-06-26T07:05:22.8643127Z }, 2024-06-26T07:05:22.8643368Z { 2024-06-26T07:05:22.8643791Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8644322Z "size": 474, 2024-06-26T07:05:22.8644880Z "digest": "sha256:1b0e27c35f07b3fbf0aeb548d1949bf1d60a420136da21789d94d196c891d981" 2024-06-26T07:05:22.8645508Z }, 2024-06-26T07:05:22.8645760Z { 2024-06-26T07:05:22.8646174Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8646717Z "size": 1374858747, 2024-06-26T07:05:22.8647259Z "digest": "sha256:95237e7311a1c5ce6937db7196c866632f44e8788514a5c0eb4a3f04da4017e6" 2024-06-26T07:05:22.8647850Z }, 2024-06-26T07:05:22.8648086Z { 2024-06-26T07:05:22.8648508Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8649041Z "size": 106, 2024-06-26T07:05:22.8649587Z "digest": "sha256:789eff6c51aada70bf75eb1acafed63c0d839b768638001500616ff6e53b854a" 2024-06-26T07:05:22.8650194Z }, 2024-06-26T07:05:22.8650440Z { 2024-06-26T07:05:22.8650866Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8651401Z "size": 558, 2024-06-26T07:05:22.8651941Z "digest": "sha256:de0f8a8c45c6077b3c5ba3b84d7f07de98353fd618dad9a1e2f368ffe00493c7" 2024-06-26T07:05:22.8652555Z }, 2024-06-26T07:05:22.8652802Z { 2024-06-26T07:05:22.8653294Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8653841Z "size": 46248513, 2024-06-26T07:05:22.8654381Z "digest": "sha256:2060028f74888d218bebdd10d723f66c33ee1c708935c4ffd4654d34912ac4dc" 2024-06-26T07:05:22.8654996Z }, 2024-06-26T07:05:22.8655261Z { 2024-06-26T07:05:22.8655684Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8656220Z "size": 111, 2024-06-26T07:05:22.8656751Z "digest": "sha256:e412c7f3b2ded440992955bd92512345cbece8a8894cd3592f714ebc758ad1a0" 2024-06-26T07:05:22.8657345Z }, 2024-06-26T07:05:22.8657588Z { 2024-06-26T07:05:22.8658014Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8658547Z "size": 32, 2024-06-26T07:05:22.8659084Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:05:22.8659698Z }, 2024-06-26T07:05:22.8659943Z { 2024-06-26T07:05:22.8660361Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8660913Z "size": 32, 2024-06-26T07:05:22.8661448Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:05:22.8662058Z }, 2024-06-26T07:05:22.8662295Z { 2024-06-26T07:05:22.8662714Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8663258Z "size": 32, 2024-06-26T07:05:22.8663778Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:05:22.8664390Z }, 2024-06-26T07:05:22.8664634Z { 2024-06-26T07:05:22.8665061Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:22.8665596Z "size": 32, 2024-06-26T07:05:22.8666124Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:05:22.8666732Z } 2024-06-26T07:05:22.8666976Z ] 2024-06-26T07:05:22.8667210Z } 2024-06-26T07:05:22.8667464Z + exit 0 2024-06-26T07:05:22.8780323Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:05:22.8780836Z tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:05:22.8781381Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-06-26T07:05:22.8789288Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:22.8789788Z env: 2024-06-26T07:05:22.8790159Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:22.8791314Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:22.8792622Z ##[endgroup] 2024-06-26T07:05:22.8814794Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:22.8864156Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-06-26T07:05:22.8864782Z with: 2024-06-26T07:05:22.8865842Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:22.8867165Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:22.8867684Z env: 2024-06-26T07:05:22.8867958Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:22.8868303Z ##[endgroup] 2024-06-26T07:05:22.8883550Z ##[group]Run set -x 2024-06-26T07:05:22.8883891Z set -x 2024-06-26T07:05:22.8884189Z set +e 2024-06-26T07:05:22.8884503Z  2024-06-26T07:05:22.8884802Z login() { 2024-06-26T07:05:22.8885471Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:05:22.8886197Z } 2024-06-26T07:05:22.8886474Z  2024-06-26T07:05:22.8886773Z retry () { 2024-06-26T07:05:22.8887167Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:05:22.8887618Z } 2024-06-26T07:05:22.8887896Z  2024-06-26T07:05:22.8888212Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:05:22.8888739Z  2024-06-26T07:05:22.8889014Z set -e 2024-06-26T07:05:22.8889496Z # ignore output since only exit code is used for conditional 2024-06-26T07:05:22.8890205Z # only pull docker image if it's not available locally 2024-06-26T07:05:22.8890977Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-06-26T07:05:22.8891683Z  retry docker pull "${DOCKER_IMAGE}" 2024-06-26T07:05:22.8892129Z fi 2024-06-26T07:05:22.8899662Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:22.8900156Z env: 2024-06-26T07:05:22.8900431Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:22.8901577Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:22.8902884Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:22.8903416Z ##[endgroup] 2024-06-26T07:05:22.8921439Z + set +e 2024-06-26T07:05:22.8921936Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:22.8922563Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:22.8924389Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:05:22.8925157Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:23.2986221Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:05:23.2987268Z Configure a credential helper to remove this warning. See 2024-06-26T07:05:23.2988295Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:05:23.2988838Z 2024-06-26T07:05:23.2988967Z Login Succeeded 2024-06-26T07:05:23.2995982Z + set -e 2024-06-26T07:05:23.2997316Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:23.3119297Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:23.3121429Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:23.5085822Z 91382da70d5719cd7007b6b80b71d2f48398f6b7: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-06-26T07:05:23.5087338Z 7a2c55901189: Pulling fs layer 2024-06-26T07:05:23.5087899Z 224fe954d725: Pulling fs layer 2024-06-26T07:05:23.5088684Z 75722010b82e: Pulling fs layer 2024-06-26T07:05:23.5089156Z d527cbbb87e3: Pulling fs layer 2024-06-26T07:05:23.5089538Z b57676e46aee: Pulling fs layer 2024-06-26T07:05:23.5089909Z a8c1e85b5e14: Pulling fs layer 2024-06-26T07:05:23.5090300Z a41a8d1c11c8: Pulling fs layer 2024-06-26T07:05:23.5090672Z 0c1227890755: Pulling fs layer 2024-06-26T07:05:23.5091112Z d8d1234baab3: Pulling fs layer 2024-06-26T07:05:23.5091490Z 7ed32bc8e469: Pulling fs layer 2024-06-26T07:05:23.5091864Z ec1e7978c1fe: Pulling fs layer 2024-06-26T07:05:23.5092240Z 309b872da805: Pulling fs layer 2024-06-26T07:05:23.5092608Z e82932cb002c: Pulling fs layer 2024-06-26T07:05:23.5092985Z ca9dd73e3e93: Pulling fs layer 2024-06-26T07:05:23.5093347Z 16d76a79d5d6: Pulling fs layer 2024-06-26T07:05:23.5093730Z 4bdf41c9ecbe: Pulling fs layer 2024-06-26T07:05:23.5094107Z f4863801e30c: Pulling fs layer 2024-06-26T07:05:23.5094473Z d37dd9ed8948: Pulling fs layer 2024-06-26T07:05:23.5094900Z cf2f34f225db: Pulling fs layer 2024-06-26T07:05:23.5095267Z 0c1227890755: Waiting 2024-06-26T07:05:23.5095607Z df546d7b3e34: Pulling fs layer 2024-06-26T07:05:23.5095972Z 3d914950f8ad: Pulling fs layer 2024-06-26T07:05:23.5096328Z d8d1234baab3: Waiting 2024-06-26T07:05:23.5096655Z a41a8d1c11c8: Waiting 2024-06-26T07:05:23.5097234Z 96d0e2032184: Pulling fs layer 2024-06-26T07:05:23.5097637Z 06e12ce59790: Pulling fs layer 2024-06-26T07:05:23.5098012Z ed712f63cde7: Pulling fs layer 2024-06-26T07:05:23.5098374Z a8c1e85b5e14: Waiting 2024-06-26T07:05:23.5098773Z a97a71ef8a4e: Pulling fs layer 2024-06-26T07:05:23.5099134Z 7ed32bc8e469: Waiting 2024-06-26T07:05:23.5099453Z e82932cb002c: Waiting 2024-06-26T07:05:23.5099792Z 25cd0c482ad6: Pulling fs layer 2024-06-26T07:05:23.5100158Z 9bf94bc45658: Pulling fs layer 2024-06-26T07:05:23.5100519Z ec1e7978c1fe: Waiting 2024-06-26T07:05:23.5100841Z ca9dd73e3e93: Waiting 2024-06-26T07:05:23.5101168Z de45a14c9b22: Pulling fs layer 2024-06-26T07:05:23.5101540Z e99241e77209: Pulling fs layer 2024-06-26T07:05:23.5101920Z 35e8f4e2a42f: Pulling fs layer 2024-06-26T07:05:23.5102295Z a823ffa4c541: Pulling fs layer 2024-06-26T07:05:23.5102651Z b57676e46aee: Waiting 2024-06-26T07:05:23.5102986Z 44ba272eae20: Pulling fs layer 2024-06-26T07:05:23.5103361Z da02d278e524: Pulling fs layer 2024-06-26T07:05:23.5103744Z 73a0ce681c29: Pulling fs layer 2024-06-26T07:05:23.5104180Z 49737bd5ae70: Pulling fs layer 2024-06-26T07:05:23.5104580Z f4863801e30c: Waiting 2024-06-26T07:05:23.5104980Z d37dd9ed8948: Waiting 2024-06-26T07:05:23.5105359Z 9cd4c745f8e4: Pulling fs layer 2024-06-26T07:05:23.5105721Z d527cbbb87e3: Waiting 2024-06-26T07:05:23.5106058Z b2950a036534: Pulling fs layer 2024-06-26T07:05:23.5106427Z 25e19d81d9fc: Pulling fs layer 2024-06-26T07:05:23.5106787Z 60156b40fcbe: Pulling fs layer 2024-06-26T07:05:23.5107143Z df546d7b3e34: Waiting 2024-06-26T07:05:23.5107472Z 7a868239227f: Pulling fs layer 2024-06-26T07:05:23.5107823Z 3d914950f8ad: Waiting 2024-06-26T07:05:23.5108142Z 83911d7c7ecf: Pulling fs layer 2024-06-26T07:05:23.5108519Z b3c5d66c679f: Pulling fs layer 2024-06-26T07:05:23.5108887Z 84b2c7a89982: Pulling fs layer 2024-06-26T07:05:23.5109235Z 96d0e2032184: Waiting 2024-06-26T07:05:23.5109559Z c1068d03a811: Pulling fs layer 2024-06-26T07:05:23.5110047Z 06e12ce59790: Waiting 2024-06-26T07:05:23.5110385Z 671b4a301d81: Pulling fs layer 2024-06-26T07:05:23.5110735Z ed712f63cde7: Waiting 2024-06-26T07:05:23.5111064Z 67760cf5ec46: Pulling fs layer 2024-06-26T07:05:23.5111437Z ddc9740bcea2: Pulling fs layer 2024-06-26T07:05:23.5111795Z a823ffa4c541: Waiting 2024-06-26T07:05:23.5112104Z 4bdf41c9ecbe: Waiting 2024-06-26T07:05:23.5112686Z fb2e97768560: Pulling fs layer 2024-06-26T07:05:23.5113115Z 73a0ce681c29: Waiting 2024-06-26T07:05:23.5113467Z 309b872da805: Waiting 2024-06-26T07:05:23.5113801Z b599c5a8c7cb: Pulling fs layer 2024-06-26T07:05:23.5114179Z e08e88c13668: Pulling fs layer 2024-06-26T07:05:23.5114534Z 44ba272eae20: Waiting 2024-06-26T07:05:23.5114844Z a97a71ef8a4e: Waiting 2024-06-26T07:05:23.5115174Z 12ba7431ff38: Pulling fs layer 2024-06-26T07:05:23.5115679Z cf7c2cb896cb: Pulling fs layer 2024-06-26T07:05:23.5116058Z 3a778ddfac2a: Pulling fs layer 2024-06-26T07:05:23.5116434Z 236c8f0245f4: Pulling fs layer 2024-06-26T07:05:23.5116790Z 7a868239227f: Waiting 2024-06-26T07:05:23.5117114Z 84b2c7a89982: Waiting 2024-06-26T07:05:23.5117445Z 4f4fb700ef54: Pulling fs layer 2024-06-26T07:05:23.5117806Z b2950a036534: Waiting 2024-06-26T07:05:23.5118127Z 35e8f4e2a42f: Waiting 2024-06-26T07:05:23.5118443Z 49737bd5ae70: Waiting 2024-06-26T07:05:23.5118753Z 25e19d81d9fc: Waiting 2024-06-26T07:05:23.5119071Z 25cd0c482ad6: Waiting 2024-06-26T07:05:23.5119397Z 9cd4c745f8e4: Waiting 2024-06-26T07:05:23.5119709Z 9bf94bc45658: Waiting 2024-06-26T07:05:23.5120033Z 3a778ddfac2a: Waiting 2024-06-26T07:05:23.5120356Z cf7c2cb896cb: Waiting 2024-06-26T07:05:23.5120675Z cf2f34f225db: Waiting 2024-06-26T07:05:23.5120991Z b3c5d66c679f: Waiting 2024-06-26T07:05:23.5121306Z e99241e77209: Waiting 2024-06-26T07:05:23.5121623Z 236c8f0245f4: Waiting 2024-06-26T07:05:23.5121940Z 671b4a301d81: Waiting 2024-06-26T07:05:23.5122259Z 67760cf5ec46: Waiting 2024-06-26T07:05:23.5122595Z dda1ce302b9a: Pulling fs layer 2024-06-26T07:05:23.5122964Z 4f4fb700ef54: Waiting 2024-06-26T07:05:23.5123298Z e797fe86caef: Pulling fs layer 2024-06-26T07:05:23.5123758Z ddc9740bcea2: Waiting 2024-06-26T07:05:23.5124089Z 967190076fc0: Pulling fs layer 2024-06-26T07:05:23.5124447Z b599c5a8c7cb: Waiting 2024-06-26T07:05:23.5124802Z 4fb0a83f0685: Pulling fs layer 2024-06-26T07:05:23.5125178Z de45a14c9b22: Waiting 2024-06-26T07:05:23.5125513Z 68af2ec4e30a: Pulling fs layer 2024-06-26T07:05:23.5125896Z e08e88c13668: Waiting 2024-06-26T07:05:23.5126219Z cc91933e985a: Pulling fs layer 2024-06-26T07:05:23.5126577Z 12ba7431ff38: Waiting 2024-06-26T07:05:23.5126906Z 9f3187916e7d: Pulling fs layer 2024-06-26T07:05:23.5127268Z 526083994857: Pulling fs layer 2024-06-26T07:05:23.5127622Z dda1ce302b9a: Waiting 2024-06-26T07:05:23.5127960Z 05a6f6dcc1c0: Pulling fs layer 2024-06-26T07:05:23.5128329Z 83911d7c7ecf: Waiting 2024-06-26T07:05:23.5128654Z 6bc9dc8315b0: Pulling fs layer 2024-06-26T07:05:23.5129027Z 1b0e27c35f07: Pulling fs layer 2024-06-26T07:05:23.5129389Z e797fe86caef: Waiting 2024-06-26T07:05:23.5129718Z 95237e7311a1: Pulling fs layer 2024-06-26T07:05:23.5130075Z 967190076fc0: Waiting 2024-06-26T07:05:23.5130406Z 789eff6c51aa: Pulling fs layer 2024-06-26T07:05:23.5130838Z 68af2ec4e30a: Waiting 2024-06-26T07:05:23.5131164Z de0f8a8c45c6: Pulling fs layer 2024-06-26T07:05:23.5131524Z 6bc9dc8315b0: Waiting 2024-06-26T07:05:23.5131855Z 2060028f7488: Pulling fs layer 2024-06-26T07:05:23.5132229Z e412c7f3b2de: Pulling fs layer 2024-06-26T07:05:23.5132578Z cc91933e985a: Waiting 2024-06-26T07:05:23.5132890Z da02d278e524: Waiting 2024-06-26T07:05:23.5133199Z 526083994857: Waiting 2024-06-26T07:05:23.5133498Z 9f3187916e7d: Waiting 2024-06-26T07:05:23.5133859Z 95237e7311a1: Waiting 2024-06-26T07:05:23.5134181Z de0f8a8c45c6: Waiting 2024-06-26T07:05:23.5134623Z 789eff6c51aa: Waiting 2024-06-26T07:05:23.5134946Z 05a6f6dcc1c0: Waiting 2024-06-26T07:05:23.5135407Z 1b0e27c35f07: Waiting 2024-06-26T07:05:23.5135768Z c1068d03a811: Waiting 2024-06-26T07:05:23.6494604Z 224fe954d725: Verifying Checksum 2024-06-26T07:05:23.6495128Z 224fe954d725: Download complete 2024-06-26T07:05:23.7251848Z d527cbbb87e3: Verifying Checksum 2024-06-26T07:05:23.7252306Z d527cbbb87e3: Download complete 2024-06-26T07:05:23.8085733Z b57676e46aee: Verifying Checksum 2024-06-26T07:05:23.8086236Z b57676e46aee: Download complete 2024-06-26T07:05:23.8600044Z 7a2c55901189: Verifying Checksum 2024-06-26T07:05:23.8600592Z 7a2c55901189: Download complete 2024-06-26T07:05:23.9638326Z a41a8d1c11c8: Download complete 2024-06-26T07:05:24.0524112Z 0c1227890755: Verifying Checksum 2024-06-26T07:05:24.0524653Z 0c1227890755: Download complete 2024-06-26T07:05:24.1257667Z d8d1234baab3: Download complete 2024-06-26T07:05:24.1285277Z 75722010b82e: Verifying Checksum 2024-06-26T07:05:24.1285831Z 75722010b82e: Download complete 2024-06-26T07:05:24.2145962Z ec1e7978c1fe: Download complete 2024-06-26T07:05:24.2944275Z 309b872da805: Download complete 2024-06-26T07:05:24.8925313Z 7a2c55901189: Pull complete 2024-06-26T07:05:25.2179771Z 224fe954d725: Pull complete 2024-06-26T07:05:26.1529946Z 75722010b82e: Pull complete 2024-06-26T07:05:26.2036714Z d527cbbb87e3: Pull complete 2024-06-26T07:05:26.2599800Z b57676e46aee: Pull complete 2024-06-26T07:05:26.8365801Z e82932cb002c: Verifying Checksum 2024-06-26T07:05:26.8366284Z e82932cb002c: Download complete 2024-06-26T07:05:26.9261617Z ca9dd73e3e93: Verifying Checksum 2024-06-26T07:05:26.9262193Z ca9dd73e3e93: Download complete 2024-06-26T07:05:26.9977201Z 16d76a79d5d6: Verifying Checksum 2024-06-26T07:05:26.9977658Z 16d76a79d5d6: Download complete 2024-06-26T07:05:27.0861143Z 4bdf41c9ecbe: Download complete 2024-06-26T07:05:28.0531667Z f4863801e30c: Verifying Checksum 2024-06-26T07:05:28.0532286Z f4863801e30c: Download complete 2024-06-26T07:05:28.1490487Z d37dd9ed8948: Download complete 2024-06-26T07:05:28.2305476Z cf2f34f225db: Verifying Checksum 2024-06-26T07:05:28.2305958Z cf2f34f225db: Download complete 2024-06-26T07:05:28.2982524Z df546d7b3e34: Verifying Checksum 2024-06-26T07:05:28.2983160Z df546d7b3e34: Download complete 2024-06-26T07:05:37.1537846Z a8c1e85b5e14: Verifying Checksum 2024-06-26T07:05:37.1538701Z a8c1e85b5e14: Download complete 2024-06-26T07:05:37.2669476Z 96d0e2032184: Verifying Checksum 2024-06-26T07:05:37.2670333Z 96d0e2032184: Download complete 2024-06-26T07:05:37.3264913Z 06e12ce59790: Download complete 2024-06-26T07:05:37.3948788Z ed712f63cde7: Download complete 2024-06-26T07:05:37.4933001Z a97a71ef8a4e: Verifying Checksum 2024-06-26T07:05:37.4933625Z a97a71ef8a4e: Download complete 2024-06-26T07:05:37.5681813Z 25cd0c482ad6: Verifying Checksum 2024-06-26T07:05:37.5682473Z 25cd0c482ad6: Download complete 2024-06-26T07:05:38.8480863Z 9bf94bc45658: Verifying Checksum 2024-06-26T07:05:38.8481398Z 9bf94bc45658: Download complete 2024-06-26T07:05:38.9241921Z de45a14c9b22: Verifying Checksum 2024-06-26T07:05:38.9242539Z de45a14c9b22: Download complete 2024-06-26T07:05:39.0138014Z e99241e77209: Verifying Checksum 2024-06-26T07:05:39.0138600Z e99241e77209: Download complete 2024-06-26T07:05:39.1089729Z 35e8f4e2a42f: Verifying Checksum 2024-06-26T07:05:39.1090392Z 35e8f4e2a42f: Download complete 2024-06-26T07:05:39.1936476Z a823ffa4c541: Download complete 2024-06-26T07:05:39.2783812Z 44ba272eae20: Verifying Checksum 2024-06-26T07:05:39.2784378Z 44ba272eae20: Download complete 2024-06-26T07:05:43.7991953Z da02d278e524: Verifying Checksum 2024-06-26T07:05:43.7992823Z da02d278e524: Download complete 2024-06-26T07:05:43.8761675Z 73a0ce681c29: Verifying Checksum 2024-06-26T07:05:43.8762200Z 73a0ce681c29: Download complete 2024-06-26T07:05:43.9501467Z 49737bd5ae70: Verifying Checksum 2024-06-26T07:05:43.9502100Z 49737bd5ae70: Download complete 2024-06-26T07:05:44.3654763Z 9cd4c745f8e4: Verifying Checksum 2024-06-26T07:05:44.3655271Z 9cd4c745f8e4: Download complete 2024-06-26T07:05:44.4408164Z b2950a036534: Download complete 2024-06-26T07:05:44.5250476Z 25e19d81d9fc: Download complete 2024-06-26T07:05:44.7912547Z 60156b40fcbe: Verifying Checksum 2024-06-26T07:05:44.7913200Z 60156b40fcbe: Download complete 2024-06-26T07:05:44.8681281Z 7a868239227f: Verifying Checksum 2024-06-26T07:05:44.8681889Z 7a868239227f: Download complete 2024-06-26T07:05:44.9348021Z 83911d7c7ecf: Download complete 2024-06-26T07:05:45.0137030Z b3c5d66c679f: Verifying Checksum 2024-06-26T07:05:45.0137568Z b3c5d66c679f: Download complete 2024-06-26T07:05:49.4718493Z 7ed32bc8e469: Verifying Checksum 2024-06-26T07:05:49.4718973Z 7ed32bc8e469: Download complete 2024-06-26T07:05:49.5511487Z c1068d03a811: Verifying Checksum 2024-06-26T07:05:49.5511971Z c1068d03a811: Download complete 2024-06-26T07:05:49.6161374Z 671b4a301d81: Verifying Checksum 2024-06-26T07:05:49.6162057Z 671b4a301d81: Download complete 2024-06-26T07:05:49.6668650Z a8c1e85b5e14: Pull complete 2024-06-26T07:05:49.9173127Z a41a8d1c11c8: Pull complete 2024-06-26T07:05:50.0982728Z 67760cf5ec46: Verifying Checksum 2024-06-26T07:05:50.0983552Z 67760cf5ec46: Download complete 2024-06-26T07:05:50.1496885Z 0c1227890755: Pull complete 2024-06-26T07:05:50.1697690Z ddc9740bcea2: Download complete 2024-06-26T07:05:50.2410705Z fb2e97768560: Download complete 2024-06-26T07:05:50.3147468Z b599c5a8c7cb: Verifying Checksum 2024-06-26T07:05:50.3147991Z b599c5a8c7cb: Download complete 2024-06-26T07:05:50.3521483Z d8d1234baab3: Pull complete 2024-06-26T07:05:50.3870481Z e08e88c13668: Verifying Checksum 2024-06-26T07:05:50.3871018Z e08e88c13668: Download complete 2024-06-26T07:05:56.2048073Z 3d914950f8ad: Verifying Checksum 2024-06-26T07:05:56.2048543Z 3d914950f8ad: Download complete 2024-06-26T07:05:56.2893785Z cf7c2cb896cb: Verifying Checksum 2024-06-26T07:05:56.2894326Z cf7c2cb896cb: Download complete 2024-06-26T07:05:56.4264968Z 3a778ddfac2a: Verifying Checksum 2024-06-26T07:05:56.4265597Z 3a778ddfac2a: Download complete 2024-06-26T07:05:56.5072911Z 236c8f0245f4: Download complete 2024-06-26T07:05:56.5159153Z 4f4fb700ef54: Download complete 2024-06-26T07:05:56.5962742Z dda1ce302b9a: Download complete 2024-06-26T07:05:56.6739548Z e797fe86caef: Download complete 2024-06-26T07:05:56.8115635Z 967190076fc0: Verifying Checksum 2024-06-26T07:05:56.8116109Z 967190076fc0: Download complete 2024-06-26T07:05:56.9288181Z 4fb0a83f0685: Verifying Checksum 2024-06-26T07:05:56.9288741Z 4fb0a83f0685: Download complete 2024-06-26T07:05:57.0004236Z 68af2ec4e30a: Download complete 2024-06-26T07:05:57.1054007Z cc91933e985a: Download complete 2024-06-26T07:05:57.2282439Z 9f3187916e7d: Download complete 2024-06-26T07:05:57.3631997Z 526083994857: Verifying Checksum 2024-06-26T07:05:57.3632882Z 526083994857: Download complete 2024-06-26T07:05:57.4406363Z 05a6f6dcc1c0: Download complete 2024-06-26T07:05:58.0432111Z 6bc9dc8315b0: Verifying Checksum 2024-06-26T07:05:58.0432907Z 6bc9dc8315b0: Download complete 2024-06-26T07:05:58.1144925Z 1b0e27c35f07: Verifying Checksum 2024-06-26T07:06:21.2173158Z 1b0e27c35f07: Download complete 2024-06-26T07:06:21.2173622Z 12ba7431ff38: Verifying Checksum 2024-06-26T07:06:21.2174033Z 12ba7431ff38: Download complete 2024-06-26T07:06:21.2947162Z 789eff6c51aa: Verifying Checksum 2024-06-26T07:06:21.2948013Z 789eff6c51aa: Download complete 2024-06-26T07:06:21.3851704Z de0f8a8c45c6: Verifying Checksum 2024-06-26T07:06:21.3852429Z de0f8a8c45c6: Download complete 2024-06-26T07:06:22.7658118Z 2060028f7488: Verifying Checksum 2024-06-26T07:06:22.7658586Z 2060028f7488: Download complete 2024-06-26T07:06:22.8404967Z e412c7f3b2de: Download complete 2024-06-26T07:06:26.3104813Z 95237e7311a1: Verifying Checksum 2024-06-26T07:06:26.3105277Z 95237e7311a1: Download complete 2024-06-26T07:06:49.0855785Z 84b2c7a89982: Verifying Checksum 2024-06-26T07:06:49.0856665Z 84b2c7a89982: Download complete 2024-06-26T07:07:06.2751038Z 7ed32bc8e469: Pull complete 2024-06-26T07:07:06.5108977Z ec1e7978c1fe: Pull complete 2024-06-26T07:07:06.7285491Z 309b872da805: Pull complete 2024-06-26T07:07:14.0889746Z e82932cb002c: Pull complete 2024-06-26T07:07:14.3136352Z ca9dd73e3e93: Pull complete 2024-06-26T07:07:14.5420163Z 16d76a79d5d6: Pull complete 2024-06-26T07:07:14.7783786Z 4bdf41c9ecbe: Pull complete 2024-06-26T07:07:17.1656277Z f4863801e30c: Pull complete 2024-06-26T07:07:17.3852640Z d37dd9ed8948: Pull complete 2024-06-26T07:07:17.6098999Z cf2f34f225db: Pull complete 2024-06-26T07:07:17.8462332Z df546d7b3e34: Pull complete 2024-06-26T07:08:07.2659811Z 3d914950f8ad: Pull complete 2024-06-26T07:08:07.4292448Z 96d0e2032184: Pull complete 2024-06-26T07:08:07.5727696Z 06e12ce59790: Pull complete 2024-06-26T07:08:07.6618079Z ed712f63cde7: Pull complete 2024-06-26T07:08:07.7760419Z a97a71ef8a4e: Pull complete 2024-06-26T07:08:07.8938942Z 25cd0c482ad6: Pull complete 2024-06-26T07:08:11.3328432Z 9bf94bc45658: Pull complete 2024-06-26T07:08:11.4347166Z de45a14c9b22: Pull complete 2024-06-26T07:08:11.6289078Z e99241e77209: Pull complete 2024-06-26T07:08:11.8638277Z 35e8f4e2a42f: Pull complete 2024-06-26T07:08:12.0842056Z a823ffa4c541: Pull complete 2024-06-26T07:08:12.3028528Z 44ba272eae20: Pull complete 2024-06-26T07:08:20.4565788Z da02d278e524: Pull complete 2024-06-26T07:08:20.6542531Z 73a0ce681c29: Pull complete 2024-06-26T07:08:20.8137109Z 49737bd5ae70: Pull complete 2024-06-26T07:08:21.7517247Z 9cd4c745f8e4: Pull complete 2024-06-26T07:08:21.9315998Z b2950a036534: Pull complete 2024-06-26T07:08:22.1560854Z 25e19d81d9fc: Pull complete 2024-06-26T07:08:22.6651952Z 60156b40fcbe: Pull complete 2024-06-26T07:08:22.7911739Z 7a868239227f: Pull complete 2024-06-26T07:08:23.2047722Z 83911d7c7ecf: Pull complete 2024-06-26T07:08:23.4481499Z b3c5d66c679f: Pull complete 2024-06-26T07:09:02.5059947Z 84b2c7a89982: Pull complete 2024-06-26T07:09:02.7313212Z c1068d03a811: Pull complete 2024-06-26T07:09:02.9563862Z 671b4a301d81: Pull complete 2024-06-26T07:09:04.8576228Z 67760cf5ec46: Pull complete 2024-06-26T07:09:04.9077602Z ddc9740bcea2: Pull complete 2024-06-26T07:09:04.9490413Z fb2e97768560: Pull complete 2024-06-26T07:09:05.0439921Z b599c5a8c7cb: Pull complete 2024-06-26T07:09:05.1057144Z e08e88c13668: Pull complete 2024-06-26T07:09:39.3175692Z 12ba7431ff38: Pull complete 2024-06-26T07:09:39.5350912Z cf7c2cb896cb: Pull complete 2024-06-26T07:09:39.7592652Z 3a778ddfac2a: Pull complete 2024-06-26T07:09:40.2218181Z 236c8f0245f4: Pull complete 2024-06-26T07:09:40.4461737Z 4f4fb700ef54: Pull complete 2024-06-26T07:09:40.6568016Z dda1ce302b9a: Pull complete 2024-06-26T07:09:40.8774076Z e797fe86caef: Pull complete 2024-06-26T07:09:41.2531567Z 967190076fc0: Pull complete 2024-06-26T07:09:41.4778835Z 4fb0a83f0685: Pull complete 2024-06-26T07:09:41.7035535Z 68af2ec4e30a: Pull complete 2024-06-26T07:09:41.9389876Z cc91933e985a: Pull complete 2024-06-26T07:09:42.1739810Z 9f3187916e7d: Pull complete 2024-06-26T07:09:43.2657790Z 526083994857: Pull complete 2024-06-26T07:09:43.4780698Z 05a6f6dcc1c0: Pull complete 2024-06-26T07:09:45.5802133Z 6bc9dc8315b0: Pull complete 2024-06-26T07:09:45.8044023Z 1b0e27c35f07: Pull complete 2024-06-26T07:09:59.4871062Z 95237e7311a1: Pull complete 2024-06-26T07:09:59.5511237Z 789eff6c51aa: Pull complete 2024-06-26T07:09:59.5869264Z de0f8a8c45c6: Pull complete 2024-06-26T07:10:00.2855055Z 2060028f7488: Pull complete 2024-06-26T07:10:00.3335568Z e412c7f3b2de: Pull complete 2024-06-26T07:10:00.5842640Z Digest: sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T07:10:00.5872887Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:10:00.5899510Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:10:00.5938810Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:10:00.5939747Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:10:00.5947854Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:00.5948356Z env: 2024-06-26T07:10:00.5948625Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:00.5948970Z ##[endgroup] 2024-06-26T07:10:00.6105103Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-06-26T07:10:00.6105674Z with: 2024-06-26T07:10:00.6105960Z driver-version: 550.54.15 2024-06-26T07:10:00.6106301Z env: 2024-06-26T07:10:00.6106580Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:00.6106913Z ##[endgroup] 2024-06-26T07:10:00.6146480Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:10:00.6147040Z with: 2024-06-26T07:10:00.6147320Z timeout_minutes: 10 2024-06-26T07:10:00.6147648Z max_attempts: 3 2024-06-26T07:10:00.6178717Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-06-26T07:10:00.6210577Z retry_wait_seconds: 10 2024-06-26T07:10:00.6210952Z polling_interval_seconds: 1 2024-06-26T07:10:00.6211324Z warning_on_retry: true 2024-06-26T07:10:00.6211678Z continue_on_error: false 2024-06-26T07:10:00.6212024Z env: 2024-06-26T07:10:00.6212296Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:00.6212652Z DRIVER_VERSION: 550.54.15 2024-06-26T07:10:00.6213114Z ##[endgroup] 2024-06-26T07:10:00.6740571Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-06-26T07:10:00.6741340Z + pre_install_nvidia_driver_amzn2 2024-06-26T07:10:00.6741851Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-06-26T07:10:00.9781575Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:10:01.0167394Z No Match for argument: nvidia-driver-latest-dkms 2024-06-26T07:10:01.0405233Z No Packages marked for removal 2024-06-26T07:10:01.0522101Z + install_nvidia_driver_common 2024-06-26T07:10:01.0524598Z + echo 'Before installing NVIDIA driver' 2024-06-26T07:10:01.0525192Z + lspci 2024-06-26T07:10:01.0525590Z Before installing NVIDIA driver 2024-06-26T07:10:01.0606223Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:10:01.0607007Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:10:01.0607956Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:10:01.0609037Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:10:01.0609821Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:10:01.0610579Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:10:01.0611300Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:10:01.0612081Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:10:01.0612674Z + lsmod 2024-06-26T07:10:01.0619849Z Module Size Used by 2024-06-26T07:10:01.0620436Z ib_core 266240 0 2024-06-26T07:10:01.0620967Z xt_nat 16384 0 2024-06-26T07:10:01.0621534Z nvidia_modeset 1339392 0 2024-06-26T07:10:01.0622106Z nvidia_uvm 4599808 0 2024-06-26T07:10:01.0622754Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:10:01.0623678Z drm 421888 1 nvidia 2024-06-26T07:10:01.0624270Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:10:01.0624927Z veth 16384 0 2024-06-26T07:10:01.0625481Z backlight 16384 1 nvidia_modeset 2024-06-26T07:10:01.0625934Z xt_conntrack 16384 1 2024-06-26T07:10:01.0626317Z ipt_MASQUERADE 16384 1 2024-06-26T07:10:01.0626750Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:10:01.0627209Z nf_conntrack_netlink 49152 0 2024-06-26T07:10:01.0627657Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:10:01.0628174Z xfrm_user 45056 1 2024-06-26T07:10:01.0628569Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:10:01.0629018Z iptable_nat 16384 1 2024-06-26T07:10:01.0629540Z nf_conntrack_ipv4 16384 3 2024-06-26T07:10:01.0630298Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:10:01.0630941Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:10:01.0631670Z nf_nat 32768 3 nf_nat_masquerade_ipv4,nf_nat_ipv4,xt_nat 2024-06-26T07:10:01.0633272Z nf_conntrack 155648 8 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink 2024-06-26T07:10:01.0634221Z xt_addrtype 16384 2 2024-06-26T07:10:01.0634652Z iptable_filter 16384 1 2024-06-26T07:10:01.0635088Z br_netfilter 24576 0 2024-06-26T07:10:01.0635493Z bridge 172032 1 br_netfilter 2024-06-26T07:10:01.0635935Z stp 16384 1 bridge 2024-06-26T07:10:01.0636391Z llc 16384 2 bridge,stp 2024-06-26T07:10:01.0636867Z overlay 86016 0 2024-06-26T07:10:01.0637233Z sunrpc 393216 1 2024-06-26T07:10:01.0637601Z dm_mirror 28672 0 2024-06-26T07:10:01.0638000Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:10:01.0638489Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:10:01.0639001Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:10:01.0639622Z dax 69632 1 dm_mod 2024-06-26T07:10:01.0640025Z crc32_pclmul 16384 0 2024-06-26T07:10:01.0640396Z ghash_clmulni_intel 16384 0 2024-06-26T07:10:01.0640779Z pcbc 16384 0 2024-06-26T07:10:01.0641148Z aesni_intel 188416 0 2024-06-26T07:10:01.0641540Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:10:01.0641983Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:10:01.0642437Z glue_helper 16384 1 aesni_intel 2024-06-26T07:10:01.0642991Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:10:01.0643624Z mousedev 24576 0 2024-06-26T07:10:01.0643986Z evdev 20480 3 2024-06-26T07:10:01.0644361Z psmouse 32768 0 2024-06-26T07:10:01.0644827Z button 16384 0 2024-06-26T07:10:01.0645193Z ena 139264 0 2024-06-26T07:10:01.0645563Z ptp 20480 1 ena 2024-06-26T07:10:01.0645971Z pps_core 20480 1 ptp 2024-06-26T07:10:01.0646367Z crc32c_intel 24576 0 2024-06-26T07:10:01.0646734Z autofs4 49152 2 2024-06-26T07:10:01.0647095Z + modinfo nvidia 2024-06-26T07:10:01.0647756Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:10:01.0648455Z alias: char-major-195-* 2024-06-26T07:10:01.0648831Z version: 550.54.15 2024-06-26T07:10:01.0649180Z supported: external 2024-06-26T07:10:01.0649521Z license: NVIDIA 2024-06-26T07:10:01.0649905Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:10:01.0650384Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:10:01.0650846Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:10:01.0651314Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:10:01.0651821Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:10:01.0652412Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:10:01.0652914Z depends: i2c-core,drm 2024-06-26T07:10:01.0653277Z retpoline: Y 2024-06-26T07:10:01.0653576Z name: nvidia 2024-06-26T07:10:01.0654129Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:10:01.0654835Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:10:01.0655489Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:10:01.0656072Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:10:01.0656517Z parm: NVreg_RmLogonRC:int 2024-06-26T07:10:01.0656960Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:10:01.0657419Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:10:01.0657851Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:10:01.0658294Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:10:01.0658824Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:10:01.0659397Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:10:01.0659877Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:10:01.0660310Z parm: NVreg_EnableMSI:int 2024-06-26T07:10:01.0660730Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:10:01.0661190Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:10:01.0661716Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:10:01.0662304Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:10:01.0662869Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:10:01.0663482Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:10:01.0664072Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:10:01.0664689Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:10:01.0665282Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:10:01.0665770Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:10:01.0666307Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:10:01.0666934Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:10:01.0667434Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:10:01.0667890Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:10:01.0668374Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:10:01.0668844Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:10:01.0669298Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:10:01.0669796Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:10:01.0670442Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:10:01.0670919Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:10:01.0671409Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:10:01.0671898Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:10:01.0672584Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:10:01.0673101Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:10:01.0673585Z parm: NVreg_RmMsg:charp 2024-06-26T07:10:01.0673996Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:10:01.0674461Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:10:01.0674933Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:10:01.0675399Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:10:01.0675865Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:10:01.0676335Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:10:01.0676786Z parm: rm_firmware_active:charp 2024-06-26T07:10:01.0677200Z + HAS_NVIDIA_DRIVER=0 2024-06-26T07:10:01.0677571Z ++ command -v nvidia-smi 2024-06-26T07:10:01.0677964Z + '[' -x /usr/bin/nvidia-smi ']' 2024-06-26T07:10:01.0678330Z + set +e 2024-06-26T07:10:01.0678816Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-06-26T07:10:03.0561864Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-06-26T07:10:03.0562596Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:10:03.0563611Z + '[' 0 -ne 0 ']' 2024-06-26T07:10:03.0564194Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-06-26T07:10:03.0564747Z + HAS_NVIDIA_DRIVER=1 2024-06-26T07:10:03.0565510Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-06-26T07:10:03.0566216Z + set -e 2024-06-26T07:10:03.0566507Z + '[' 1 -eq 0 ']' 2024-06-26T07:10:03.0567093Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-06-26T07:10:03.0567778Z + post_install_nvidia_driver_common 2024-06-26T07:10:03.0568186Z + sudo modprobe nvidia 2024-06-26T07:10:03.0660161Z + echo 'After installing NVIDIA driver' 2024-06-26T07:10:03.0660733Z + lspci 2024-06-26T07:10:03.0661159Z After installing NVIDIA driver 2024-06-26T07:10:03.0737981Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:10:03.0738828Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:10:03.0739777Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:10:03.0740584Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:10:03.0741339Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:10:03.0742097Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:10:03.0742812Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:10:03.0743602Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:10:03.0744191Z + lsmod 2024-06-26T07:10:03.0752113Z Module Size Used by 2024-06-26T07:10:03.0752876Z ib_core 266240 0 2024-06-26T07:10:03.0753376Z xt_nat 16384 0 2024-06-26T07:10:03.0753873Z nvidia_modeset 1339392 0 2024-06-26T07:10:03.0754332Z nvidia_uvm 4599808 0 2024-06-26T07:10:03.0754771Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:10:03.0755333Z drm 421888 1 nvidia 2024-06-26T07:10:03.0755853Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:10:03.0756466Z veth 16384 0 2024-06-26T07:10:03.0756876Z backlight 16384 1 nvidia_modeset 2024-06-26T07:10:03.0757313Z xt_conntrack 16384 1 2024-06-26T07:10:03.0757685Z ipt_MASQUERADE 16384 1 2024-06-26T07:10:03.0758120Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:10:03.0758580Z nf_conntrack_netlink 49152 0 2024-06-26T07:10:03.0759019Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:10:03.0759481Z xfrm_user 45056 1 2024-06-26T07:10:03.0759868Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:10:03.0760284Z iptable_nat 16384 1 2024-06-26T07:10:03.0760659Z nf_conntrack_ipv4 16384 3 2024-06-26T07:10:03.0761087Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:10:03.0761552Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:10:03.0762119Z nf_nat 32768 3 nf_nat_masquerade_ipv4,nf_nat_ipv4,xt_nat 2024-06-26T07:10:03.0763405Z nf_conntrack 155648 8 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink 2024-06-26T07:10:03.0764579Z xt_addrtype 16384 2 2024-06-26T07:10:03.0765091Z iptable_filter 16384 1 2024-06-26T07:10:03.0765597Z br_netfilter 24576 0 2024-06-26T07:10:03.0766135Z bridge 172032 1 br_netfilter 2024-06-26T07:10:03.0766676Z stp 16384 1 bridge 2024-06-26T07:10:03.0767092Z llc 16384 2 bridge,stp 2024-06-26T07:10:03.0767510Z overlay 86016 0 2024-06-26T07:10:03.0767875Z sunrpc 393216 1 2024-06-26T07:10:03.0768242Z dm_mirror 28672 0 2024-06-26T07:10:03.0768628Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:10:03.0769116Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:10:03.0769868Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:10:03.0770349Z dax 69632 1 dm_mod 2024-06-26T07:10:03.0770776Z crc32_pclmul 16384 0 2024-06-26T07:10:03.0771154Z ghash_clmulni_intel 16384 0 2024-06-26T07:10:03.0771529Z pcbc 16384 0 2024-06-26T07:10:03.0771886Z aesni_intel 188416 0 2024-06-26T07:10:03.0772316Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:10:03.0772752Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:10:03.0773194Z glue_helper 16384 1 aesni_intel 2024-06-26T07:10:03.0773750Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:10:03.0774275Z mousedev 24576 0 2024-06-26T07:10:03.0774654Z evdev 20480 3 2024-06-26T07:10:03.0775054Z psmouse 32768 0 2024-06-26T07:10:03.0775414Z button 16384 0 2024-06-26T07:10:03.0775765Z ena 139264 0 2024-06-26T07:10:03.0776135Z ptp 20480 1 ena 2024-06-26T07:10:03.0776526Z pps_core 20480 1 ptp 2024-06-26T07:10:03.0776919Z crc32c_intel 24576 0 2024-06-26T07:10:03.0777278Z autofs4 49152 2 2024-06-26T07:10:03.0777637Z + modinfo nvidia 2024-06-26T07:10:03.0778283Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:10:03.0778968Z alias: char-major-195-* 2024-06-26T07:10:03.0779340Z version: 550.54.15 2024-06-26T07:10:03.0779682Z supported: external 2024-06-26T07:10:03.0780024Z license: NVIDIA 2024-06-26T07:10:03.0780391Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:10:03.0780870Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:10:03.0781325Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:10:03.0781781Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:10:03.0782264Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:10:03.0782759Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:10:03.0783233Z depends: i2c-core,drm 2024-06-26T07:10:03.0783661Z retpoline: Y 2024-06-26T07:10:03.0783958Z name: nvidia 2024-06-26T07:10:03.0784501Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:10:03.0785148Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:10:03.0785784Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:10:03.0786359Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:10:03.0786801Z parm: NVreg_RmLogonRC:int 2024-06-26T07:10:03.0787230Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:10:03.0787671Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:10:03.0788099Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:10:03.0788534Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:10:03.0789059Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:10:03.0789611Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:10:03.0790225Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:10:03.0790650Z parm: NVreg_EnableMSI:int 2024-06-26T07:10:03.0791072Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:10:03.0791518Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:10:03.0792038Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:10:03.0792879Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:10:03.0793445Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:10:03.0794044Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:10:03.0794640Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:10:03.0795251Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:10:03.0795853Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:10:03.0796330Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:10:03.0796954Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:10:03.0797503Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:10:03.0797992Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:10:03.0798439Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:10:03.0798922Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:10:03.0799392Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:10:03.0799833Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:10:03.0800324Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:10:03.0800845Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:10:03.0801319Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:10:03.0801793Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:10:03.0802290Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:10:03.0802771Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:10:03.0803275Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:10:03.0803746Z parm: NVreg_RmMsg:charp 2024-06-26T07:10:03.0804161Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:10:03.0804653Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:10:03.0805141Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:10:03.0805584Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:10:03.0806055Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:10:03.0806524Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:10:03.0806972Z parm: rm_firmware_active:charp 2024-06-26T07:10:03.0807361Z + set +e 2024-06-26T07:10:03.0807663Z + nvidia-smi 2024-06-26T07:10:04.6332003Z Wed Jun 26 07:10:04 2024 2024-06-26T07:10:04.6332873Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:10:04.6333894Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-06-26T07:10:04.6334692Z |-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:10:04.6335724Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-06-26T07:10:04.6336611Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-06-26T07:10:04.6337313Z | | | MIG M. | 2024-06-26T07:10:04.6337876Z |=========================================+========================+======================| 2024-06-26T07:10:04.6471248Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-06-26T07:10:04.6473500Z | 0% 34C P0 59W / 300W | 0MiB / 23028MiB | 0% Default | 2024-06-26T07:10:04.6474817Z | | | N/A | 2024-06-26T07:10:04.6475810Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:10:04.6476414Z 2024-06-26T07:10:04.6490200Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:10:04.6490863Z | Processes: | 2024-06-26T07:10:04.6491590Z | GPU GI CI PID Type Process name GPU Memory | 2024-06-26T07:10:04.6492298Z | ID ID Usage | 2024-06-26T07:10:04.6492863Z |=========================================================================================| 2024-06-26T07:10:04.6493511Z | No running processes found | 2024-06-26T07:10:04.6494265Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:10:05.2695449Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-06-26T07:10:06.8274555Z NVIDIA A10G 2024-06-26T07:10:07.2626504Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:10:07.2627271Z + '[' 0 -eq 0 ']' 2024-06-26T07:10:07.2627795Z + echo 'INFO: Ignoring allowed status 0' 2024-06-26T07:10:07.2628229Z + set -e 2024-06-26T07:10:07.2628524Z INFO: Ignoring allowed status 0 2024-06-26T07:10:07.2629558Z == Installing nvidia container toolkit for amzn2 == 2024-06-26T07:10:07.2632106Z + sudo yum install -y yum-utils 2024-06-26T07:10:07.5638267Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:10:09.0734043Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-06-26T07:10:09.0734872Z Nothing to do 2024-06-26T07:10:09.2222637Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]] 2024-06-26T07:10:09.2223646Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:10:09.2224716Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:10:09.5526778Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:10:09.5775272Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:10:09.5776442Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:10:09.5777434Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:10:09.5893466Z + sudo yum install -y nvidia-docker2 2024-06-26T07:10:09.8916090Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:10:11.3588320Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-06-26T07:10:11.3588985Z Nothing to do 2024-06-26T07:10:11.5121673Z + sudo systemctl restart docker 2024-06-26T07:10:50.7194822Z Command completed after 1 attempt(s). 2024-06-26T07:10:50.7250915Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:10:50.7251697Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:10:50.7252514Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-06-26T07:10:50.7253183Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:10:50.7261090Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:50.7261593Z env: 2024-06-26T07:10:50.7261874Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:50.7262333Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:50.7262821Z ##[endgroup] 2024-06-26T07:10:50.9467720Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:10:50.9621043Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-06-26T07:10:50.9722198Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-06-26T07:10:51.0646786Z Prepare all required actions 2024-06-26T07:10:51.0647287Z Getting action download info 2024-06-26T07:10:51.1768092Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-06-26T07:10:51.3731818Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-06-26T07:10:51.4866129Z ##[group]Run ./.github/actions/download-build-artifacts 2024-06-26T07:10:51.4866608Z with: 2024-06-26T07:10:51.4866943Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:10:51.4867400Z s3-bucket: gha-artifacts 2024-06-26T07:10:51.4867750Z env: 2024-06-26T07:10:51.4868024Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:51.4868478Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:51.4869008Z ##[endgroup] 2024-06-26T07:10:51.4900668Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:10:51.4901113Z with: 2024-06-26T07:10:51.4901470Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:10:51.4901926Z s3-bucket: gha-artifacts 2024-06-26T07:10:51.4902274Z region: us-east-1 2024-06-26T07:10:51.4902583Z env: 2024-06-26T07:10:51.4902846Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:51.4903290Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:51.4903769Z ##[endgroup] 2024-06-26T07:10:51.9115903Z (node:10411) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:10:51.9117514Z 2024-06-26T07:10:51.9118032Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:10:51.9119054Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:10:51.9120047Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:10:52.0019244Z Found 1 objects with prefix pytorch/pytorch/9673646806/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-06-26T07:10:52.0020367Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:10:59.4899921Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:10:59.4906224Z Artifact download has finished successfully 2024-06-26T07:10:59.5060818Z ##[group]Run unzip -o artifacts.zip 2024-06-26T07:10:59.5061294Z unzip -o artifacts.zip 2024-06-26T07:10:59.5069140Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:59.5069640Z env: 2024-06-26T07:10:59.5070054Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:59.5070509Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:59.5070993Z ##[endgroup] 2024-06-26T07:10:59.5098529Z Archive: artifacts.zip 2024-06-26T07:10:59.5100097Z creating: dist/ 2024-06-26T07:11:01.4824017Z inflating: dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:11:01.4824726Z creating: build/custom_test_artifacts/ 2024-06-26T07:11:01.4825322Z creating: build/custom_test_artifacts/custom-op-build/ 2024-06-26T07:11:01.4826059Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-06-26T07:11:01.4826900Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:11:01.4831010Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:11:01.4831958Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-06-26T07:11:01.4833195Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:11:01.4834182Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:11:01.4835148Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:11:01.4836240Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:11:01.4837487Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:11:01.4838483Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:11:01.4839487Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:11:01.4840630Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:11:01.4841779Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:11:01.4842905Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:11:01.4844020Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:11:01.4845169Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:11:01.4846324Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:11:01.4847351Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:11:01.4848350Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:11:01.4883593Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:11:01.4921259Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:11:01.4922735Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:11:01.4965463Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:11:01.4966871Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:11:01.4968275Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:11:01.4969738Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:11:01.4971141Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:11:01.4972574Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:11:01.4973963Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:11:01.4975329Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:11:01.4976678Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:11:01.4978228Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:11:01.4979699Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:11:01.4980910Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:11:01.4982179Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:11:01.4983361Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:11:01.4984564Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:11:01.5034246Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:11:01.5092600Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:11:01.5093809Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:11:01.5094793Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:11:01.5095654Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:11:01.5096554Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-06-26T07:11:01.5097471Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-06-26T07:11:01.5098494Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-06-26T07:11:01.5099642Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-06-26T07:11:01.5100741Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-06-26T07:11:01.5101780Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-06-26T07:11:01.5102899Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-06-26T07:11:01.5103984Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-06-26T07:11:01.5105060Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-06-26T07:11:01.5106133Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-06-26T07:11:01.5107197Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-06-26T07:11:01.5119408Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-06-26T07:11:01.5246907Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-06-26T07:11:01.5247921Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-06-26T07:11:01.5249017Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-06-26T07:11:01.5250229Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-06-26T07:11:01.5251397Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-06-26T07:11:01.5252526Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-06-26T07:11:01.5253653Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-06-26T07:11:01.5254794Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-06-26T07:11:01.5255932Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-06-26T07:11:01.5257178Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-06-26T07:11:01.5258296Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-06-26T07:11:01.5272674Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-06-26T07:11:01.5352574Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-06-26T07:11:01.5353746Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:11:01.5354799Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:11:01.5355859Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-06-26T07:11:01.5356760Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-06-26T07:11:01.5357645Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-06-26T07:11:01.5358519Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-06-26T07:11:01.5359352Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-06-26T07:11:01.5360120Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-06-26T07:11:01.5360920Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-06-26T07:11:01.5467952Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-06-26T07:11:01.5528667Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-06-26T07:11:01.5529402Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-06-26T07:11:01.5530104Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-06-26T07:11:01.5530908Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:11:01.5536260Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:11:01.5537177Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-06-26T07:11:01.5538111Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:11:01.5539091Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:11:01.5540044Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:11:01.5541112Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:11:01.5542223Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:11:01.5543244Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:11:01.5544216Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:11:01.5545356Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:11:01.5546475Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:11:01.5547579Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:11:01.5548689Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:11:01.5549935Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:11:01.5551068Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:11:01.5552132Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:11:01.5553433Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:11:01.5590554Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:11:01.5628616Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:11:01.5630153Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:11:01.5673207Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:11:01.5674590Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:11:01.5676119Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:11:01.5677577Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:11:01.5678966Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:11:01.5680323Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:11:01.5681689Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:11:01.5683108Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:11:01.5684455Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:11:01.5685720Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:11:01.5686948Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:11:01.5688136Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:11:01.5689349Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:11:01.5690528Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:11:01.5691734Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:11:01.5742593Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:11:01.5801245Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:11:01.5802461Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:11:01.5803429Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:11:01.5804266Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:11:01.5805143Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-06-26T07:11:01.5806061Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-06-26T07:11:01.5807122Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-06-26T07:11:01.5808286Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-06-26T07:11:01.5809414Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-06-26T07:11:01.5810482Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-06-26T07:11:01.5811711Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-06-26T07:11:01.5812854Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-06-26T07:11:01.5813948Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-06-26T07:11:01.5815038Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-06-26T07:11:01.5816109Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-06-26T07:11:01.5829479Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-06-26T07:11:01.5891426Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-06-26T07:11:01.5892642Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:11:01.5893708Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:11:01.5894656Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-06-26T07:11:01.5917403Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-06-26T07:11:01.5918559Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-06-26T07:11:01.5919481Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-06-26T07:11:01.5920343Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-06-26T07:11:01.5921131Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-06-26T07:11:01.5921997Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-06-26T07:11:01.5946184Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-06-26T07:11:01.5946949Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-06-26T07:11:01.5947711Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-06-26T07:11:01.5948595Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:11:01.5954062Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:11:01.5955365Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-06-26T07:11:01.5956459Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:11:01.5957527Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:11:01.5958693Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:11:01.5959955Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:11:01.5961110Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:11:01.5962156Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:11:01.5963212Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:11:01.5964421Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:11:01.5965635Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:11:01.5966824Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:11:01.5968022Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:11:01.5969455Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:11:01.5970666Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:11:01.5971800Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:11:01.5972847Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:11:01.6008254Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:11:01.6046234Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:11:01.6047881Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:11:01.6090426Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:11:01.6092555Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:11:01.6095470Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:11:01.6098492Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:11:01.6101401Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:11:01.6102900Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:11:01.6104354Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:11:01.6105786Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:11:01.6107180Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:11:01.6108515Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:11:01.6109778Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:11:01.6111138Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:11:01.6112549Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:11:01.6113802Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:11:01.6115070Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:11:01.6159667Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:11:01.6217636Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:11:01.6219188Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:11:01.6220227Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:11:01.6221160Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:11:01.6222121Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-06-26T07:11:01.6223254Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-06-26T07:11:01.6224389Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-06-26T07:11:01.6225631Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-06-26T07:11:01.6226835Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-06-26T07:11:01.6227967Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-06-26T07:11:01.6229138Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-06-26T07:11:01.6230512Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-06-26T07:11:01.6231738Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-06-26T07:11:01.6233077Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-06-26T07:11:01.6234257Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-06-26T07:11:01.6235500Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-06-26T07:11:01.6349713Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-06-26T07:11:01.6351133Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-06-26T07:11:01.6352450Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-06-26T07:11:01.6353755Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-06-26T07:11:01.6355011Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-06-26T07:11:01.6356193Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-06-26T07:11:01.6357404Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-06-26T07:11:01.6358622Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-06-26T07:11:01.6359844Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-06-26T07:11:01.6361063Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-06-26T07:11:01.6362269Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-06-26T07:11:01.6375149Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-06-26T07:11:01.6427988Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-06-26T07:11:01.6429285Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:11:01.6430508Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:11:01.6431536Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-06-26T07:11:01.6432639Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-06-26T07:11:01.6433620Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-06-26T07:11:01.6434595Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-06-26T07:11:01.6435645Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-06-26T07:11:01.6436494Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-06-26T07:11:01.6437364Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-06-26T07:11:01.6536259Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-06-26T07:11:01.6577541Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-06-26T07:11:01.6578762Z creating: build/lib/ 2024-06-26T07:11:01.6663211Z inflating: build/lib/libprotobuf-lite.a 2024-06-26T07:11:01.7101863Z inflating: build/lib/libprotobuf.a 2024-06-26T07:11:01.7111523Z inflating: build/lib/libpthreadpool.a 2024-06-26T07:11:01.7119908Z inflating: build/lib/libcpuinfo.a 2024-06-26T07:11:01.7127657Z inflating: build/lib/libcpuinfo_internals.a 2024-06-26T07:11:01.7128831Z inflating: build/lib/libclog.a 2024-06-26T07:11:01.7146508Z inflating: build/lib/libnnpack.a 2024-06-26T07:11:01.7148827Z inflating: build/lib/libnnpack_reference_layers.a 2024-06-26T07:11:01.7211717Z inflating: build/lib/libgtest.a 2024-06-26T07:11:01.7284985Z inflating: build/lib/libbenchmark.a 2024-06-26T07:11:01.7347135Z inflating: build/lib/libasmjit.a 2024-06-26T07:11:01.7354876Z inflating: build/lib/libittnotify.a 2024-06-26T07:11:01.7381785Z inflating: build/lib/libtensorpipe_uv.a 2024-06-26T07:11:01.7504599Z inflating: build/lib/libgloo.a 2024-06-26T07:11:01.7526084Z inflating: build/lib/libfmt.a 2024-06-26T07:11:01.7615208Z inflating: build/lib/libc10.so 2024-06-26T07:11:01.7616868Z inflating: build/lib/libcaffe2_nvrtc.so 2024-06-26T07:11:01.7617582Z inflating: build/lib/libfoxi_loader.a 2024-06-26T07:11:01.7618996Z inflating: build/lib/libtorch_global_deps.so 2024-06-26T07:11:01.7638443Z inflating: build/lib/libpytorch_qnnpack.a 2024-06-26T07:11:01.7656093Z inflating: build/lib/libgmock.a 2024-06-26T07:11:01.7656763Z inflating: build/lib/libgtest_main.a 2024-06-26T07:11:01.7657449Z inflating: build/lib/libbenchmark_main.a 2024-06-26T07:11:01.8145565Z inflating: build/lib/libprotoc.a 2024-06-26T07:11:01.8516367Z inflating: build/lib/libgloo_cuda.a 2024-06-26T07:11:02.8256109Z inflating: build/lib/libdnnl.a 2024-06-26T07:11:02.8806362Z inflating: build/lib/libtensorpipe.a 2024-06-26T07:11:02.8860333Z inflating: build/lib/libc10_cuda.so 2024-06-26T07:11:02.8861484Z inflating: build/lib/libgmock_main.a 2024-06-26T07:11:03.0063315Z inflating: build/lib/libfbgemm.a 2024-06-26T07:11:03.0309341Z inflating: build/lib/libtensorpipe_cuda.a 2024-06-26T07:11:03.0807615Z inflating: build/lib/libkineto.a 2024-06-26T07:11:03.0985612Z inflating: build/lib/libXNNPACK.a 2024-06-26T07:11:03.1026761Z inflating: build/lib/libonnx_proto.a 2024-06-26T07:11:03.1692236Z inflating: build/lib/libonnx.a 2024-06-26T07:11:05.5027402Z inflating: build/lib/libtorch_cpu.so 2024-06-26T07:11:05.5033592Z inflating: build/lib/libunbox_lib.a 2024-06-26T07:11:05.5035679Z inflating: build/lib/libshm.so 2024-06-26T07:11:07.4923831Z inflating: build/lib/libtorch_cuda.so 2024-06-26T07:11:07.4924583Z inflating: build/lib/libtorch.so 2024-06-26T07:11:08.2885260Z inflating: build/lib/libtorch_cuda_linalg.so 2024-06-26T07:11:08.2886915Z inflating: build/lib/libc10d_cuda_test.so 2024-06-26T07:11:08.4782483Z inflating: build/lib/libtorch_python.so 2024-06-26T07:11:08.4849535Z inflating: build/lib/libtorchbind_test.so 2024-06-26T07:11:08.4868966Z inflating: build/lib/libjitbackend_test.so 2024-06-26T07:11:08.4893678Z inflating: build/lib/libbackend_with_compiler.so 2024-06-26T07:11:08.4926970Z inflating: build/lib/libnnapi_backend.so 2024-06-26T07:11:08.4927436Z creating: build/bin/ 2024-06-26T07:11:08.4976530Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-06-26T07:11:08.5025723Z inflating: build/bin/c10_DeviceGuard_test 2024-06-26T07:11:08.5075099Z inflating: build/bin/c10_Device_test 2024-06-26T07:11:08.5131900Z inflating: build/bin/c10_DispatchKeySet_test 2024-06-26T07:11:08.5183834Z inflating: build/bin/c10_Scalar_test 2024-06-26T07:11:08.5231307Z inflating: build/bin/c10_StreamGuard_test 2024-06-26T07:11:08.5284091Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-06-26T07:11:08.5333414Z inflating: build/bin/c10_SymInt_test 2024-06-26T07:11:08.5387361Z inflating: build/bin/c10_InlineStreamGuard_test 2024-06-26T07:11:08.5441981Z inflating: build/bin/c10_SizesAndStrides_test 2024-06-26T07:11:08.5493227Z inflating: build/bin/c10_Bitset_test 2024-06-26T07:11:08.5562082Z inflating: build/bin/c10_cow_test 2024-06-26T07:11:08.5609513Z inflating: build/bin/c10_ConstexprCrc_test 2024-06-26T07:11:08.5657975Z inflating: build/bin/c10_DeadlockDetection_test 2024-06-26T07:11:08.5707415Z inflating: build/bin/c10_Half_test 2024-06-26T07:11:08.5761971Z inflating: build/bin/c10_LeftRight_test 2024-06-26T07:11:08.5815169Z inflating: build/bin/c10_Metaprogramming_test 2024-06-26T07:11:08.5863443Z inflating: build/bin/c10_Synchronized_test 2024-06-26T07:11:08.5917808Z inflating: build/bin/c10_ThreadLocal_test 2024-06-26T07:11:08.5967698Z inflating: build/bin/c10_TypeIndex_test 2024-06-26T07:11:08.6017344Z inflating: build/bin/c10_TypeList_test 2024-06-26T07:11:08.6064592Z inflating: build/bin/c10_TypeTraits_test 2024-06-26T07:11:08.6114932Z inflating: build/bin/c10_accumulate_test 2024-06-26T07:11:08.6168536Z inflating: build/bin/c10_bfloat16_test 2024-06-26T07:11:08.6217565Z inflating: build/bin/c10_bit_cast_test 2024-06-26T07:11:08.6272837Z inflating: build/bin/c10_complex_math_test 2024-06-26T07:11:08.6326278Z inflating: build/bin/c10_complex_test 2024-06-26T07:11:08.6377698Z inflating: build/bin/c10_exception_test 2024-06-26T07:11:08.6426623Z inflating: build/bin/c10_flags_test 2024-06-26T07:11:08.6475309Z inflating: build/bin/c10_generic_math_test 2024-06-26T07:11:08.6636099Z inflating: build/bin/c10_intrusive_ptr_test 2024-06-26T07:11:08.6685369Z inflating: build/bin/c10_irange_test 2024-06-26T07:11:08.6737256Z inflating: build/bin/c10_lazy_test 2024-06-26T07:11:08.6792708Z inflating: build/bin/c10_logging_test 2024-06-26T07:11:08.6865145Z inflating: build/bin/c10_optional_test 2024-06-26T07:11:08.6925595Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-06-26T07:11:08.6978115Z inflating: build/bin/c10_registry_test 2024-06-26T07:11:08.7123077Z inflating: build/bin/c10_small_vector_test 2024-06-26T07:11:08.7173312Z inflating: build/bin/c10_ssize_test 2024-06-26T07:11:08.7224308Z inflating: build/bin/c10_string_util_test 2024-06-26T07:11:08.7281601Z inflating: build/bin/c10_string_view_test 2024-06-26T07:11:08.7330375Z inflating: build/bin/c10_tempfile_test 2024-06-26T07:11:08.7377421Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-06-26T07:11:08.7431741Z inflating: build/bin/c10_typeid_test 2024-06-26T07:11:08.7864547Z inflating: build/bin/protoc-3.13.0.0 2024-06-26T07:11:08.8303289Z inflating: build/bin/protoc 2024-06-26T07:11:08.8354005Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-06-26T07:11:08.8405305Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-06-26T07:11:08.8456599Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-06-26T07:11:08.8506712Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-06-26T07:11:08.8558093Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-06-26T07:11:08.8609069Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-06-26T07:11:08.8656719Z inflating: build/bin/c10_cuda_CUDATest 2024-06-26T07:11:08.8707770Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-06-26T07:11:08.9026413Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-06-26T07:11:08.9358198Z inflating: build/bin/vec_test_all_types_AVX512 2024-06-26T07:11:08.9704544Z inflating: build/bin/vec_test_all_types_AVX2 2024-06-26T07:11:08.9755482Z inflating: build/bin/FileStoreTest 2024-06-26T07:11:08.9809530Z inflating: build/bin/TCPStoreTest 2024-06-26T07:11:08.9862793Z inflating: build/bin/HashStoreTest 2024-06-26T07:11:08.9875769Z inflating: build/bin/ProcessGroupMPITest 2024-06-26T07:11:08.9927751Z inflating: build/bin/test_edge_op_registration 2024-06-26T07:11:08.9931700Z inflating: build/bin/torch_shm_manager 2024-06-26T07:11:08.9934531Z inflating: build/bin/example_allreduce 2024-06-26T07:11:08.9987763Z inflating: build/bin/test_dist_autograd 2024-06-26T07:11:09.0055232Z inflating: build/bin/test_cpp_rpc 2024-06-26T07:11:09.0057332Z inflating: build/bin/parallel_benchmark 2024-06-26T07:11:09.0122601Z inflating: build/bin/test_mobile_nnc 2024-06-26T07:11:09.0131237Z inflating: build/bin/aot_model_compiler_test 2024-06-26T07:11:09.0464370Z inflating: build/bin/test_lazy 2024-06-26T07:11:09.1611368Z inflating: build/bin/test_api 2024-06-26T07:11:09.1682545Z inflating: build/bin/Dict_test 2024-06-26T07:11:09.1733024Z inflating: build/bin/Dimname_test 2024-06-26T07:11:09.1795453Z inflating: build/bin/MaybeOwned_test 2024-06-26T07:11:09.1851105Z inflating: build/bin/NamedTensor_test 2024-06-26T07:11:09.1908359Z inflating: build/bin/apply_utils_test 2024-06-26T07:11:09.1967070Z inflating: build/bin/atest 2024-06-26T07:11:09.2028123Z inflating: build/bin/basic 2024-06-26T07:11:09.2081394Z inflating: build/bin/broadcast_test 2024-06-26T07:11:09.2130558Z inflating: build/bin/cpu_allocator_test 2024-06-26T07:11:09.2187011Z inflating: build/bin/cpu_generator_test 2024-06-26T07:11:09.2239848Z inflating: build/bin/cpu_profiling_allocator_test 2024-06-26T07:11:09.2330264Z inflating: build/bin/cpu_rng_test 2024-06-26T07:11:09.2378744Z inflating: build/bin/dispatch_key_set_test 2024-06-26T07:11:09.2427551Z inflating: build/bin/dlconvertor_test 2024-06-26T07:11:09.2483980Z inflating: build/bin/extension_backend_test 2024-06-26T07:11:09.2537773Z inflating: build/bin/half_test 2024-06-26T07:11:09.2630915Z inflating: build/bin/ivalue_test 2024-06-26T07:11:09.2678873Z inflating: build/bin/lazy_tensor_test 2024-06-26T07:11:09.2731596Z inflating: build/bin/math_kernel_test 2024-06-26T07:11:09.2784109Z inflating: build/bin/memory_format_test 2024-06-26T07:11:09.2835595Z inflating: build/bin/memory_overlapping_test 2024-06-26T07:11:09.2886884Z inflating: build/bin/mobile_memory_cleanup 2024-06-26T07:11:09.2941571Z inflating: build/bin/native_test 2024-06-26T07:11:09.2991603Z inflating: build/bin/operator_name_test 2024-06-26T07:11:09.3041617Z inflating: build/bin/operators_test 2024-06-26T07:11:09.3091743Z inflating: build/bin/packedtensoraccessor_test 2024-06-26T07:11:09.3157098Z inflating: build/bin/pow_test 2024-06-26T07:11:09.3212499Z inflating: build/bin/quantized_test 2024-06-26T07:11:09.3261302Z inflating: build/bin/reduce_ops_test 2024-06-26T07:11:09.3311339Z inflating: build/bin/reportMemoryUsage_test 2024-06-26T07:11:09.3365838Z inflating: build/bin/scalar_tensor_test 2024-06-26T07:11:09.3422430Z inflating: build/bin/scalar_test 2024-06-26T07:11:09.3472651Z inflating: build/bin/StorageUtils_test 2024-06-26T07:11:09.3524150Z inflating: build/bin/stride_properties_test 2024-06-26T07:11:09.3600259Z inflating: build/bin/tensor_iterator_test 2024-06-26T07:11:09.3652647Z inflating: build/bin/test_parallel 2024-06-26T07:11:09.3655964Z inflating: build/bin/thread_init_test 2024-06-26T07:11:09.3709478Z inflating: build/bin/type_ptr_test 2024-06-26T07:11:09.3767584Z inflating: build/bin/type_test 2024-06-26T07:11:09.3818275Z inflating: build/bin/undefined_tensor_test 2024-06-26T07:11:09.3819426Z inflating: build/bin/verify_api_visibility 2024-06-26T07:11:09.3886812Z inflating: build/bin/legacy_vmap_test 2024-06-26T07:11:09.3938047Z inflating: build/bin/weakref_test 2024-06-26T07:11:09.3987929Z inflating: build/bin/wrapdim_test 2024-06-26T07:11:09.4038243Z inflating: build/bin/xla_tensor_test 2024-06-26T07:11:09.4096757Z inflating: build/bin/IListRef_test 2024-06-26T07:11:09.4199605Z inflating: build/bin/List_test 2024-06-26T07:11:09.4263987Z inflating: build/bin/KernelFunction_test 2024-06-26T07:11:09.4381764Z inflating: build/bin/kernel_function_legacy_test 2024-06-26T07:11:09.4475297Z inflating: build/bin/kernel_function_test 2024-06-26T07:11:09.4599439Z inflating: build/bin/kernel_lambda_legacy_test 2024-06-26T07:11:09.4699792Z inflating: build/bin/kernel_lambda_test 2024-06-26T07:11:09.4759493Z inflating: build/bin/kernel_stackbased_test 2024-06-26T07:11:09.4852664Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-06-26T07:11:09.4902440Z inflating: build/bin/CppSignature_test 2024-06-26T07:11:09.4956229Z inflating: build/bin/backend_fallback_test 2024-06-26T07:11:09.5003467Z inflating: build/bin/op_allowlist_test 2024-06-26T07:11:09.5296040Z inflating: build/bin/op_registration_test 2024-06-26T07:11:09.5357397Z inflating: build/bin/inline_container_test 2024-06-26T07:11:09.5408594Z inflating: build/bin/cuda_apply_test 2024-06-26T07:11:09.5457988Z inflating: build/bin/cuda_allocator_test 2024-06-26T07:11:09.5515317Z inflating: build/bin/cuda_atomic_ops_test 2024-06-26T07:11:09.5568076Z inflating: build/bin/cuda_caching_host_allocator_test 2024-06-26T07:11:09.5635875Z inflating: build/bin/cuda_complex_math_test 2024-06-26T07:11:09.5695147Z inflating: build/bin/cuda_complex_test 2024-06-26T07:11:09.5743410Z inflating: build/bin/cuda_device_test 2024-06-26T07:11:09.5799126Z inflating: build/bin/cuda_cub_test 2024-06-26T07:11:09.5847791Z inflating: build/bin/cuda_dlconvertor_test 2024-06-26T07:11:09.5910432Z inflating: build/bin/cuda_distributions_test 2024-06-26T07:11:09.5965514Z inflating: build/bin/cuda_generator_test 2024-06-26T07:11:09.6013777Z inflating: build/bin/cuda_half_test 2024-06-26T07:11:09.6063266Z inflating: build/bin/cuda_integer_divider_test 2024-06-26T07:11:09.6110892Z inflating: build/bin/cuda_optional_test 2024-06-26T07:11:09.6161044Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-06-26T07:11:09.6212038Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-06-26T07:11:09.6260736Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-06-26T07:11:09.6319512Z inflating: build/bin/cuda_stream_test 2024-06-26T07:11:09.6367272Z inflating: build/bin/cuda_cudnn_test 2024-06-26T07:11:09.6417717Z inflating: build/bin/cuda_vectorized_test 2024-06-26T07:11:09.6432433Z inflating: build/bin/tutorial_tensorexpr 2024-06-26T07:11:09.6496256Z inflating: build/bin/ProcessGroupGlooTest 2024-06-26T07:11:09.6551769Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-06-26T07:11:09.6613167Z inflating: build/bin/ProcessGroupNCCLTest 2024-06-26T07:11:09.6674259Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-06-26T07:11:09.7479796Z inflating: build/bin/test_tensorexpr 2024-06-26T07:11:09.8039103Z inflating: build/bin/test_jit 2024-06-26T07:11:09.8039562Z creating: .additional_ci_files/ 2024-06-26T07:11:09.8092781Z inflating: .additional_ci_files/test-times.json 2024-06-26T07:11:09.8303768Z inflating: .additional_ci_files/test-class-times.json 2024-06-26T07:11:09.8333207Z ##[group]Run rm artifacts.zip 2024-06-26T07:11:09.8333605Z rm artifacts.zip 2024-06-26T07:11:09.8341259Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:09.8341763Z env: 2024-06-26T07:11:09.8342056Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:09.8342522Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:09.8343143Z ##[endgroup] 2024-06-26T07:11:09.9047522Z ##[group]Run df -H 2024-06-26T07:11:09.9047846Z df -H 2024-06-26T07:11:09.9056354Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:09.9056975Z env: 2024-06-26T07:11:09.9057321Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:09.9057878Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:09.9058479Z ##[endgroup] 2024-06-26T07:11:09.9090497Z Filesystem Size Used Avail Use% Mounted on 2024-06-26T07:11:09.9091289Z devtmpfs 34G 0 34G 0% /dev 2024-06-26T07:11:09.9092343Z tmpfs 34G 2.8M 34G 1% /dev/shm 2024-06-26T07:11:09.9093073Z tmpfs 34G 455k 34G 1% /run 2024-06-26T07:11:09.9093821Z tmpfs 34G 0 34G 0% /sys/fs/cgroup 2024-06-26T07:11:09.9094593Z /dev/nvme0n1p1 162G 45G 117G 28% / 2024-06-26T07:11:09.9152113Z Prepare all required actions 2024-06-26T07:11:09.9152890Z Getting action download info 2024-06-26T07:11:10.0171624Z ##[group]Run ./.github/actions/download-td-artifacts 2024-06-26T07:11:10.0172101Z with: 2024-06-26T07:11:10.0172353Z env: 2024-06-26T07:11:10.0172625Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:10.0173078Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:10.0173555Z ##[endgroup] 2024-06-26T07:11:10.0204050Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:11:10.0204489Z with: 2024-06-26T07:11:10.0204752Z name: td_results 2024-06-26T07:11:10.0205072Z s3-bucket: gha-artifacts 2024-06-26T07:11:10.0205425Z region: us-east-1 2024-06-26T07:11:10.0205729Z env: 2024-06-26T07:11:10.0205994Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:10.0206449Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:10.0206923Z ##[endgroup] 2024-06-26T07:11:10.4306275Z (node:10432) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:11:10.4306966Z 2024-06-26T07:11:10.4307232Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:11:10.4307973Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:11:10.4308814Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:11:10.5004156Z Found 0 objects with prefix pytorch/pytorch/9673646806/td_results/ 2024-06-26T07:11:10.5011147Z Artifact download has finished successfully 2024-06-26T07:11:10.5136936Z ##[group]Run mkdir -p .additional_ci_files 2024-06-26T07:11:10.5137420Z mkdir -p .additional_ci_files 2024-06-26T07:11:10.5137986Z mv td_results.json .additional_ci_files/td_results.json 2024-06-26T07:11:10.5145827Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:10.5146327Z env: 2024-06-26T07:11:10.5146605Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:10.5147071Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:10.5163747Z ##[endgroup] 2024-06-26T07:11:10.5195518Z mv: cannot stat 'td_results.json': No such file or directory 2024-06-26T07:11:10.5207800Z ##[error]Process completed with exit code 1. 2024-06-26T07:11:10.5239681Z ##[group]Run .github/scripts/parse_ref.py 2024-06-26T07:11:10.5240152Z .github/scripts/parse_ref.py 2024-06-26T07:11:10.5247700Z shell: /usr/bin/bash -e {0} 2024-06-26T07:11:10.5248048Z env: 2024-06-26T07:11:10.5248325Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:10.5248779Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:10.5249260Z ##[endgroup] 2024-06-26T07:11:10.5474605Z Prepare all required actions 2024-06-26T07:11:10.5511087Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-06-26T07:11:10.5511545Z with: 2024-06-26T07:11:10.5512089Z github-token: *** 2024-06-26T07:11:10.5512590Z env: 2024-06-26T07:11:10.5512861Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:10.5513311Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:10.5513789Z ##[endgroup] 2024-06-26T07:11:10.5528343Z ##[group]Run set -eux 2024-06-26T07:11:10.5528669Z set -eux 2024-06-26T07:11:10.5529373Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-06-26T07:11:10.5537133Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:10.5537629Z env: 2024-06-26T07:11:10.5537906Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:10.5538352Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:10.5538980Z GITHUB_TOKEN: *** 2024-06-26T07:11:10.5539283Z ##[endgroup] 2024-06-26T07:11:10.5558622Z + python3 .github/scripts/get_workflow_job_id.py 9673646806 i-02844b9639d793963 2024-06-26T07:11:11.1695131Z setting job-id=26688916210 2024-06-26T07:11:11.1696208Z setting job-name=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:11:11.1898064Z Prepare all required actions 2024-06-26T07:11:11.1898510Z Getting action download info 2024-06-26T07:11:11.3082355Z ##[group]Run ./.github/actions/filter-test-configs 2024-06-26T07:11:11.3082825Z with: 2024-06-26T07:11:11.3083281Z github-token: *** 2024-06-26T07:11:11.3089838Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:11:11.3097432Z job-name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:11:11.3098507Z env: 2024-06-26T07:11:11.3098787Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:11.3099250Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:11.3099737Z ##[endgroup] 2024-06-26T07:11:11.3139892Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:11:11.3140436Z with: 2024-06-26T07:11:11.3140699Z shell: bash 2024-06-26T07:11:11.3140997Z timeout_minutes: 10 2024-06-26T07:11:11.3141319Z max_attempts: 5 2024-06-26T07:11:11.3141621Z retry_wait_seconds: 30 2024-06-26T07:11:11.3142728Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:11:11.3143879Z polling_interval_seconds: 1 2024-06-26T07:11:11.3144248Z warning_on_retry: true 2024-06-26T07:11:11.3144636Z continue_on_error: false 2024-06-26T07:11:11.3144975Z env: 2024-06-26T07:11:11.3145243Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:11.3145680Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:11.3146305Z GITHUB_TOKEN: *** 2024-06-26T07:11:11.3146758Z ##[endgroup] 2024-06-26T07:11:11.3633479Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:11:11.5810464Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:11:11.5957408Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1) 2024-06-26T07:11:11.6102299Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-06-26T07:11:11.6111309Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12) 2024-06-26T07:11:11.6132068Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.6.2) 2024-06-26T07:11:11.6140441Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19) 2024-06-26T07:11:11.6340044Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7) 2024-06-26T07:11:12.3642750Z Command completed after 1 attempt(s). 2024-06-26T07:11:12.3682464Z ##[group]Run set -x 2024-06-26T07:11:12.3682795Z set -x 2024-06-26T07:11:12.3683092Z  2024-06-26T07:11:12.3683638Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:11:12.3684316Z # in runner workspace 2024-06-26T07:11:12.3684845Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-06-26T07:11:12.3692477Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:12.3692978Z env: 2024-06-26T07:11:12.3693257Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:12.3693708Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:12.3694186Z ##[endgroup] 2024-06-26T07:11:12.3715273Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-06-26T07:11:12.3929694Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:11:12.3930284Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:11:12.3930751Z echo "Job name: ${JOB_NAME}" 2024-06-26T07:11:12.3931156Z  2024-06-26T07:11:12.3931700Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:11:12.3932387Z # in runner workspace 2024-06-26T07:11:12.3932964Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-06-26T07:11:12.3933738Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-06-26T07:11:12.3934199Z  --job-name "${JOB_NAME}" \ 2024-06-26T07:11:12.3941136Z  --test-matrix "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-06-26T07:11:12.3948403Z  --selected-test-configs "" \ 2024-06-26T07:11:12.3948863Z  --pr-number "${PR_NUMBER}" \ 2024-06-26T07:11:12.3949283Z  --tag "${TAG}" \ 2024-06-26T07:11:12.3949678Z  --event-name "${EVENT_NAME}" \ 2024-06-26T07:11:12.3950224Z  --schedule "${SCHEDULE}" \ 2024-06-26T07:11:12.3950763Z  --branch "${HEAD_BRANCH}" 2024-06-26T07:11:12.3958385Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:12.3958889Z env: 2024-06-26T07:11:12.3959155Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:12.3959618Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:12.3960283Z GITHUB_TOKEN: *** 2024-06-26T07:11:12.3961098Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:11:12.3961978Z PR_NUMBER: 2024-06-26T07:11:12.3962285Z TAG: ciflow/inductor/129470 2024-06-26T07:11:12.3962649Z EVENT_NAME: push 2024-06-26T07:11:12.3962955Z SCHEDULE: 2024-06-26T07:11:12.3963230Z HEAD_BRANCH: 2024-06-26T07:11:12.3963526Z ##[endgroup] 2024-06-26T07:11:12.3982456Z Workflow: inductor-periodic 2024-06-26T07:11:12.3983454Z Job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:11:12.5952537Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-06-26T07:11:12.7396169Z ##[group]Run echo "Filtered matrix:" 2024-06-26T07:11:12.7396628Z echo "Filtered matrix:" 2024-06-26T07:11:12.7403464Z echo "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-06-26T07:11:12.7410546Z  2024-06-26T07:11:12.7410821Z echo 2024-06-26T07:11:12.7411203Z echo "Is the current job unstable? False" 2024-06-26T07:11:12.7411670Z  2024-06-26T07:11:12.7411938Z echo 2024-06-26T07:11:12.7412297Z echo "Is keep-going label set? False" 2024-06-26T07:11:12.7412862Z  2024-06-26T07:11:12.7413129Z echo 2024-06-26T07:11:12.7413446Z echo "Renabled issues? " 2024-06-26T07:11:12.7421122Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:12.7421631Z env: 2024-06-26T07:11:12.7421898Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:12.7422355Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:12.7422845Z ##[endgroup] 2024-06-26T07:11:12.7442912Z Filtered matrix: 2024-06-26T07:11:12.7451040Z {include: [{config: dynamo_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-06-26T07:11:12.7457901Z 2024-06-26T07:11:12.7458047Z Is the current job unstable? False 2024-06-26T07:11:12.7458338Z 2024-06-26T07:11:12.7458618Z Is keep-going label set? False 2024-06-26T07:11:12.7458871Z 2024-06-26T07:11:12.7458999Z Renabled issues? 2024-06-26T07:11:12.7499004Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:11:12.7499712Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:11:12.7507429Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:12.7507931Z env: 2024-06-26T07:11:12.7508215Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:12.7508671Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:12.7509168Z JOB_TIMEOUT: 240 2024-06-26T07:11:12.7509478Z ##[endgroup] 2024-06-26T07:11:12.7577393Z ##[group]Run set -x 2024-06-26T07:11:12.7577787Z set -x 2024-06-26T07:11:12.7578087Z  2024-06-26T07:11:12.7578444Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-06-26T07:11:12.7578982Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-06-26T07:11:12.7579563Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-06-26T07:11:12.7580075Z  TEST_COMMAND=.ci/onnx/test.sh 2024-06-26T07:11:12.7580494Z else 2024-06-26T07:11:12.7580831Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:11:12.7581258Z fi 2024-06-26T07:11:12.7581539Z  2024-06-26T07:11:12.7582005Z # detached container should get cleaned up by teardown_ec2_linux 2024-06-26T07:11:12.7582751Z # TODO: Stop building test binaries as part of the build phase 2024-06-26T07:11:12.7583427Z # Used for GPU_FLAG since that doesn't play nice 2024-06-26T07:11:12.7584028Z # shellcheck disable=SC2086,SC2090 2024-06-26T07:11:12.7584487Z container_name=$(docker run \ 2024-06-26T07:11:12.7584917Z  ${GPU_FLAG:-} \ 2024-06-26T07:11:12.7585333Z  -e BUILD_ENVIRONMENT \ 2024-06-26T07:11:12.7585734Z  -e PR_NUMBER \ 2024-06-26T07:11:12.7586090Z  -e GITHUB_ACTIONS \ 2024-06-26T07:11:12.7586602Z  -e GITHUB_REPOSITORY \ 2024-06-26T07:11:12.7587007Z  -e GITHUB_WORKFLOW \ 2024-06-26T07:11:12.7587393Z  -e GITHUB_JOB \ 2024-06-26T07:11:12.7587750Z  -e GITHUB_RUN_ID \ 2024-06-26T07:11:12.7588139Z  -e GITHUB_RUN_NUMBER \ 2024-06-26T07:11:12.7588543Z  -e GITHUB_RUN_ATTEMPT \ 2024-06-26T07:11:12.7588932Z  -e JOB_ID \ 2024-06-26T07:11:12.7589273Z  -e JOB_NAME \ 2024-06-26T07:11:12.7589620Z  -e BASE_SHA \ 2024-06-26T07:11:12.7590069Z  -e BRANCH \ 2024-06-26T07:11:12.7590415Z  -e SHA1 \ 2024-06-26T07:11:12.7590766Z  -e AWS_DEFAULT_REGION \ 2024-06-26T07:11:12.7591175Z  -e IN_WHEEL_TEST \ 2024-06-26T07:11:12.7591556Z  -e SHARD_NUMBER \ 2024-06-26T07:11:12.7591919Z  -e TEST_CONFIG \ 2024-06-26T07:11:12.7592439Z  -e NUM_TEST_SHARDS \ 2024-06-26T07:11:12.7592892Z  -e REENABLED_ISSUES \ 2024-06-26T07:11:12.7593319Z  -e CONTINUE_THROUGH_ERROR \ 2024-06-26T07:11:12.7593742Z  -e VERBOSE_TEST_LOGS \ 2024-06-26T07:11:12.7594146Z  -e NO_TEST_TIMEOUT \ 2024-06-26T07:11:12.7594527Z  -e NO_TD \ 2024-06-26T07:11:12.7594866Z  -e TD_DISTRIBUTED \ 2024-06-26T07:11:12.7595252Z  -e PR_LABELS \ 2024-06-26T07:11:12.7595661Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-06-26T07:11:12.7596123Z  -e SCCACHE_BUCKET \ 2024-06-26T07:11:12.7596513Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-06-26T07:11:12.7596930Z  -e XLA_CUDA \ 2024-06-26T07:11:12.7597340Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-06-26T07:11:12.7597856Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-06-26T07:11:12.7598379Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-06-26T07:11:12.7598899Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-06-26T07:11:12.7599373Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-06-26T07:11:12.7599805Z  -e DASHBOARD_TAG \ 2024-06-26T07:11:12.7600282Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-06-26T07:11:12.7600844Z  --security-opt seccomp=unconfined \ 2024-06-26T07:11:12.7601311Z  --cap-add=SYS_PTRACE \ 2024-06-26T07:11:12.7601706Z  --ipc=host \ 2024-06-26T07:11:12.7602061Z  --shm-size="${SHM_SIZE}" \ 2024-06-26T07:11:12.7602460Z  --tty \ 2024-06-26T07:11:12.7602777Z  --detach \ 2024-06-26T07:11:12.7603139Z  --name="${container_name}" \ 2024-06-26T07:11:12.7603556Z  --user jenkins \ 2024-06-26T07:11:12.7604140Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-06-26T07:11:12.7604759Z  -w /var/lib/jenkins/workspace \ 2024-06-26T07:11:12.7605218Z  "${DOCKER_IMAGE}" 2024-06-26T07:11:12.7605566Z ) 2024-06-26T07:11:12.7605977Z # Propagate download.pytorch.org IP to container 2024-06-26T07:11:12.7606931Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-06-26T07:11:12.7607928Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-06-26T07:11:12.7608857Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-06-26T07:11:12.7615989Z shell: /usr/bin/bash -e {0} 2024-06-26T07:11:12.7616346Z env: 2024-06-26T07:11:12.7616627Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:12.7617081Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:12.7617701Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:11:12.7618220Z PR_NUMBER: 2024-06-26T07:11:12.7618549Z GITHUB_REPOSITORY: pytorch/pytorch 2024-06-26T07:11:12.7618982Z GITHUB_WORKFLOW: inductor-periodic 2024-06-26T07:11:12.7619393Z GITHUB_JOB: test 2024-06-26T07:11:12.7619718Z GITHUB_RUN_ID: 9673646806 2024-06-26T07:11:12.7620154Z GITHUB_RUN_NUMBER: 45567 2024-06-26T07:11:12.7620500Z GITHUB_RUN_ATTEMPT: 1 2024-06-26T07:11:12.7620832Z JOB_ID: 26688916210 2024-06-26T07:11:12.7621672Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:11:12.7622566Z BRANCH: 2024-06-26T07:11:12.7622900Z SHA1: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:12.7623423Z BASE_SHA: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:12.7623905Z TEST_CONFIG: aot_eager_torchbench 2024-06-26T07:11:12.7624303Z SHARD_NUMBER: 1 2024-06-26T07:11:12.7624608Z NUM_TEST_SHARDS: 2 2024-06-26T07:11:12.7624933Z REENABLED_ISSUES: 2024-06-26T07:11:12.7625281Z CONTINUE_THROUGH_ERROR: False 2024-06-26T07:11:12.7625665Z VERBOSE_TEST_LOGS: False 2024-06-26T07:11:12.7626028Z NO_TEST_TIMEOUT: False 2024-06-26T07:11:12.7626365Z NO_TD: False 2024-06-26T07:11:12.7626670Z TD_DISTRIBUTED: False 2024-06-26T07:11:12.7627085Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-06-26T07:11:12.7627614Z SCCACHE_S3_KEY_PREFIX: inductor-periodic 2024-06-26T07:11:12.7628044Z SHM_SIZE: 2g 2024-06-26T07:11:12.7629149Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:11:12.7630442Z XLA_CUDA: 2024-06-26T07:11:12.7630933Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:11:12.7631557Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-06-26T07:11:12.7632002Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-06-26T07:11:12.7632609Z DASHBOARD_TAG: 2024-06-26T07:11:12.7633134Z HUGGING_FACE_HUB_TOKEN: *** 2024-06-26T07:11:12.7633500Z ##[endgroup] 2024-06-26T07:11:12.7651864Z + [[ aot_eager_torchbench == \m\u\l\t\i\g\p\u ]] 2024-06-26T07:11:12.7652605Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-06-26T07:11:12.7653132Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:11:12.7658691Z +++ nproc --ignore=2 2024-06-26T07:11:12.7671925Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9673646806 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:11:24.0243018Z + container_name=299b633493af89e9314f5bbb416079e7c4e43079b580c4c703a3fdf1a00e6a50 2024-06-26T07:11:24.0244752Z + grep download.pytorch.org /etc/hosts 2024-06-26T07:11:24.0245942Z + docker exec -i 299b633493af89e9314f5bbb416079e7c4e43079b580c4c703a3fdf1a00e6a50 sudo bash -c '/bin/cat >> /etc/hosts' 2024-06-26T07:11:24.1034418Z + echo DOCKER_CONTAINER_ID=299b633493af89e9314f5bbb416079e7c4e43079b580c4c703a3fdf1a00e6a50 2024-06-26T07:11:24.1037536Z ++ echo dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:11:24.1039343Z + docker exec -t 299b633493af89e9314f5bbb416079e7c4e43079b580c4c703a3fdf1a00e6a50 sh -c 'pip install dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-06-26T07:11:24.4881003Z Processing ./dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+gitb8c4c54) 2024-06-26T07:11:24.7999514Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.13.1) 2024-06-26T07:11:24.8004517Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (4.12.2) 2024-06-26T07:11:24.8007319Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.12.1) 2024-06-26T07:11:24.8009963Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.8.8) 2024-06-26T07:11:24.8013119Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.1.4) 2024-06-26T07:11:24.8015380Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2024.2.0) 2024-06-26T07:11:24.8042053Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.3.0) 2024-06-26T07:11:24.8102397Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.21.2) 2024-06-26T07:11:24.8536572Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.1.5) 2024-06-26T07:11:24.8694224Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.3.0) 2024-06-26T07:11:26.1744596Z Installing collected packages: torch 2024-06-26T07:11:35.0215847Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-06-26T07:11:35.0217235Z timm 0.9.7 requires torchvision, which is not installed. 2024-06-26T07:11:35.0217905Z Successfully installed torch-2.5.0a0+gitb8c4c54 2024-06-26T07:11:35.1047334Z ++ dirname .ci/pytorch/test.sh 2024-06-26T07:11:35.1052459Z + source .ci/pytorch/common.sh 2024-06-26T07:11:35.1054301Z +++ dirname .ci/pytorch/common.sh 2024-06-26T07:11:35.1060225Z ++ source .ci/pytorch/common_utils.sh 2024-06-26T07:11:35.1061906Z +++ declare -f -t trap_add 2024-06-26T07:11:35.1067648Z ++ set -ex 2024-06-26T07:11:35.1068679Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:11:35.1069373Z ++ BUILD_TEST_LIBTORCH=0 2024-06-26T07:11:35.1070256Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-06-26T07:11:35.1072073Z ++ stat -c %u /var/lib/jenkins/workspace 2024-06-26T07:11:35.1081366Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-06-26T07:11:35.1081822Z + trap_add cleanup_workspace EXIT 2024-06-26T07:11:35.1084452Z + trap_add_cmd=cleanup_workspace 2024-06-26T07:11:35.1087327Z + shift 2024-06-26T07:11:35.1087755Z + for trap_add_name in "$@" 2024-06-26T07:11:35.1088281Z +++ trap -p EXIT 2024-06-26T07:11:35.1089283Z ++ eval 'extract_trap_cmd ' 2024-06-26T07:11:35.1089788Z +++ extract_trap_cmd 2024-06-26T07:11:35.1090154Z +++ printf '%s\n' '' 2024-06-26T07:11:35.1090527Z ++ printf '%s\n' cleanup_workspace 2024-06-26T07:11:35.1091665Z + trap -- ' 2024-06-26T07:11:35.1092079Z cleanup_workspace' EXIT 2024-06-26T07:11:35.1092569Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-06-26T07:11:35.6417561Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-06-26T07:11:35.6430050Z + echo 'Environment variables:' 2024-06-26T07:11:35.6430870Z Environment variables: 2024-06-26T07:11:35.6431267Z + env 2024-06-26T07:11:35.6436358Z INSTALLED_DB=yes 2024-06-26T07:11:35.6436907Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:11:35.6437360Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:11:35.6437750Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:11:35.6438417Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:11:35.6439190Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:11:35.6439804Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:11:35.6441074Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:11:35.6441876Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:11:35.6442402Z HOSTNAME=299b633493af 2024-06-26T07:11:35.6443499Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_801a59d5-7adb-428a-b662-11ff67fdf6ef 2024-06-26T07:11:35.6444408Z GITHUB_ACTION=__self 2024-06-26T07:11:35.6444758Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:11:35.6449142Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:11:35.6453380Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:11:35.6453952Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:11:35.6454312Z GITHUB_RUN_NUMBER=45567 2024-06-26T07:11:35.6454713Z TEST_CONFIG=aot_eager_torchbench 2024-06-26T07:11:35.6455126Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:11:35.6455653Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:11:35.6456126Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:11:35.6456612Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:11:35.6457093Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:11:35.6457539Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:11:35.6458076Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:11:35.6458545Z GITHUB_REF_TYPE=tag 2024-06-26T07:11:35.6458923Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:11:35.6459310Z NCCL_VERSION=2.17.1-1 2024-06-26T07:11:35.6459937Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:35.6460458Z XLA_CUDA= 2024-06-26T07:11:35.6460980Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:11:35.6463045Z *** 2024-06-26T07:11:35.6463333Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:11:35.6463768Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:11:35.6464160Z GITHUB_ACTIONS=true 2024-06-26T07:11:35.6464516Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:35.6465069Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:11:35.6465672Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:11:35.6466135Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:35.6466694Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:11:35.6467230Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:35.6468218Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:11:35.6469042Z UCC_HOME=/usr 2024-06-26T07:11:35.6469373Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:11:35.6469846Z VERBOSE_TEST_LOGS=False 2024-06-26T07:11:35.6470289Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:11:35.6470816Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:11:35.6471310Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:11:35.6471833Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:11:35.6472219Z SHARD_NUMBER=1 2024-06-26T07:11:35.6472963Z GITHUB_REF_PROTECTED=false 2024-06-26T07:11:35.6473314Z HOME=/var/lib/jenkins 2024-06-26T07:11:35.6473679Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:11:35.6474125Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:11:35.6474590Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:11:35.6475216Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-06-26T07:11:35.6475637Z CUDA_VERSION=12.1.1 2024-06-26T07:11:35.6476057Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:11:35.6476493Z NUM_TEST_SHARDS=2 2024-06-26T07:11:35.6476796Z UCX_HOME=/usr 2024-06-26T07:11:35.6477313Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:11:35.6478393Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_801a59d5-7adb-428a-b662-11ff67fdf6ef 2024-06-26T07:11:35.6479877Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:11:35.6481307Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_801a59d5-7adb-428a-b662-11ff67fdf6ef 2024-06-26T07:11:35.6482456Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:11:35.6483121Z GITHUB_EVENT_NAME=push 2024-06-26T07:11:35.6483445Z DASHBOARD_TAG= 2024-06-26T07:11:35.6483751Z GITHUB_RUN_ID=9673646806 2024-06-26T07:11:35.6484209Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:11:35.6484742Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:11:35.6485788Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_801a59d5-7adb-428a-b662-11ff67fdf6ef 2024-06-26T07:11:35.6486711Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:11:35.6487116Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:11:35.6487493Z PR_NUMBER= 2024-06-26T07:11:35.6487765Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:11:35.6488097Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:11:35.6488545Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:11:35.6488994Z TERM=xterm 2024-06-26T07:11:35.6489317Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:11:35.6489730Z INSTALLED_VISION=yes 2024-06-26T07:11:35.6490038Z BRANCH= 2024-06-26T07:11:35.6490317Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:11:35.6490713Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:11:35.6491133Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:11:35.6491871Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:11:35.6492609Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:11:35.6493095Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:11:35.6497013Z REENABLED_ISSUES= 2024-06-26T07:11:35.6497330Z SHLVL=1 2024-06-26T07:11:35.6497582Z MAX_JOBS=14 2024-06-26T07:11:35.6497897Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:11:35.6498249Z NVARCH=x86_64 2024-06-26T07:11:35.6498541Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:11:35.6498987Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:35.6499604Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:11:35.6500100Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:11:35.6500750Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:11:35.6501365Z GITHUB_JOB=test 2024-06-26T07:11:35.6501765Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:11:35.6502327Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:11:35.6502842Z NO_TEST_TIMEOUT=False 2024-06-26T07:11:35.6503171Z TD_DISTRIBUTED=False 2024-06-26T07:11:35.6503553Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:11:35.6503999Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:11:35.6504415Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:11:35.6504775Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:11:35.6505120Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:11:35.6505459Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:11:35.6506490Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:35.6507651Z GITHUB_BASE_REF= 2024-06-26T07:11:35.6507977Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:11:35.6508333Z CI=true 2024-06-26T07:11:35.6508645Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:11:35.6509055Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:11:35.6509425Z JOB_ID=26688916210 2024-06-26T07:11:35.6509730Z INSTALLED_PROTOBUF=yes 2024-06-26T07:11:35.6510184Z GITHUB_HEAD_REF= 2024-06-26T07:11:35.6510491Z GITHUB_ACTION_REF= 2024-06-26T07:11:35.6510909Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:11:35.6511491Z GITHUB_WORKFLOW=inductor-periodic 2024-06-26T07:11:35.6511893Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:11:35.6513015Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_801a59d5-7adb-428a-b662-11ff67fdf6ef 2024-06-26T07:11:35.6513836Z NO_TD=False 2024-06-26T07:11:35.6514136Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:11:35.6514505Z _=/usr/bin/env 2024-06-26T07:11:35.6514981Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-06-26T07:11:35.6581064Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-06-26T07:11:35.6582287Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T07:11:35.6583162Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-06-26T07:11:35.6584040Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-06-26T07:11:35.6584710Z + BUILD_DIR=build 2024-06-26T07:11:35.6585115Z + BUILD_RENAMED_DIR=build_renamed 2024-06-26T07:11:35.6585547Z + BUILD_BIN_DIR=build/bin 2024-06-26T07:11:35.6585892Z + SHARD_NUMBER=1 2024-06-26T07:11:35.6586188Z + NUM_TEST_SHARDS=2 2024-06-26T07:11:35.6586512Z + export VALGRIND=ON 2024-06-26T07:11:35.6586831Z + VALGRIND=ON 2024-06-26T07:11:35.6587299Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-06-26T07:11:35.6587793Z + [[ 0 == \1 ]] 2024-06-26T07:11:35.6588090Z + [[ False == \1 ]] 2024-06-26T07:11:35.6588561Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-06-26T07:11:35.6589072Z ++ realpath build/custom_test_artifacts 2024-06-26T07:11:35.6592087Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-06-26T07:11:35.6593102Z + [[ -n '' ]] 2024-06-26T07:11:35.6593454Z + echo 'Environment variables' 2024-06-26T07:11:35.6593834Z Environment variables 2024-06-26T07:11:35.6594142Z + env 2024-06-26T07:11:35.6598258Z INSTALLED_DB=yes 2024-06-26T07:11:35.6598676Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:11:35.6600142Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:11:35.6600756Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:11:35.6601570Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:11:35.6602283Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:11:35.6603005Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:11:35.6603742Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:11:35.6604508Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:11:35.6605071Z HOSTNAME=299b633493af 2024-06-26T07:11:35.6605908Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_801a59d5-7adb-428a-b662-11ff67fdf6ef 2024-06-26T07:11:35.6606718Z GITHUB_ACTION=__self 2024-06-26T07:11:35.6607071Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:11:35.6610798Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:11:35.6615030Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:11:35.6615545Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:11:35.6615892Z GITHUB_RUN_NUMBER=45567 2024-06-26T07:11:35.6616245Z TEST_CONFIG=aot_eager_torchbench 2024-06-26T07:11:35.6616647Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:11:35.6617105Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:11:35.6617564Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:11:35.6618007Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:11:35.6618429Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:11:35.6618860Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:11:35.6619350Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:11:35.6619806Z GITHUB_REF_TYPE=tag 2024-06-26T07:11:35.6620140Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:11:35.6620504Z NCCL_VERSION=2.17.1-1 2024-06-26T07:11:35.6620886Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:35.6621349Z XLA_CUDA= 2024-06-26T07:11:35.6621898Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:11:35.6622284Z *** 2024-06-26T07:11:35.6622572Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:11:35.6623022Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:11:35.6623387Z GITHUB_ACTIONS=true 2024-06-26T07:11:35.6623718Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:35.6624196Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:11:35.6624720Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:11:35.6625194Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:35.6625688Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:11:35.6626157Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:35.6627066Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:11:35.6627832Z UCC_HOME=/usr 2024-06-26T07:11:35.6628164Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:11:35.6628553Z VERBOSE_TEST_LOGS=False 2024-06-26T07:11:35.6628894Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:11:35.6629345Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:11:35.6629830Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:11:35.6630440Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:11:35.6630817Z SHARD_NUMBER=1 2024-06-26T07:11:35.6631124Z GITHUB_REF_PROTECTED=false 2024-06-26T07:11:35.6631468Z HOME=/var/lib/jenkins 2024-06-26T07:11:35.6631830Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:11:35.6632685Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:11:35.6633204Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:11:35.6633781Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-06-26T07:11:35.6634197Z CUDA_VERSION=12.1.1 2024-06-26T07:11:35.6634609Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:11:35.6635049Z NUM_TEST_SHARDS=2 2024-06-26T07:11:35.6635349Z UCX_HOME=/usr 2024-06-26T07:11:35.6635857Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:11:35.6636933Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_801a59d5-7adb-428a-b662-11ff67fdf6ef 2024-06-26T07:11:35.6638345Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:11:35.6639732Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_801a59d5-7adb-428a-b662-11ff67fdf6ef 2024-06-26T07:11:35.6641058Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:11:35.6641715Z GITHUB_EVENT_NAME=push 2024-06-26T07:11:35.6642042Z DASHBOARD_TAG= 2024-06-26T07:11:35.6642337Z GITHUB_RUN_ID=9673646806 2024-06-26T07:11:35.6642788Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:11:35.6643432Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:11:35.6644414Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_801a59d5-7adb-428a-b662-11ff67fdf6ef 2024-06-26T07:11:35.6645321Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:11:35.6645729Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:11:35.6646111Z PR_NUMBER= 2024-06-26T07:11:35.6646387Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:11:35.6646699Z VALGRIND=ON 2024-06-26T07:11:35.6646998Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:11:35.6647437Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:11:35.6647887Z TERM=xterm 2024-06-26T07:11:35.6648221Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:11:35.6667644Z INSTALLED_VISION=yes 2024-06-26T07:11:35.6668021Z BRANCH= 2024-06-26T07:11:35.6668315Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:11:35.6668718Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:11:35.6669141Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:11:35.6669991Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:11:35.6670752Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:11:35.6671228Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:11:35.6671700Z REENABLED_ISSUES= 2024-06-26T07:11:35.6672033Z SHLVL=1 2024-06-26T07:11:35.6672531Z MAX_JOBS=14 2024-06-26T07:11:35.6672849Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:11:35.6673190Z NVARCH=x86_64 2024-06-26T07:11:35.6673469Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:11:35.6673906Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:35.6674427Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:11:35.6674916Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:11:35.6675551Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:11:35.6676121Z GITHUB_JOB=test 2024-06-26T07:11:35.6676498Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:11:35.6677041Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:11:35.6677548Z NO_TEST_TIMEOUT=False 2024-06-26T07:11:35.6677860Z TD_DISTRIBUTED=False 2024-06-26T07:11:35.6678222Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:11:35.6678647Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:11:35.6679045Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:11:35.6679395Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:11:35.6679728Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:11:35.6680058Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:11:35.6681088Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:35.6682282Z GITHUB_BASE_REF= 2024-06-26T07:11:35.6682606Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:11:35.6682963Z CI=true 2024-06-26T07:11:35.6683271Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:11:35.6683670Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:11:35.6684031Z JOB_ID=26688916210 2024-06-26T07:11:35.6684330Z INSTALLED_PROTOBUF=yes 2024-06-26T07:11:35.6684657Z GITHUB_HEAD_REF= 2024-06-26T07:11:35.6684959Z GITHUB_ACTION_REF= 2024-06-26T07:11:35.6685375Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:11:35.6685893Z GITHUB_WORKFLOW=inductor-periodic 2024-06-26T07:11:35.6686300Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:11:35.6687192Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_801a59d5-7adb-428a-b662-11ff67fdf6ef 2024-06-26T07:11:35.6688001Z NO_TD=False 2024-06-26T07:11:35.6688295Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:11:35.6688652Z _=/usr/bin/env 2024-06-26T07:11:35.6688981Z + echo 'Testing pytorch' 2024-06-26T07:11:35.6689318Z Testing pytorch 2024-06-26T07:11:35.6689635Z + export LANG=C.UTF-8 2024-06-26T07:11:35.6689966Z + LANG=C.UTF-8 2024-06-26T07:11:35.6690244Z + PR_NUMBER= 2024-06-26T07:11:35.6690581Z + [[ aot_eager_torchbench == \d\e\f\a\u\l\t ]] 2024-06-26T07:11:35.6691088Z + [[ aot_eager_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-06-26T07:11:35.6691745Z + [[ aot_eager_torchbench == \s\l\o\w ]] 2024-06-26T07:11:35.6692363Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-06-26T07:11:35.6693055Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-06-26T07:11:35.6693599Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:11:35.6694069Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:11:35.6694504Z + [[ aot_eager_torchbench == *crossref* ]] 2024-06-26T07:11:35.6695069Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:11:35.6695692Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-06-26T07:11:35.6696331Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:11:35.6696881Z + pip_install --user ninja==1.10.2 2024-06-26T07:11:35.6697405Z + pip install --progress-bar off --user ninja==1.10.2 2024-06-26T07:11:36.0239285Z Collecting ninja==1.10.2 2024-06-26T07:11:36.0408099Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-06-26T07:11:36.0542670Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-06-26T07:11:37.3680363Z Installing collected packages: ninja 2024-06-26T07:11:37.3748802Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-06-26T07:11:37.3750155Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-06-26T07:11:37.3828654Z Successfully installed ninja-1.10.2 2024-06-26T07:11:37.4722666Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:37.4724832Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:37.4726377Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:11:37.4726891Z + install_tlparse 2024-06-26T07:11:37.4727262Z + pip_install --user tlparse==0.3.7 2024-06-26T07:11:37.4727808Z + pip install --progress-bar off --user tlparse==0.3.7 2024-06-26T07:11:37.8240258Z Collecting tlparse==0.3.7 2024-06-26T07:11:37.8387002Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-06-26T07:11:37.8490319Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-06-26T07:11:39.1680372Z Installing collected packages: tlparse 2024-06-26T07:11:39.2044356Z Successfully installed tlparse-0.3.7 2024-06-26T07:11:39.2913639Z ++ python -m site --user-base 2024-06-26T07:11:39.3067309Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:39.3070874Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-06-26T07:11:39.3072140Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-06-26T07:11:39.3073126Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:11:39.3074064Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-06-26T07:11:39.3075192Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-06-26T07:11:39.3075891Z + cd test 2024-06-26T07:11:39.3076418Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-06-26T07:11:40.7923142Z + [[ aot_eager_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-06-26T07:11:40.7923784Z + [[ aot_eager_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-06-26T07:11:40.7924556Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:11:40.7925060Z + pushd test 2024-06-26T07:11:40.7925384Z ~/workspace/test ~/workspace 2024-06-26T07:11:40.7926631Z ++ python -c 'import torch; print(torch.version.cuda)' 2024-06-26T07:11:42.2801893Z + CUDA_VERSION=12.1 2024-06-26T07:11:42.2803050Z + '[' 12.1 == 12.4 ']' 2024-06-26T07:11:42.2803558Z + ISCUDA124= 2024-06-26T07:11:42.2803956Z + popd 2024-06-26T07:11:42.2804320Z ~/workspace 2024-06-26T07:11:42.2810367Z + DYNAMO_BENCHMARK_FLAGS=() 2024-06-26T07:11:42.2810896Z + [[ aot_eager_torchbench == *dynamo_eager* ]] 2024-06-26T07:11:42.2811450Z + [[ aot_eager_torchbench == *aot_eager* ]] 2024-06-26T07:11:42.2812101Z + DYNAMO_BENCHMARK_FLAGS+=(--backend aot_eager) 2024-06-26T07:11:42.2812593Z + [[ aot_eager_torchbench == *dynamic* ]] 2024-06-26T07:11:42.2813104Z + [[ aot_eager_torchbench == *cpu_inductor* ]] 2024-06-26T07:11:42.2813649Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-06-26T07:11:42.2837789Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:11:42.2838486Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-06-26T07:11:42.2840004Z + cd test 2024-06-26T07:11:42.2840488Z + python -c 'import torch; print(torch.__config__.show())' 2024-06-26T07:11:43.6353608Z PyTorch built with: 2024-06-26T07:11:43.6354382Z - GCC 9.4 2024-06-26T07:11:43.6354835Z - C++ Version: 201703 2024-06-26T07:11:43.6355717Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:11:43.6356796Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:11:43.6357459Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:11:43.6357975Z - LAPACK is enabled (usually provided by MKL) 2024-06-26T07:11:43.6358447Z - NNPACK is enabled 2024-06-26T07:11:43.6358865Z - CPU capability usage: AVX2 2024-06-26T07:11:43.6359275Z - CUDA Runtime 12.1 2024-06-26T07:11:43.6359785Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-06-26T07:11:43.6360394Z - CuDNN 90.1 (built against CUDA 12.4) 2024-06-26T07:11:43.6360836Z - Magma 2.6.1 2024-06-26T07:11:43.6368012Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-06-26T07:11:43.6374626Z 2024-06-26T07:11:43.9135689Z + cd test 2024-06-26T07:11:43.9136368Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-06-26T07:11:45.1323951Z ATen/Parallel: 2024-06-26T07:11:45.1324415Z at::get_num_threads() : 8 2024-06-26T07:11:45.1324917Z at::get_num_interop_threads() : 16 2024-06-26T07:11:45.1325494Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:11:45.1326069Z omp_get_max_threads() : 8 2024-06-26T07:11:45.1327279Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:11:45.1328103Z mkl_get_max_threads() : 8 2024-06-26T07:11:45.1328727Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:11:45.1329648Z std::thread::hardware_concurrency() : 16 2024-06-26T07:11:45.1330081Z Environment variables: 2024-06-26T07:11:45.1330434Z OMP_NUM_THREADS : [not set] 2024-06-26T07:11:45.1330811Z MKL_NUM_THREADS : [not set] 2024-06-26T07:11:45.1331190Z ATen parallel backend: OpenMP 2024-06-26T07:11:45.1331446Z 2024-06-26T07:11:45.3738383Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:11:45.3739127Z + [[ aot_eager_torchbench == *backward* ]] 2024-06-26T07:11:45.3739744Z + [[ aot_eager_torchbench == *xla* ]] 2024-06-26T07:11:45.3740287Z + [[ aot_eager_torchbench == *executorch* ]] 2024-06-26T07:11:45.3740790Z + [[ aot_eager_torchbench == \j\i\t\_\l\e\g\a\c\y ]] 2024-06-26T07:11:45.3741455Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:11:45.3742041Z + [[ aot_eager_torchbench == distributed ]] 2024-06-26T07:11:45.3742730Z + [[ aot_eager_torchbench == *inductor_distributed* ]] 2024-06-26T07:11:45.3743523Z + [[ aot_eager_torchbench == *inductor-halide* ]] 2024-06-26T07:11:45.3744363Z + [[ aot_eager_torchbench == *inductor-micro-benchmark* ]] 2024-06-26T07:11:45.3745120Z + [[ aot_eager_torchbench == *huggingface* ]] 2024-06-26T07:11:45.3745765Z + [[ aot_eager_torchbench == *timm* ]] 2024-06-26T07:11:45.3746389Z + [[ aot_eager_torchbench == *torchbench* ]] 2024-06-26T07:11:45.3747066Z + [[ aot_eager_torchbench == *cpu_inductor* ]] 2024-06-26T07:11:45.3747685Z + install_torchaudio cuda 2024-06-26T07:11:45.3748176Z + local commit 2024-06-26T07:11:45.3748623Z ++ get_pinned_commit audio 2024-06-26T07:11:45.3749172Z ++ cat .github/ci_commit_pins/audio.txt 2024-06-26T07:11:45.3754248Z + commit=b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:11:45.3754733Z + [[ cuda == \c\u\d\a ]] 2024-06-26T07:11:45.3755152Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-06-26T07:11:45.3756052Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:11:45.3757477Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:11:45.6907375Z Collecting git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:11:45.6911335Z Cloning https://github.com/pytorch/audio.git (to revision b829e936f7cc61b48149f5f957a451a38bf2a178) to /tmp/pip-req-build-i9jgxuvg 2024-06-26T07:11:45.6928810Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-i9jgxuvg 2024-06-26T07:11:46.3313027Z Running command git rev-parse -q --verify 'sha^b829e936f7cc61b48149f5f957a451a38bf2a178' 2024-06-26T07:11:46.3328273Z Running command git fetch -q https://github.com/pytorch/audio.git b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:11:46.7046686Z Running command git checkout -q b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:11:46.8151275Z Resolved https://github.com/pytorch/audio.git to commit b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:11:46.8152648Z Running command git submodule update --init --recursive -q 2024-06-26T07:11:48.9774821Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:11:48.9805636Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.4.0a0+b829e93) (2.5.0a0+gitb8c4c54) 2024-06-26T07:11:48.9867588Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (3.13.1) 2024-06-26T07:11:48.9873520Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (4.12.2) 2024-06-26T07:11:48.9876784Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (1.12.1) 2024-06-26T07:11:48.9879543Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (2.8.8) 2024-06-26T07:11:48.9882919Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (3.1.4) 2024-06-26T07:11:48.9886069Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (2024.2.0) 2024-06-26T07:11:49.0314953Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.4.0a0+b829e93) (2.1.5) 2024-06-26T07:11:49.0471159Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchaudio==2.4.0a0+b829e93) (1.3.0) 2024-06-26T07:11:49.0544123Z Building wheels for collected packages: torchaudio 2024-06-26T07:14:13.9649214Z Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done 2024-06-26T07:14:13.9678352Z [?25h Created wheel for torchaudio: filename=torchaudio-2.4.0a0+b829e93-cp310-cp310-linux_x86_64.whl size=2308816 sha256=df228cdf759f83c94c767aa9982702c9fa80a41a97830c849287dead450f30ee 2024-06-26T07:14:13.9680003Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/30/b3/9c/882f183f00af2e582825654a92a04400924519922db53ff522 2024-06-26T07:14:13.9706421Z Successfully built torchaudio 2024-06-26T07:14:15.1993611Z Installing collected packages: torchaudio 2024-06-26T07:14:15.4000475Z Successfully installed torchaudio-2.4.0a0+b829e93 2024-06-26T07:14:15.5339387Z + install_torchtext 2024-06-26T07:14:15.5339777Z + local data_commit 2024-06-26T07:14:15.5340131Z + local text_commit 2024-06-26T07:14:15.5342443Z ++ get_pinned_commit data 2024-06-26T07:14:15.5342867Z ++ cat .github/ci_commit_pins/data.txt 2024-06-26T07:14:15.5352026Z + data_commit=11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:14:15.5354083Z ++ get_pinned_commit text 2024-06-26T07:14:15.5354475Z ++ cat .github/ci_commit_pins/text.txt 2024-06-26T07:14:15.5361967Z + text_commit=b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:14:15.5363155Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:14:15.5364912Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:14:15.8490368Z Collecting git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:14:15.8494316Z Cloning https://github.com/pytorch/data.git (to revision 11bb5b847ea8b9e0d9bb82db3304daf383008d3f) to /tmp/pip-req-build-48ltj_6_ 2024-06-26T07:14:15.8512546Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/data.git /tmp/pip-req-build-48ltj_6_ 2024-06-26T07:14:16.2530017Z Running command git rev-parse -q --verify 'sha^11bb5b847ea8b9e0d9bb82db3304daf383008d3f' 2024-06-26T07:14:16.2548415Z Running command git fetch -q https://github.com/pytorch/data.git 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:14:16.4770733Z Running command git checkout -q 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:14:16.6036848Z Resolved https://github.com/pytorch/data.git to commit 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:14:16.6037982Z Running command git submodule update --init --recursive -q 2024-06-26T07:15:28.0974524Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:15:28.1026698Z [?25hRequirement already satisfied: urllib3>=1.25 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (1.26.19) 2024-06-26T07:15:28.1029518Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.32.3) 2024-06-26T07:15:28.1035365Z Requirement already satisfied: torch>2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.5.0a0+gitb8c4c54) 2024-06-26T07:15:28.1098641Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.13.1) 2024-06-26T07:15:28.1104288Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (4.12.2) 2024-06-26T07:15:28.1107269Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.12.1) 2024-06-26T07:15:28.1110620Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.8.8) 2024-06-26T07:15:28.1114273Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.1.4) 2024-06-26T07:15:28.1117306Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2024.2.0) 2024-06-26T07:15:28.1309397Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.3.2) 2024-06-26T07:15:28.1316824Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.7) 2024-06-26T07:15:28.1322694Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (2024.6.2) 2024-06-26T07:15:28.1792022Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.1.5) 2024-06-26T07:15:28.1957794Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.3.0) 2024-06-26T07:15:28.2040408Z Building wheels for collected packages: torchdata 2024-06-26T07:15:29.9625320Z Building wheel for torchdata (setup.py) ... [?25l- \ | done 2024-06-26T07:15:29.9631851Z [?25h Created wheel for torchdata: filename=torchdata-0.7.0a0+11bb5b8-py3-none-any.whl size=182956 sha256=73c6eef2aaa453fcc14a9876f2d2b3114fab2d78a56b0b3387389303f5000907 2024-06-26T07:15:29.9634282Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/ca/59/a3/c8250bfc8d3d4d639498d4beb2e0f0e70b9a508ac61fde85ce 2024-06-26T07:15:29.9660029Z Successfully built torchdata 2024-06-26T07:15:31.1827573Z Installing collected packages: torchdata 2024-06-26T07:15:31.3527361Z Successfully installed torchdata-0.7.0a0+11bb5b8 2024-06-26T07:15:33.8493226Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:15:33.8494697Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:15:34.1610972Z Collecting git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:15:34.1614727Z Cloning https://github.com/pytorch/text.git (to revision b0ebddc648d279826089db91775375221777a2db) to /tmp/pip-req-build-mgivh04y 2024-06-26T07:15:34.1631489Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-mgivh04y 2024-06-26T07:15:35.1732095Z Running command git rev-parse -q --verify 'sha^b0ebddc648d279826089db91775375221777a2db' 2024-06-26T07:15:35.1749721Z Running command git fetch -q https://github.com/pytorch/text.git b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:15:35.6302656Z Running command git checkout -q b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:15:35.7791488Z Resolved https://github.com/pytorch/text.git to commit b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:15:35.7792942Z Running command git submodule update --init --recursive -q 2024-06-26T07:15:42.7293243Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:15:42.7342744Z [?25hRequirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (4.66.4) 2024-06-26T07:15:42.7346204Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.32.3) 2024-06-26T07:15:42.7349390Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.5.0a0+gitb8c4c54) 2024-06-26T07:15:42.7353156Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (1.21.2) 2024-06-26T07:15:42.7356570Z Requirement already satisfied: torchdata in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (0.7.0a0+11bb5b8) 2024-06-26T07:15:42.7418704Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.3.2) 2024-06-26T07:15:42.7422697Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.7) 2024-06-26T07:15:42.7429411Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (1.26.19) 2024-06-26T07:15:42.7434905Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (2024.6.2) 2024-06-26T07:15:42.7490716Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.13.1) 2024-06-26T07:15:42.7497368Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (4.12.2) 2024-06-26T07:15:42.7500312Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (1.12.1) 2024-06-26T07:15:42.7503242Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2.8.8) 2024-06-26T07:15:42.7506609Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.1.4) 2024-06-26T07:15:42.7509030Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2024.2.0) 2024-06-26T07:15:42.8203030Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchtext==0.16.0a0+b0ebddc) (2.1.5) 2024-06-26T07:15:42.8366771Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchtext==0.16.0a0+b0ebddc) (1.3.0) 2024-06-26T07:15:42.8450907Z Building wheels for collected packages: torchtext 2024-06-26T07:16:19.1288066Z Building wheel for torchtext (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2024-06-26T07:16:19.1315224Z [?25h Created wheel for torchtext: filename=torchtext-0.16.0a0+b0ebddc-cp310-cp310-linux_x86_64.whl size=2045299 sha256=2cd48778feff77447f7e54b7d8bd7ba23be8532e3433a81e964738ac482a9dcf 2024-06-26T07:16:19.1317087Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/e5/3d/d7/faafad098ec9437ecdf4495c0f5e72b817fffbb06063e32cbf 2024-06-26T07:16:19.1340643Z Successfully built torchtext 2024-06-26T07:16:20.3393569Z Installing collected packages: torchtext 2024-06-26T07:16:20.4431230Z Successfully installed torchtext-0.16.0a0+b0ebddc 2024-06-26T07:16:20.5652114Z + install_torchvision 2024-06-26T07:16:20.5652479Z + local orig_preload 2024-06-26T07:16:20.5652811Z + local commit 2024-06-26T07:16:20.5655552Z ++ get_pinned_commit vision 2024-06-26T07:16:20.5656032Z ++ cat .github/ci_commit_pins/vision.txt 2024-06-26T07:16:20.5665886Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:16:20.5666372Z + orig_preload= 2024-06-26T07:16:20.5666743Z + '[' -n '' ']' 2024-06-26T07:16:20.5667587Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:16:20.5669017Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:16:20.8797695Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:16:20.8801779Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-d_hw1asf 2024-06-26T07:16:20.8819770Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-d_hw1asf 2024-06-26T07:16:22.3029614Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-06-26T07:16:22.3048131Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:16:23.5548918Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:16:23.7845136Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:16:26.1011949Z Preparing metadata (setup.py) ... [?25l- \ done 2024-06-26T07:16:26.1061014Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2) 2024-06-26T07:16:26.1064056Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+gitb8c4c54) 2024-06-26T07:16:26.1069506Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-06-26T07:16:26.1278114Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-06-26T07:16:26.1283922Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-06-26T07:16:26.1286345Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12.1) 2024-06-26T07:16:26.1290051Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-06-26T07:16:26.1293519Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-06-26T07:16:26.1296914Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.2.0) 2024-06-26T07:16:26.1727606Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-06-26T07:16:26.1885015Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-06-26T07:16:26.1960709Z Building wheels for collected packages: torchvision 2024-06-26T07:17:37.1635904Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2024-06-26T07:17:37.1661687Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2112685 sha256=37e9c09212607a82b18a6378eee08dcae02aa4abece44f362172e16954f37d2f 2024-06-26T07:17:37.1663464Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-06-26T07:17:37.1697486Z Successfully built torchvision 2024-06-26T07:17:38.3769270Z Installing collected packages: torchvision 2024-06-26T07:17:38.7763956Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-06-26T07:17:38.8883985Z + '[' -n '' ']' 2024-06-26T07:17:38.8884345Z + id=0 2024-06-26T07:17:38.8887916Z + pip_install opencv-python==4.8.0.74 2024-06-26T07:17:38.8888778Z + pip install --progress-bar off opencv-python==4.8.0.74 2024-06-26T07:17:39.3168490Z Collecting opencv-python==4.8.0.74 2024-06-26T07:17:39.3358819Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB) 2024-06-26T07:17:39.3548844Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.21.2) 2024-06-26T07:17:39.3622130Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB) 2024-06-26T07:17:41.2308847Z Installing collected packages: opencv-python 2024-06-26T07:17:42.0507076Z Successfully installed opencv-python-4.8.0.74 2024-06-26T07:17:42.1458539Z + [[ aot_eager_torchbench == *inductor_torchbench_smoketest_perf* ]] 2024-06-26T07:17:42.1459445Z + [[ aot_eager_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]] 2024-06-26T07:17:42.1460286Z + [[ aot_eager_torchbench == *torchbench_gcp_smoketest* ]] 2024-06-26T07:17:42.1460992Z + checkout_install_torchbench 2024-06-26T07:17:42.1461489Z + local commit 2024-06-26T07:17:42.1462214Z ++ get_pinned_commit torchbench 2024-06-26T07:17:42.1462827Z ++ cat .github/ci_commit_pins/torchbench.txt 2024-06-26T07:17:42.1473177Z + commit=23512dbebd44a11eb84afbf53c3c071dd105297e 2024-06-26T07:17:42.1473949Z + git clone https://github.com/pytorch/benchmark torchbench 2024-06-26T07:17:42.1482583Z Cloning into 'torchbench'... 2024-06-26T07:17:42.2778420Z remote: Enumerating objects: 30187, done. 2024-06-26T07:17:42.2779208Z remote: Counting objects: 0% (1/546) 2024-06-26T07:17:42.2780210Z remote: Counting objects: 1% (6/546) 2024-06-26T07:17:42.2780845Z remote: Counting objects: 2% (11/546) 2024-06-26T07:17:42.2781353Z remote: Counting objects: 3% (17/546) 2024-06-26T07:17:42.2781850Z remote: Counting objects: 4% (22/546) 2024-06-26T07:17:42.2782403Z remote: Counting objects: 5% (28/546) 2024-06-26T07:17:42.2783108Z remote: Counting objects: 6% (33/546) 2024-06-26T07:17:42.2783811Z remote: Counting objects: 7% (39/546) 2024-06-26T07:17:42.2784345Z remote: Counting objects: 8% (44/546) 2024-06-26T07:17:42.2784837Z remote: Counting objects: 9% (50/546) 2024-06-26T07:17:42.2785347Z remote: Counting objects: 10% (55/546) 2024-06-26T07:17:42.2786074Z remote: Counting objects: 11% (61/546) 2024-06-26T07:17:42.2786661Z remote: Counting objects: 12% (66/546) 2024-06-26T07:17:42.2787350Z remote: Counting objects: 13% (71/546) 2024-06-26T07:17:42.2787990Z remote: Counting objects: 14% (77/546) 2024-06-26T07:17:42.2788494Z remote: Counting objects: 15% (82/546) 2024-06-26T07:17:42.2789001Z remote: Counting objects: 16% (88/546) 2024-06-26T07:17:42.2789490Z remote: Counting objects: 17% (93/546) 2024-06-26T07:17:42.2790106Z remote: Counting objects: 18% (99/546) 2024-06-26T07:17:42.2790611Z remote: Counting objects: 19% (104/546) 2024-06-26T07:17:42.2791293Z remote: Counting objects: 20% (110/546) 2024-06-26T07:17:42.2792058Z remote: Counting objects: 21% (115/546) 2024-06-26T07:17:42.2792828Z remote: Counting objects: 22% (121/546) 2024-06-26T07:17:42.2793349Z remote: Counting objects: 23% (126/546) 2024-06-26T07:17:42.2793872Z remote: Counting objects: 24% (132/546) 2024-06-26T07:17:42.2794369Z remote: Counting objects: 25% (137/546) 2024-06-26T07:17:42.2794873Z remote: Counting objects: 26% (142/546) 2024-06-26T07:17:42.2795389Z remote: Counting objects: 27% (148/546) 2024-06-26T07:17:42.2795887Z remote: Counting objects: 28% (153/546) 2024-06-26T07:17:42.2796389Z remote: Counting objects: 29% (159/546) 2024-06-26T07:17:42.2796920Z remote: Counting objects: 30% (164/546) 2024-06-26T07:17:42.2797539Z remote: Counting objects: 31% (170/546) 2024-06-26T07:17:42.2798235Z remote: Counting objects: 32% (175/546) 2024-06-26T07:17:42.2798911Z remote: Counting objects: 33% (181/546) 2024-06-26T07:17:42.2799451Z remote: Counting objects: 34% (186/546) 2024-06-26T07:17:42.2799964Z remote: Counting objects: 35% (192/546) 2024-06-26T07:17:42.2800455Z remote: Counting objects: 36% (197/546) 2024-06-26T07:17:42.2800953Z remote: Counting objects: 37% (203/546) 2024-06-26T07:17:42.2801450Z remote: Counting objects: 38% (208/546) 2024-06-26T07:17:42.2801951Z remote: Counting objects: 39% (213/546) 2024-06-26T07:17:42.2802447Z remote: Counting objects: 40% (219/546) 2024-06-26T07:17:42.2802947Z remote: Counting objects: 41% (224/546) 2024-06-26T07:17:42.2803443Z remote: Counting objects: 42% (230/546) 2024-06-26T07:17:42.2803940Z remote: Counting objects: 43% (235/546) 2024-06-26T07:17:42.2804440Z remote: Counting objects: 44% (241/546) 2024-06-26T07:17:42.2804937Z remote: Counting objects: 45% (246/546) 2024-06-26T07:17:42.2805438Z remote: Counting objects: 46% (252/546) 2024-06-26T07:17:42.2805938Z remote: Counting objects: 47% (257/546) 2024-06-26T07:17:42.2806427Z remote: Counting objects: 48% (263/546) 2024-06-26T07:17:42.2806939Z remote: Counting objects: 49% (268/546) 2024-06-26T07:17:42.2807489Z remote: Counting objects: 50% (273/546) 2024-06-26T07:17:42.2808013Z remote: Counting objects: 51% (279/546) 2024-06-26T07:17:42.2808503Z remote: Counting objects: 52% (284/546) 2024-06-26T07:17:42.2809003Z remote: Counting objects: 53% (290/546) 2024-06-26T07:17:42.2809505Z remote: Counting objects: 54% (295/546) 2024-06-26T07:17:42.2810002Z remote: Counting objects: 55% (301/546) 2024-06-26T07:17:42.2810488Z remote: Counting objects: 56% (306/546) 2024-06-26T07:17:42.2810982Z remote: Counting objects: 57% (312/546) 2024-06-26T07:17:42.2811677Z remote: Counting objects: 58% (317/546) 2024-06-26T07:17:42.2812185Z remote: Counting objects: 59% (323/546) 2024-06-26T07:17:42.2812676Z remote: Counting objects: 60% (328/546) 2024-06-26T07:17:42.2813173Z remote: Counting objects: 61% (334/546) 2024-06-26T07:17:42.2813671Z remote: Counting objects: 62% (339/546) 2024-06-26T07:17:42.2814167Z remote: Counting objects: 63% (344/546) 2024-06-26T07:17:42.2814663Z remote: Counting objects: 64% (350/546) 2024-06-26T07:17:42.2815161Z remote: Counting objects: 65% (355/546) 2024-06-26T07:17:42.2815657Z remote: Counting objects: 66% (361/546) 2024-06-26T07:17:42.2816153Z remote: Counting objects: 67% (366/546) 2024-06-26T07:17:42.2816650Z remote: Counting objects: 68% (372/546) 2024-06-26T07:17:42.2817194Z remote: Counting objects: 69% (377/546) 2024-06-26T07:17:42.2817700Z remote: Counting objects: 70% (383/546) 2024-06-26T07:17:42.2818192Z remote: Counting objects: 71% (388/546) 2024-06-26T07:17:42.2818696Z remote: Counting objects: 72% (394/546) 2024-06-26T07:17:42.2819195Z remote: Counting objects: 73% (399/546) 2024-06-26T07:17:42.2819696Z remote: Counting objects: 74% (405/546) 2024-06-26T07:17:42.2820185Z remote: Counting objects: 75% (410/546) 2024-06-26T07:17:42.2820682Z remote: Counting objects: 76% (415/546) 2024-06-26T07:17:42.2821268Z remote: Counting objects: 77% (421/546) 2024-06-26T07:17:42.2821758Z remote: Counting objects: 78% (426/546) 2024-06-26T07:17:42.2822255Z remote: Counting objects: 79% (432/546) 2024-06-26T07:17:42.2822751Z remote: Counting objects: 80% (437/546) 2024-06-26T07:17:42.2823249Z remote: Counting objects: 81% (443/546) 2024-06-26T07:17:42.2823740Z remote: Counting objects: 82% (448/546) 2024-06-26T07:17:42.2824242Z remote: Counting objects: 83% (454/546) 2024-06-26T07:17:42.2824743Z remote: Counting objects: 84% (459/546) 2024-06-26T07:17:42.2825244Z remote: Counting objects: 85% (465/546) 2024-06-26T07:17:42.2825741Z remote: Counting objects: 86% (470/546) 2024-06-26T07:17:42.2826239Z remote: Counting objects: 87% (476/546) 2024-06-26T07:17:42.2826740Z remote: Counting objects: 88% (481/546) 2024-06-26T07:17:42.2827242Z remote: Counting objects: 89% (486/546) 2024-06-26T07:17:42.2827732Z remote: Counting objects: 90% (492/546) 2024-06-26T07:17:42.2828237Z remote: Counting objects: 91% (497/546) 2024-06-26T07:17:42.2828734Z remote: Counting objects: 92% (503/546) 2024-06-26T07:17:42.2829236Z remote: Counting objects: 93% (508/546) 2024-06-26T07:17:42.2829731Z remote: Counting objects: 94% (514/546) 2024-06-26T07:17:42.2830381Z remote: Counting objects: 95% (519/546) 2024-06-26T07:17:42.2830882Z remote: Counting objects: 96% (525/546) 2024-06-26T07:17:42.2831371Z remote: Counting objects: 97% (530/546) 2024-06-26T07:17:42.2831873Z remote: Counting objects: 98% (536/546) 2024-06-26T07:17:42.2832597Z remote: Counting objects: 99% (541/546) 2024-06-26T07:17:42.2833159Z remote: Counting objects: 100% (546/546) 2024-06-26T07:17:42.2833746Z remote: Counting objects: 100% (546/546), done. 2024-06-26T07:17:42.2834305Z remote: Compressing objects: 0% (1/318) 2024-06-26T07:17:42.2834834Z remote: Compressing objects: 1% (4/318) 2024-06-26T07:17:42.2835359Z remote: Compressing objects: 2% (7/318) 2024-06-26T07:17:42.2835892Z remote: Compressing objects: 3% (10/318) 2024-06-26T07:17:42.2836436Z remote: Compressing objects: 4% (13/318) 2024-06-26T07:17:42.2836988Z remote: Compressing objects: 5% (16/318) 2024-06-26T07:17:42.2837542Z remote: Compressing objects: 6% (20/318) 2024-06-26T07:17:42.2838062Z remote: Compressing objects: 7% (23/318) 2024-06-26T07:17:42.2838583Z remote: Compressing objects: 8% (26/318) 2024-06-26T07:17:42.2839109Z remote: Compressing objects: 9% (29/318) 2024-06-26T07:17:42.2839636Z remote: Compressing objects: 10% (32/318) 2024-06-26T07:17:42.2840163Z remote: Compressing objects: 11% (35/318) 2024-06-26T07:17:42.2840780Z remote: Compressing objects: 12% (39/318) 2024-06-26T07:17:42.2841544Z remote: Compressing objects: 13% (42/318) 2024-06-26T07:17:42.2845653Z remote: Compressing objects: 14% (45/318) 2024-06-26T07:17:42.2846304Z remote: Compressing objects: 15% (48/318) 2024-06-26T07:17:42.2846846Z remote: Compressing objects: 16% (51/318) 2024-06-26T07:17:42.2847632Z remote: Compressing objects: 17% (55/318) 2024-06-26T07:17:42.2848314Z remote: Compressing objects: 18% (58/318) 2024-06-26T07:17:42.2848918Z remote: Compressing objects: 19% (61/318) 2024-06-26T07:17:42.2849584Z remote: Compressing objects: 20% (64/318) 2024-06-26T07:17:42.2850111Z remote: Compressing objects: 21% (67/318) 2024-06-26T07:17:42.2850642Z remote: Compressing objects: 22% (70/318) 2024-06-26T07:17:42.2851159Z remote: Compressing objects: 23% (74/318) 2024-06-26T07:17:42.2851688Z remote: Compressing objects: 24% (77/318) 2024-06-26T07:17:42.2852174Z remote: Compressing objects: 25% (80/318) 2024-06-26T07:17:42.2852709Z remote: Compressing objects: 26% (83/318) 2024-06-26T07:17:42.2853228Z remote: Compressing objects: 27% (86/318) 2024-06-26T07:17:42.2853758Z remote: Compressing objects: 28% (90/318) 2024-06-26T07:17:42.2854299Z remote: Compressing objects: 29% (93/318) 2024-06-26T07:17:42.2854952Z remote: Compressing objects: 30% (96/318) 2024-06-26T07:17:42.2855467Z remote: Compressing objects: 31% (99/318) 2024-06-26T07:17:42.2855999Z remote: Compressing objects: 32% (102/318) 2024-06-26T07:17:42.2856545Z remote: Compressing objects: 33% (105/318) 2024-06-26T07:17:42.2857092Z remote: Compressing objects: 34% (109/318) 2024-06-26T07:17:42.2857673Z remote: Compressing objects: 35% (112/318) 2024-06-26T07:17:42.2858212Z remote: Compressing objects: 36% (115/318) 2024-06-26T07:17:42.2858747Z remote: Compressing objects: 37% (118/318) 2024-06-26T07:17:42.2859288Z remote: Compressing objects: 38% (121/318) 2024-06-26T07:17:42.2859823Z remote: Compressing objects: 39% (125/318) 2024-06-26T07:17:42.2860356Z remote: Compressing objects: 40% (128/318) 2024-06-26T07:17:42.2860899Z remote: Compressing objects: 41% (131/318) 2024-06-26T07:17:42.2867469Z remote: Compressing objects: 42% (134/318) 2024-06-26T07:17:42.2868149Z remote: Compressing objects: 43% (137/318) 2024-06-26T07:17:42.2868813Z remote: Compressing objects: 44% (140/318) 2024-06-26T07:17:42.2869550Z remote: Compressing objects: 45% (144/318) 2024-06-26T07:17:42.2870405Z remote: Compressing objects: 46% (147/318) 2024-06-26T07:17:42.2871106Z remote: Compressing objects: 47% (150/318) 2024-06-26T07:17:42.2871646Z remote: Compressing objects: 48% (153/318) 2024-06-26T07:17:42.2872208Z remote: Compressing objects: 49% (156/318) 2024-06-26T07:17:42.2873046Z remote: Compressing objects: 50% (159/318) 2024-06-26T07:17:42.2873587Z remote: Compressing objects: 51% (163/318) 2024-06-26T07:17:42.2874121Z remote: Compressing objects: 52% (166/318) 2024-06-26T07:17:42.2874656Z remote: Compressing objects: 53% (169/318) 2024-06-26T07:17:42.2875194Z remote: Compressing objects: 54% (172/318) 2024-06-26T07:17:42.2875725Z remote: Compressing objects: 55% (175/318) 2024-06-26T07:17:42.2876263Z remote: Compressing objects: 56% (179/318) 2024-06-26T07:17:42.2876807Z remote: Compressing objects: 57% (182/318) 2024-06-26T07:17:42.2877389Z remote: Compressing objects: 58% (185/318) 2024-06-26T07:17:42.2877929Z remote: Compressing objects: 59% (188/318) 2024-06-26T07:17:42.2878519Z remote: Compressing objects: 60% (191/318) 2024-06-26T07:17:42.2879052Z remote: Compressing objects: 61% (194/318) 2024-06-26T07:17:42.2879635Z remote: Compressing objects: 62% (198/318) 2024-06-26T07:17:42.2880160Z remote: Compressing objects: 63% (201/318) 2024-06-26T07:17:42.2880688Z remote: Compressing objects: 64% (204/318) 2024-06-26T07:17:42.2881217Z remote: Compressing objects: 65% (207/318) 2024-06-26T07:17:42.2881867Z remote: Compressing objects: 66% (210/318) 2024-06-26T07:17:42.2882400Z remote: Compressing objects: 67% (214/318) 2024-06-26T07:17:42.2882925Z remote: Compressing objects: 68% (217/318) 2024-06-26T07:17:42.2883453Z remote: Compressing objects: 69% (220/318) 2024-06-26T07:17:42.2883986Z remote: Compressing objects: 70% (223/318) 2024-06-26T07:17:42.2884504Z remote: Compressing objects: 71% (226/318) 2024-06-26T07:17:42.2885033Z remote: Compressing objects: 72% (229/318) 2024-06-26T07:17:42.2885558Z remote: Compressing objects: 73% (233/318) 2024-06-26T07:17:42.2886094Z remote: Compressing objects: 74% (236/318) 2024-06-26T07:17:42.2886618Z remote: Compressing objects: 75% (239/318) 2024-06-26T07:17:42.2887145Z remote: Compressing objects: 76% (242/318) 2024-06-26T07:17:42.2887670Z remote: Compressing objects: 77% (245/318) 2024-06-26T07:17:42.2888191Z remote: Compressing objects: 78% (249/318) 2024-06-26T07:17:42.2888726Z remote: Compressing objects: 79% (252/318) 2024-06-26T07:17:42.2889255Z remote: Compressing objects: 80% (255/318) 2024-06-26T07:17:42.2889786Z remote: Compressing objects: 81% (258/318) 2024-06-26T07:17:42.2890310Z remote: Compressing objects: 82% (261/318) 2024-06-26T07:17:42.2890838Z remote: Compressing objects: 83% (264/318) 2024-06-26T07:17:42.2891469Z remote: Compressing objects: 84% (268/318) 2024-06-26T07:17:42.2891993Z remote: Compressing objects: 85% (271/318) 2024-06-26T07:17:42.2892511Z remote: Compressing objects: 86% (274/318) 2024-06-26T07:17:42.2893037Z remote: Compressing objects: 87% (277/318) 2024-06-26T07:17:42.2893567Z remote: Compressing objects: 88% (280/318) 2024-06-26T07:17:42.2894109Z remote: Compressing objects: 89% (284/318) 2024-06-26T07:17:42.2894627Z remote: Compressing objects: 90% (287/318) 2024-06-26T07:17:42.2895154Z remote: Compressing objects: 91% (290/318) 2024-06-26T07:17:42.2895677Z remote: Compressing objects: 92% (293/318) 2024-06-26T07:17:42.2896209Z remote: Compressing objects: 93% (296/318) 2024-06-26T07:17:42.2896736Z remote: Compressing objects: 94% (299/318) 2024-06-26T07:17:42.2897311Z remote: Compressing objects: 95% (303/318) 2024-06-26T07:17:42.2897842Z remote: Compressing objects: 96% (306/318) 2024-06-26T07:17:42.2898374Z remote: Compressing objects: 97% (309/318) 2024-06-26T07:17:42.2898895Z remote: Compressing objects: 98% (312/318) 2024-06-26T07:17:42.2899419Z remote: Compressing objects: 99% (315/318) 2024-06-26T07:17:42.2899947Z remote: Compressing objects: 100% (318/318) 2024-06-26T07:17:42.2900510Z remote: Compressing objects: 100% (318/318), done. 2024-06-26T07:17:42.2939809Z Receiving objects: 0% (1/30187) 2024-06-26T07:17:42.2976698Z Receiving objects: 1% (302/30187) 2024-06-26T07:17:42.3019559Z Receiving objects: 2% (604/30187) 2024-06-26T07:17:42.3057820Z Receiving objects: 3% (906/30187) 2024-06-26T07:17:42.3087153Z Receiving objects: 4% (1208/30187) 2024-06-26T07:17:42.3122514Z Receiving objects: 5% (1510/30187) 2024-06-26T07:17:42.3156303Z Receiving objects: 6% (1812/30187) 2024-06-26T07:17:42.3198584Z Receiving objects: 7% (2114/30187) 2024-06-26T07:17:42.3233542Z Receiving objects: 8% (2415/30187) 2024-06-26T07:17:42.3267747Z Receiving objects: 9% (2717/30187) 2024-06-26T07:17:42.3306398Z Receiving objects: 10% (3019/30187) 2024-06-26T07:17:42.3368252Z Receiving objects: 11% (3321/30187) 2024-06-26T07:17:42.3432732Z Receiving objects: 12% (3623/30187) 2024-06-26T07:17:42.3488729Z Receiving objects: 13% (3925/30187) 2024-06-26T07:17:42.3658766Z Receiving objects: 14% (4227/30187) 2024-06-26T07:17:42.3759715Z Receiving objects: 15% (4529/30187) 2024-06-26T07:17:42.3846213Z Receiving objects: 16% (4830/30187) 2024-06-26T07:17:42.3900425Z Receiving objects: 17% (5132/30187) 2024-06-26T07:17:42.3937257Z Receiving objects: 18% (5434/30187) 2024-06-26T07:17:42.3964740Z Receiving objects: 19% (5736/30187) 2024-06-26T07:17:42.4029877Z Receiving objects: 20% (6038/30187) 2024-06-26T07:17:42.4049012Z Receiving objects: 21% (6340/30187) 2024-06-26T07:17:42.4067151Z Receiving objects: 22% (6642/30187) 2024-06-26T07:17:42.4867714Z Receiving objects: 23% (6944/30187) 2024-06-26T07:17:43.2895372Z Receiving objects: 24% (7245/30187) 2024-06-26T07:17:43.3774606Z Receiving objects: 24% (7447/30187), 80.51 MiB | 80.58 MiB/s 2024-06-26T07:17:43.8705480Z Receiving objects: 25% (7547/30187), 80.51 MiB | 80.58 MiB/s 2024-06-26T07:17:43.9370067Z Receiving objects: 26% (7849/30187), 119.01 MiB | 79.39 MiB/s 2024-06-26T07:17:44.0029913Z Receiving objects: 27% (8151/30187), 119.01 MiB | 79.39 MiB/s 2024-06-26T07:17:44.0692958Z Receiving objects: 28% (8453/30187), 119.01 MiB | 79.39 MiB/s 2024-06-26T07:17:44.1345970Z Receiving objects: 29% (8755/30187), 119.01 MiB | 79.39 MiB/s 2024-06-26T07:17:44.2001658Z Receiving objects: 30% (9057/30187), 119.01 MiB | 79.39 MiB/s 2024-06-26T07:17:44.2668705Z Receiving objects: 31% (9358/30187), 119.01 MiB | 79.39 MiB/s 2024-06-26T07:17:44.2894375Z Receiving objects: 32% (9660/30187), 119.01 MiB | 79.39 MiB/s 2024-06-26T07:17:44.3324961Z Receiving objects: 32% (9763/30187), 161.26 MiB | 80.67 MiB/s 2024-06-26T07:17:44.5576638Z Receiving objects: 33% (9962/30187), 161.26 MiB | 80.67 MiB/s 2024-06-26T07:17:44.6295952Z Receiving objects: 34% (10264/30187), 161.26 MiB | 80.67 MiB/s 2024-06-26T07:17:44.6414909Z Receiving objects: 35% (10566/30187), 161.26 MiB | 80.67 MiB/s 2024-06-26T07:17:44.8374504Z Receiving objects: 36% (10868/30187), 161.26 MiB | 80.67 MiB/s 2024-06-26T07:17:45.1082437Z Receiving objects: 37% (11170/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1284989Z Receiving objects: 38% (11472/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1298286Z Receiving objects: 39% (11773/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1323494Z Receiving objects: 40% (12075/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1335923Z Receiving objects: 41% (12377/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1348827Z Receiving objects: 42% (12679/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1359623Z Receiving objects: 43% (12981/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1368604Z Receiving objects: 44% (13283/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1387669Z Receiving objects: 45% (13585/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1398939Z Receiving objects: 46% (13887/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1402725Z Receiving objects: 47% (14188/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1409526Z Receiving objects: 48% (14490/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1430238Z Receiving objects: 49% (14792/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1446619Z Receiving objects: 50% (15094/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1456385Z Receiving objects: 51% (15396/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1650061Z Receiving objects: 52% (15698/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1654047Z Receiving objects: 53% (16000/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1661435Z Receiving objects: 54% (16301/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1666767Z Receiving objects: 55% (16603/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1671971Z Receiving objects: 56% (16905/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1688148Z Receiving objects: 57% (17207/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1698090Z Receiving objects: 58% (17509/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1706263Z Receiving objects: 59% (17811/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1719529Z Receiving objects: 60% (18113/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1727628Z Receiving objects: 61% (18415/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1736872Z Receiving objects: 62% (18716/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1864385Z Receiving objects: 63% (19018/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1870091Z Receiving objects: 64% (19320/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1876531Z Receiving objects: 65% (19622/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1882269Z Receiving objects: 66% (19924/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1890780Z Receiving objects: 67% (20226/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.1898025Z Receiving objects: 68% (20528/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.2140434Z Receiving objects: 69% (20830/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.2526138Z Receiving objects: 70% (21131/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.2532154Z Receiving objects: 71% (21433/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.2543072Z Receiving objects: 72% (21735/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.2555494Z Receiving objects: 73% (22037/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.2567868Z Receiving objects: 74% (22339/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.2577809Z Receiving objects: 75% (22641/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.2586070Z Receiving objects: 76% (22943/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.2604132Z Receiving objects: 77% (23244/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.2609998Z Receiving objects: 78% (23546/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.2617867Z Receiving objects: 79% (23848/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.2639713Z Receiving objects: 80% (24150/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.2894578Z Receiving objects: 81% (24452/30187), 192.81 MiB | 77.15 MiB/s 2024-06-26T07:17:45.4579629Z Receiving objects: 81% (24547/30187), 230.67 MiB | 76.94 MiB/s 2024-06-26T07:17:45.4790150Z Receiving objects: 82% (24754/30187), 230.67 MiB | 76.94 MiB/s 2024-06-26T07:17:45.4821616Z Receiving objects: 83% (25056/30187), 230.67 MiB | 76.94 MiB/s 2024-06-26T07:17:45.4834203Z Receiving objects: 84% (25358/30187), 230.67 MiB | 76.94 MiB/s 2024-06-26T07:17:45.4845076Z Receiving objects: 85% (25659/30187), 230.67 MiB | 76.94 MiB/s 2024-06-26T07:17:45.4874318Z Receiving objects: 86% (25961/30187), 230.67 MiB | 76.94 MiB/s 2024-06-26T07:17:45.4903171Z Receiving objects: 87% (26263/30187), 230.67 MiB | 76.94 MiB/s 2024-06-26T07:17:45.5031786Z Receiving objects: 88% (26565/30187), 230.67 MiB | 76.94 MiB/s 2024-06-26T07:17:45.5037767Z Receiving objects: 89% (26867/30187), 230.67 MiB | 76.94 MiB/s 2024-06-26T07:17:45.5045073Z Receiving objects: 90% (27169/30187), 230.67 MiB | 76.94 MiB/s 2024-06-26T07:17:45.5050351Z Receiving objects: 91% (27471/30187), 230.67 MiB | 76.94 MiB/s 2024-06-26T07:17:45.5057189Z Receiving objects: 92% (27773/30187), 230.67 MiB | 76.94 MiB/s 2024-06-26T07:17:45.5063005Z Receiving objects: 93% (28074/30187), 230.67 MiB | 76.94 MiB/s 2024-06-26T07:17:45.5174294Z Receiving objects: 94% (28376/30187), 230.67 MiB | 76.94 MiB/s 2024-06-26T07:17:45.5188639Z Receiving objects: 95% (28678/30187), 230.67 MiB | 76.94 MiB/s 2024-06-26T07:17:45.5198943Z Receiving objects: 96% (28980/30187), 230.67 MiB | 76.94 MiB/s 2024-06-26T07:17:45.5216315Z Receiving objects: 97% (29282/30187), 230.67 MiB | 76.94 MiB/s 2024-06-26T07:17:45.5230880Z Receiving objects: 98% (29584/30187), 230.67 MiB | 76.94 MiB/s 2024-06-26T07:17:45.5270971Z Receiving objects: 99% (29886/30187), 230.67 MiB | 76.94 MiB/s 2024-06-26T07:17:45.5272058Z remote: Total 30187 (delta 256), reused 486 (delta 216), pack-reused 29641 2024-06-26T07:17:45.5288776Z Receiving objects: 100% (30187/30187), 230.67 MiB | 76.94 MiB/s 2024-06-26T07:17:45.5289585Z Receiving objects: 100% (30187/30187), 245.25 MiB | 75.71 MiB/s, done. 2024-06-26T07:17:45.5318243Z Resolving deltas: 0% (0/16205) 2024-06-26T07:17:45.5326946Z Resolving deltas: 1% (163/16205) 2024-06-26T07:17:45.5341390Z Resolving deltas: 2% (328/16205) 2024-06-26T07:17:45.5356058Z Resolving deltas: 3% (514/16205) 2024-06-26T07:17:45.5376365Z Resolving deltas: 4% (649/16205) 2024-06-26T07:17:45.5386869Z Resolving deltas: 5% (821/16205) 2024-06-26T07:17:45.5394871Z Resolving deltas: 6% (974/16205) 2024-06-26T07:17:45.5405509Z Resolving deltas: 7% (1139/16205) 2024-06-26T07:17:45.5414537Z Resolving deltas: 8% (1300/16205) 2024-06-26T07:17:45.5425069Z Resolving deltas: 9% (1460/16205) 2024-06-26T07:17:45.5436534Z Resolving deltas: 10% (1634/16205) 2024-06-26T07:17:45.5447631Z Resolving deltas: 11% (1785/16205) 2024-06-26T07:17:45.5456913Z Resolving deltas: 12% (1946/16205) 2024-06-26T07:17:45.5471291Z Resolving deltas: 13% (2110/16205) 2024-06-26T07:17:45.5480740Z Resolving deltas: 14% (2272/16205) 2024-06-26T07:17:45.5499460Z Resolving deltas: 15% (2435/16205) 2024-06-26T07:17:45.5515728Z Resolving deltas: 16% (2620/16205) 2024-06-26T07:17:45.5531823Z Resolving deltas: 17% (2759/16205) 2024-06-26T07:17:45.5533954Z Resolving deltas: 18% (3043/16205) 2024-06-26T07:17:45.5541999Z Resolving deltas: 19% (3087/16205) 2024-06-26T07:17:45.5561469Z Resolving deltas: 20% (3267/16205) 2024-06-26T07:17:45.5588193Z Resolving deltas: 21% (3417/16205) 2024-06-26T07:17:45.5612103Z Resolving deltas: 22% (3566/16205) 2024-06-26T07:17:45.5615302Z Resolving deltas: 23% (3845/16205) 2024-06-26T07:17:45.5623709Z Resolving deltas: 24% (3897/16205) 2024-06-26T07:17:45.5634490Z Resolving deltas: 25% (4054/16205) 2024-06-26T07:17:45.5644399Z Resolving deltas: 26% (4235/16205) 2024-06-26T07:17:45.5653008Z Resolving deltas: 27% (4414/16205) 2024-06-26T07:17:45.5690483Z Resolving deltas: 28% (4539/16205) 2024-06-26T07:17:45.5712059Z Resolving deltas: 29% (4700/16205) 2024-06-26T07:17:45.5725868Z Resolving deltas: 30% (4864/16205) 2024-06-26T07:17:45.5739341Z Resolving deltas: 31% (5030/16205) 2024-06-26T07:17:45.5755326Z Resolving deltas: 32% (5188/16205) 2024-06-26T07:17:45.5770407Z Resolving deltas: 33% (5348/16205) 2024-06-26T07:17:45.5780057Z Resolving deltas: 34% (5534/16205) 2024-06-26T07:17:45.5788602Z Resolving deltas: 35% (5708/16205) 2024-06-26T07:17:45.5801084Z Resolving deltas: 36% (5834/16205) 2024-06-26T07:17:45.5820809Z Resolving deltas: 37% (6008/16205) 2024-06-26T07:17:45.5835046Z Resolving deltas: 38% (6161/16205) 2024-06-26T07:17:45.5853225Z Resolving deltas: 39% (6324/16205) 2024-06-26T07:17:45.5861066Z Resolving deltas: 40% (6483/16205) 2024-06-26T07:17:45.5882302Z Resolving deltas: 41% (6663/16205) 2024-06-26T07:17:45.5893346Z Resolving deltas: 42% (6825/16205) 2024-06-26T07:17:45.5900376Z Resolving deltas: 43% (7027/16205) 2024-06-26T07:17:45.5923086Z Resolving deltas: 44% (7165/16205) 2024-06-26T07:17:45.5927915Z Resolving deltas: 45% (7363/16205) 2024-06-26T07:17:45.5935940Z Resolving deltas: 46% (7473/16205) 2024-06-26T07:17:45.5945804Z Resolving deltas: 47% (7627/16205) 2024-06-26T07:17:45.5966134Z Resolving deltas: 48% (7784/16205) 2024-06-26T07:17:45.5985297Z Resolving deltas: 49% (7941/16205) 2024-06-26T07:17:45.5997197Z Resolving deltas: 50% (8109/16205) 2024-06-26T07:17:45.6020177Z Resolving deltas: 51% (8268/16205) 2024-06-26T07:17:45.6029455Z Resolving deltas: 52% (8430/16205) 2024-06-26T07:17:45.6036225Z Resolving deltas: 53% (8590/16205) 2024-06-26T07:17:45.6045255Z Resolving deltas: 54% (8755/16205) 2024-06-26T07:17:45.6060415Z Resolving deltas: 55% (8914/16205) 2024-06-26T07:17:45.6068765Z Resolving deltas: 56% (9093/16205) 2024-06-26T07:17:45.6084315Z Resolving deltas: 57% (9239/16205) 2024-06-26T07:17:45.6092471Z Resolving deltas: 58% (9401/16205) 2024-06-26T07:17:45.6102471Z Resolving deltas: 59% (9613/16205) 2024-06-26T07:17:45.6103023Z Resolving deltas: 60% (9871/16205) 2024-06-26T07:17:45.6110983Z Resolving deltas: 61% (9886/16205) 2024-06-26T07:17:45.6124627Z Resolving deltas: 62% (10048/16205) 2024-06-26T07:17:45.6136510Z Resolving deltas: 63% (10306/16205) 2024-06-26T07:17:45.6136953Z Resolving deltas: 64% (10521/16205) 2024-06-26T07:17:45.6144439Z Resolving deltas: 65% (10536/16205) 2024-06-26T07:17:45.6153218Z Resolving deltas: 66% (10700/16205) 2024-06-26T07:17:45.6172870Z Resolving deltas: 67% (10875/16205) 2024-06-26T07:17:45.6179798Z Resolving deltas: 68% (11178/16205) 2024-06-26T07:17:45.6195345Z Resolving deltas: 69% (11283/16205) 2024-06-26T07:17:45.6203779Z Resolving deltas: 70% (11369/16205) 2024-06-26T07:17:45.6210784Z Resolving deltas: 71% (11515/16205) 2024-06-26T07:17:45.6224899Z Resolving deltas: 72% (11671/16205) 2024-06-26T07:17:45.6238028Z Resolving deltas: 73% (11856/16205) 2024-06-26T07:17:45.6251640Z Resolving deltas: 74% (11993/16205) 2024-06-26T07:17:45.6284846Z Resolving deltas: 75% (12176/16205) 2024-06-26T07:17:45.6289371Z Resolving deltas: 78% (12717/16205) 2024-06-26T07:17:45.6299416Z Resolving deltas: 79% (12803/16205) 2024-06-26T07:17:45.6307438Z Resolving deltas: 80% (12966/16205) 2024-06-26T07:17:45.6318066Z Resolving deltas: 81% (13133/16205) 2024-06-26T07:17:45.6327370Z Resolving deltas: 82% (13292/16205) 2024-06-26T07:17:45.6336444Z Resolving deltas: 83% (13451/16205) 2024-06-26T07:17:45.6346776Z Resolving deltas: 84% (13622/16205) 2024-06-26T07:17:45.6367926Z Resolving deltas: 85% (13775/16205) 2024-06-26T07:17:45.6378076Z Resolving deltas: 86% (13937/16205) 2024-06-26T07:17:45.6385829Z Resolving deltas: 87% (14107/16205) 2024-06-26T07:17:45.6393536Z Resolving deltas: 88% (14281/16205) 2024-06-26T07:17:45.6401286Z Resolving deltas: 89% (14445/16205) 2024-06-26T07:17:45.6411180Z Resolving deltas: 90% (14598/16205) 2024-06-26T07:17:45.6418465Z Resolving deltas: 91% (14766/16205) 2024-06-26T07:17:45.6430279Z Resolving deltas: 92% (14917/16205) 2024-06-26T07:17:45.6434901Z Resolving deltas: 93% (15121/16205) 2024-06-26T07:17:45.6442665Z Resolving deltas: 94% (15244/16205) 2024-06-26T07:17:45.6450117Z Resolving deltas: 95% (15396/16205) 2024-06-26T07:17:45.6456571Z Resolving deltas: 96% (15557/16205) 2024-06-26T07:17:45.6461683Z Resolving deltas: 97% (15722/16205) 2024-06-26T07:17:45.6469408Z Resolving deltas: 98% (15886/16205) 2024-06-26T07:17:45.6503256Z Resolving deltas: 99% (16045/16205) 2024-06-26T07:17:45.6503767Z Resolving deltas: 100% (16205/16205) 2024-06-26T07:17:45.6504212Z Resolving deltas: 100% (16205/16205), done. 2024-06-26T07:17:46.7068322Z + pushd torchbench 2024-06-26T07:17:46.7068728Z ~/workspace/torchbench ~/workspace 2024-06-26T07:17:46.7069240Z + git checkout 23512dbebd44a11eb84afbf53c3c071dd105297e 2024-06-26T07:17:46.7644813Z Note: switching to '23512dbebd44a11eb84afbf53c3c071dd105297e'. 2024-06-26T07:17:46.7645323Z 2024-06-26T07:17:46.7645721Z You are in 'detached HEAD' state. You can look around, make experimental 2024-06-26T07:17:46.7646512Z changes and commit them, and you can discard any commits you make in this 2024-06-26T07:17:46.7647275Z state without impacting any branches by switching back to a branch. 2024-06-26T07:17:46.7647802Z 2024-06-26T07:17:46.7648113Z If you want to create a new branch to retain commits you create, you may 2024-06-26T07:17:46.7649037Z do so (now or later) by using -c with the switch command. Example: 2024-06-26T07:17:46.7649499Z 2024-06-26T07:17:46.7649695Z git switch -c 2024-06-26T07:17:46.7649985Z 2024-06-26T07:17:46.7650134Z Or undo this operation with: 2024-06-26T07:17:46.7650411Z 2024-06-26T07:17:46.7650526Z git switch - 2024-06-26T07:17:46.7650719Z 2024-06-26T07:17:46.7651079Z Turn off this advice by setting config variable advice.detachedHead to false 2024-06-26T07:17:46.7651612Z 2024-06-26T07:17:46.7651785Z HEAD is now at 23512dbe Pin numpy version 2024-06-26T07:17:46.7652248Z + '[' '' ']' 2024-06-26T07:17:46.7652625Z + python install.py --continue_on_fail 2024-06-26T07:17:48.7439118Z checking packages torch, torchvision, torchaudio are installed...OK 2024-06-26T07:18:04.4399383Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK 2024-06-26T07:18:06.8237768Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK 2024-06-26T07:18:09.4362741Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK 2024-06-26T07:18:11.8635847Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK 2024-06-26T07:18:11.8782055Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK 2024-06-26T07:18:20.5273356Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK 2024-06-26T07:18:23.9135370Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK 2024-06-26T07:18:27.3181586Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK 2024-06-26T07:18:30.6722701Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK 2024-06-26T07:18:30.6725662Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found 2024-06-26T07:18:32.4971296Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK 2024-06-26T07:18:36.0313055Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK 2024-06-26T07:18:36.0465570Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK 2024-06-26T07:19:30.7739189Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK 2024-06-26T07:19:39.7376457Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK 2024-06-26T07:19:47.3682195Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK 2024-06-26T07:19:54.5963615Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK 2024-06-26T07:20:03.3161345Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK 2024-06-26T07:20:10.8114031Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK 2024-06-26T07:20:17.9526673Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK 2024-06-26T07:20:25.4291654Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK 2024-06-26T07:20:33.0984441Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK 2024-06-26T07:20:40.6493922Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK 2024-06-26T07:20:48.0131500Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK 2024-06-26T07:20:55.5257969Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK 2024-06-26T07:20:59.5506933Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK 2024-06-26T07:21:40.0267233Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK 2024-06-26T07:21:46.0697984Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK 2024-06-26T07:21:52.7758093Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK 2024-06-26T07:22:03.7116997Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK 2024-06-26T07:22:05.9377649Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK 2024-06-26T07:22:08.4283281Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK 2024-06-26T07:22:16.3990022Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK 2024-06-26T07:22:24.4966486Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK 2024-06-26T07:22:31.2262154Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK 2024-06-26T07:22:39.8645868Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK 2024-06-26T07:22:46.9636788Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK 2024-06-26T07:22:53.2775017Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK 2024-06-26T07:23:00.6117845Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK 2024-06-26T07:23:16.8949042Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK 2024-06-26T07:23:24.5490570Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK 2024-06-26T07:23:29.9150896Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK 2024-06-26T07:23:36.0141564Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK 2024-06-26T07:23:43.9250206Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK 2024-06-26T07:23:43.9256257Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found 2024-06-26T07:23:57.2695214Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK 2024-06-26T07:24:02.5878528Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK 2024-06-26T07:24:02.5881263Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found 2024-06-26T07:24:11.3391390Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK 2024-06-26T07:24:13.6318684Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK 2024-06-26T07:24:15.9558065Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK 2024-06-26T07:24:55.6257679Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK 2024-06-26T07:26:25.7758805Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK 2024-06-26T07:26:25.7915797Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK 2024-06-26T07:26:28.3443308Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK 2024-06-26T07:26:28.3600294Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK 2024-06-26T07:26:28.3756466Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK 2024-06-26T07:26:28.3906064Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK 2024-06-26T07:26:28.4061388Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK 2024-06-26T07:26:28.4215177Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK 2024-06-26T07:26:48.8165352Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK 2024-06-26T07:26:48.8168849Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found 2024-06-26T07:26:51.1362928Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK 2024-06-26T07:26:54.4801962Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK 2024-06-26T07:26:54.4955608Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK 2024-06-26T07:26:54.5105743Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK 2024-06-26T07:26:54.5263422Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK 2024-06-26T07:26:54.5413972Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK 2024-06-26T07:26:54.5575029Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK 2024-06-26T07:27:01.8124289Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK 2024-06-26T07:27:04.3228544Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK 2024-06-26T07:27:08.9516244Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK 2024-06-26T07:27:08.9675232Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK 2024-06-26T07:27:08.9828392Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK 2024-06-26T07:27:08.9989638Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK 2024-06-26T07:27:09.0141408Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK 2024-06-26T07:27:09.0302842Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK 2024-06-26T07:27:20.9760859Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK 2024-06-26T07:27:32.8568350Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK 2024-06-26T07:27:32.8721400Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK 2024-06-26T07:27:32.8723997Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found 2024-06-26T07:27:32.8725932Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found 2024-06-26T07:27:36.6335011Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK 2024-06-26T07:27:39.5539665Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK 2024-06-26T07:27:39.5691123Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK 2024-06-26T07:27:59.4252437Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK 2024-06-26T07:28:06.3041547Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK 2024-06-26T07:28:10.6422902Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK 2024-06-26T07:28:18.6584129Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK 2024-06-26T07:28:18.6745368Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK 2024-06-26T07:28:18.6903765Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK 2024-06-26T07:28:18.7062398Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK 2024-06-26T07:28:18.7216128Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK 2024-06-26T07:28:18.7367336Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK 2024-06-26T07:28:18.7517812Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK 2024-06-26T07:28:18.7670318Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK 2024-06-26T07:28:30.5170636Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK 2024-06-26T07:28:43.4767268Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK 2024-06-26T07:28:43.4928833Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK 2024-06-26T07:28:46.4862764Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK 2024-06-26T07:28:49.6849528Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK 2024-06-26T07:28:50.1026698Z + popd 2024-06-26T07:28:50.1027097Z ~/workspace 2024-06-26T07:28:50.1027544Z + [[ aot_eager_torchbench != *cpu_inductor* ]] 2024-06-26T07:28:50.1031944Z + install_torchrec_and_fbgemm 2024-06-26T07:28:50.1032625Z + local torchrec_commit 2024-06-26T07:28:50.1033090Z ++ get_pinned_commit torchrec 2024-06-26T07:28:50.1033498Z ++ cat .github/ci_commit_pins/torchrec.txt 2024-06-26T07:28:50.1080655Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T07:28:50.1081200Z + local fbgemm_commit 2024-06-26T07:28:50.1083937Z ++ get_pinned_commit fbgemm 2024-06-26T07:28:50.1084366Z ++ cat .github/ci_commit_pins/fbgemm.txt 2024-06-26T07:28:50.1093150Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:28:50.1093951Z + pip_uninstall torchrec-nightly 2024-06-26T07:28:50.1094414Z + pip uninstall -y torchrec-nightly 2024-06-26T07:28:50.4670537Z WARNING: Skipping torchrec-nightly as it is not installed. 2024-06-26T07:28:50.5083893Z + pip_uninstall fbgemm-gpu-nightly 2024-06-26T07:28:50.5084775Z + pip uninstall -y fbgemm-gpu-nightly 2024-06-26T07:28:50.8564587Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed. 2024-06-26T07:28:50.8983733Z + pip_install setuptools-git-versioning scikit-build pyre-extensions 2024-06-26T07:28:50.8985645Z + pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions 2024-06-26T07:28:51.3406135Z Collecting setuptools-git-versioning 2024-06-26T07:28:51.3555492Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl.metadata (5.8 kB) 2024-06-26T07:28:51.4503571Z Collecting scikit-build 2024-06-26T07:28:51.4533571Z Downloading scikit_build-0.18.0-py3-none-any.whl.metadata (17 kB) 2024-06-26T07:28:51.5508247Z Collecting pyre-extensions 2024-06-26T07:28:51.5539422Z Downloading pyre_extensions-0.0.30-py3-none-any.whl.metadata (4.0 kB) 2024-06-26T07:28:51.6024657Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.1) 2024-06-26T07:28:51.6026864Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (69.5.1) 2024-06-26T07:28:51.6586446Z Collecting toml>=0.10.2 (from setuptools-git-versioning) 2024-06-26T07:28:51.6614043Z Downloading toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB) 2024-06-26T07:28:51.7202168Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.9.0) 2024-06-26T07:28:51.7212894Z Requirement already satisfied: tomli in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (2.0.1) 2024-06-26T07:28:51.7217870Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.43.0) 2024-06-26T07:28:51.7728890Z Collecting typing-inspect (from pyre-extensions) 2024-06-26T07:28:51.7756109Z Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB) 2024-06-26T07:28:51.8240233Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.12.2) 2024-06-26T07:28:51.8471660Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0) 2024-06-26T07:28:51.8511833Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl (10 kB) 2024-06-26T07:28:51.9095046Z Downloading scikit_build-0.18.0-py3-none-any.whl (85 kB) 2024-06-26T07:28:51.9697199Z Downloading pyre_extensions-0.0.30-py3-none-any.whl (12 kB) 2024-06-26T07:28:52.0238452Z Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB) 2024-06-26T07:28:52.0824027Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2024-06-26T07:28:54.2790423Z Installing collected packages: typing-inspect, toml, scikit-build, setuptools-git-versioning, pyre-extensions 2024-06-26T07:28:54.6137062Z Successfully installed pyre-extensions-0.0.30 scikit-build-0.18.0 setuptools-git-versioning-2.0.0 toml-0.10.2 typing-inspect-0.9.0 2024-06-26T07:28:54.7311994Z + CUDA_PATH=/usr/local/cuda-12.1 2024-06-26T07:28:54.7313467Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-06-26T07:28:54.7315320Z + pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-06-26T07:28:55.0699526Z Collecting fbgemm-gpu 2024-06-26T07:28:55.0701778Z Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-b7hxev3h/fbgemm-gpu_4f3ee537b13e4b94a1ad0c262de097ba 2024-06-26T07:28:55.0718331Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-b7hxev3h/fbgemm-gpu_4f3ee537b13e4b94a1ad0c262de097ba 2024-06-26T07:28:56.0758167Z Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd' 2024-06-26T07:28:56.0771949Z Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:28:56.4742065Z Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:28:56.9824929Z Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:28:56.9826226Z Running command git submodule update --init --recursive -q 2024-06-26T07:29:04.2575046Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:29:04.2595830Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.21.2) 2024-06-26T07:29:04.2602996Z Building wheels for collected packages: fbgemm-gpu 2024-06-26T08:11:16.3214708Z Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-06-26T08:11:16.5016334Z [?25h Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=252634518 sha256=7906b1bfe637c579254d565de738a45549b8b5edbe84f9bbe82f2d727762fad8 2024-06-26T08:11:16.5018307Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39 2024-06-26T08:11:16.5036883Z Successfully built fbgemm-gpu 2024-06-26T08:11:18.6985524Z Installing collected packages: fbgemm-gpu 2024-06-26T08:11:21.8550224Z Successfully installed fbgemm-gpu-0.4.1rc0.post421 2024-06-26T08:11:22.4743725Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:11:22.4745258Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:11:22.8143825Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:11:22.8146496Z Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-61heia4h 2024-06-26T08:11:22.8162722Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-61heia4h 2024-06-26T08:11:23.4117807Z Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2' 2024-06-26T08:11:23.4132019Z Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:11:23.6840268Z Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:11:23.9854429Z Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:11:24.2151317Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T08:11:24.2207742Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9) 2024-06-26T08:11:24.2211217Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.0.3) 2024-06-26T08:11:24.2215542Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0) 2024-06-26T08:11:24.2713963Z Collecting torchmetrics (from torchrec==0.3.2) 2024-06-26T08:11:24.2888812Z Downloading torchmetrics-1.4.0.post0-py3-none-any.whl.metadata (19 kB) 2024-06-26T08:11:24.2956428Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.66.4) 2024-06-26T08:11:24.2959555Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421) 2024-06-26T08:11:24.2972496Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.21.2) 2024-06-26T08:11:24.2994028Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (2.10.0) 2024-06-26T08:11:24.3657537Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0) 2024-06-26T08:11:24.3661928Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-06-26T08:11:24.3666396Z Requirement already satisfied: tzdata>=2022.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-06-26T08:11:24.4441379Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.1) 2024-06-26T08:11:24.4446023Z Requirement already satisfied: torch>=1.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.5.0a0+gitb8c4c54) 2024-06-26T08:11:24.4635443Z Collecting lightning-utilities>=0.8.0 (from torchmetrics->torchrec==0.3.2) 2024-06-26T08:11:24.4666031Z Downloading lightning_utilities-0.11.2-py3-none-any.whl.metadata (4.7 kB) 2024-06-26T08:11:24.4837187Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (69.5.1) 2024-06-26T08:11:24.4840924Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.12.2) 2024-06-26T08:11:24.4872769Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.16.0) 2024-06-26T08:11:24.4944931Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.13.1) 2024-06-26T08:11:24.4946828Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.12.1) 2024-06-26T08:11:24.4950035Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.8.8) 2024-06-26T08:11:24.4952620Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.1.4) 2024-06-26T08:11:24.4955692Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2024.2.0) 2024-06-26T08:11:24.5621171Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.1.5) 2024-06-26T08:11:24.5929297Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.3.0) 2024-06-26T08:11:24.6064844Z Downloading torchmetrics-1.4.0.post0-py3-none-any.whl (868 kB) 2024-06-26T08:11:24.6248478Z Downloading lightning_utilities-0.11.2-py3-none-any.whl (26 kB) 2024-06-26T08:11:24.6344978Z Building wheels for collected packages: torchrec 2024-06-26T08:11:25.0688955Z Building wheel for torchrec (setup.py) ... [?25l- \ | done 2024-06-26T08:11:25.0697872Z [?25h Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374488 sha256=f783c441273f812a837472b6d68b9a18730c1e6c56fa82b7b7b43578ede87002 2024-06-26T08:11:25.0699652Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8 2024-06-26T08:11:25.0721795Z Successfully built torchrec 2024-06-26T08:11:27.1662801Z Installing collected packages: lightning-utilities, torchmetrics, torchrec 2024-06-26T08:11:27.7343902Z Successfully installed lightning-utilities-0.11.2 torchmetrics-1.4.0.post0 torchrec-0.3.2 2024-06-26T08:11:27.8742578Z ++ pwd 2024-06-26T08:11:27.8743789Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench 2024-06-26T08:11:27.8744821Z + test_dynamo_benchmark torchbench 0 2024-06-26T08:11:27.8747944Z ++ pwd 2024-06-26T08:11:27.8749137Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:11:27.8749785Z + local suite=torchbench 2024-06-26T08:11:27.8750270Z + shift 2024-06-26T08:11:27.8750541Z + local shard_id=0 2024-06-26T08:11:27.8750845Z + shift 2024-06-26T08:11:27.8751174Z + [[ aot_eager_torchbench == *perf_compare* ]] 2024-06-26T08:11:27.8751675Z + [[ aot_eager_torchbench == *perf* ]] 2024-06-26T08:11:27.8753056Z + [[ aot_eager_torchbench == *cpu_inductor* ]] 2024-06-26T08:11:27.8753735Z + [[ aot_eager_torchbench == *aot_inductor* ]] 2024-06-26T08:11:27.8754695Z + test_single_dynamo_benchmark inference torchbench 0 --inference --bfloat16 2024-06-26T08:11:27.8755514Z ++ pwd 2024-06-26T08:11:27.8756171Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:11:27.8757108Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:11:27.8782459Z + local name=inference 2024-06-26T08:11:27.8782949Z + shift 2024-06-26T08:11:27.8783328Z + local suite=torchbench 2024-06-26T08:11:27.8783804Z + shift 2024-06-26T08:11:27.8784303Z + local shard_id=0 2024-06-26T08:11:27.8784644Z + shift 2024-06-26T08:11:27.8785016Z + partition_flags=() 2024-06-26T08:11:27.8785428Z + local partition_flags 2024-06-26T08:11:27.8785963Z + [[ -n 2 ]] 2024-06-26T08:11:27.8786355Z + [[ -n 0 ]] 2024-06-26T08:11:27.8786935Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-06-26T08:11:27.8787640Z + [[ aot_eager_torchbench == *perf_compare* ]] 2024-06-26T08:11:27.8788110Z + [[ aot_eager_torchbench == *perf* ]] 2024-06-26T08:11:27.8788564Z + [[ aot_eager_torchbench == *aot_inductor* ]] 2024-06-26T08:11:27.8790315Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --backend aot_eager --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv 2024-06-26T08:11:33.5301667Z 2024-06-26T08:11:34.0971690Z loading model: 0it [00:00, ?it/s]/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/optim/apply_optimizer_in_backward.py:47: DeprecationWarning: `TorchScript` support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the `torch.compile` optimizer instead. 2024-06-26T08:11:34.0973772Z from torch.distributed.optim import _apply_optimizer_in_backward 2024-06-26T08:11:34.2873978Z 2024-06-26T08:11:34.2874455Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:11:34.2874933Z cuda eval torchrec_dlrm 2024-06-26T08:11:34.2885859Z Traceback (most recent call last): 2024-06-26T08:11:34.2886594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:11:34.2887410Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:11:34.2888185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 447, in forward_pass 2024-06-26T08:11:34.2888940Z return mod(*inputs) 2024-06-26T08:11:34.2889874Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:11:34.2890755Z return self._call_impl(*args, **kwargs) 2024-06-26T08:11:34.2891671Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:11:34.2892511Z return forward_call(*args, **kwargs) 2024-06-26T08:11:34.2893379Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 893, in forward 2024-06-26T08:11:34.2894282Z logits = self.model(batch.dense_features, batch.sparse_features) 2024-06-26T08:11:34.2895383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:11:34.2896284Z return self._call_impl(*args, **kwargs) 2024-06-26T08:11:34.2897225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:11:34.2898071Z return forward_call(*args, **kwargs) 2024-06-26T08:11:34.2898932Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 572, in forward 2024-06-26T08:11:34.2899721Z concatenated_dense = self.inter_arch( 2024-06-26T08:11:34.2900681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:11:34.2901558Z return self._call_impl(*args, **kwargs) 2024-06-26T08:11:34.2902469Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:11:34.2903288Z return forward_call(*args, **kwargs) 2024-06-26T08:11:34.2904134Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 284, in forward 2024-06-26T08:11:34.2905039Z return self.crossnet(combined_values.reshape([B, -1])) 2024-06-26T08:11:34.2906069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:11:34.2907116Z return self._call_impl(*args, **kwargs) 2024-06-26T08:11:34.2908036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:11:34.2908860Z return forward_call(*args, **kwargs) 2024-06-26T08:11:34.2909738Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/crossnet.py", line 175, in forward 2024-06-26T08:11:34.2910792Z x_l_v = torch.nn.functional.linear(x_l, self.V_kernels[layer]) 2024-06-26T08:11:34.2911636Z RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::BFloat16 2024-06-26T08:11:34.2912197Z 2024-06-26T08:11:34.2912662Z The above exception was the direct cause of the following exception: 2024-06-26T08:11:34.2913116Z 2024-06-26T08:11:34.2913258Z Traceback (most recent call last): 2024-06-26T08:11:34.2913904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:11:34.2914562Z ) = runner.load_model( 2024-06-26T08:11:34.2915227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:11:34.2915983Z self.validate_model(model, example_inputs) 2024-06-26T08:11:34.2916741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:11:34.2917595Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:11:34.2918066Z RuntimeError: Eager run failed 2024-06-26T08:11:34.2918317Z 2024-06-26T08:11:34.2918432Z eager_fail_to_run 2024-06-26T08:11:37.6218008Z 2024-06-26T08:11:39.5892762Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:11:39.5893352Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:11:39.5959080Z cuda eval BERT_pytorch 2024-06-26T08:11:47.0281082Z pass 2024-06-26T08:11:47.0283625Z TIMING: backend_compile:3.87248 entire_frame_compile:6.62031 2024-06-26T08:11:47.0284762Z STATS: call_* op count: 1076 | FakeTensor.__torch_dispatch__:897 | FakeTensorMode.__torch_dispatch__:25610 | ProxyTorchDispatchMode.__torch_dispatch__:9196 2024-06-26T08:11:47.0285911Z Dynamo produced 1 graphs covering 1076 ops with 0 graph breaks (0 unique) 2024-06-26T08:11:50.4144957Z 2024-06-26T08:11:52.9931939Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:11:52.9932597Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:11:52.9983048Z cuda eval Background_Matting 2024-06-26T08:11:52.9986327Z pass_due_to_skip 2024-06-26T08:11:53.0525389Z TIMING: 2024-06-26T08:11:53.0526019Z STATS: call_* op count: 0 2024-06-26T08:11:53.0526596Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T08:11:56.3319468Z 2024-06-26T08:11:58.9995014Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:11:58.9995500Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:11:59.0019135Z cuda eval LearningToPaint 2024-06-26T08:12:01.3316974Z pass 2024-06-26T08:12:01.3317733Z TIMING: backend_compile:1.1877 entire_frame_compile:1.83766 2024-06-26T08:12:01.3319421Z STATS: call_* op count: 142 | FakeTensor.__torch_dispatch__:408 | FakeTensorMode.__torch_dispatch__:8715 | ProxyTorchDispatchMode.__torch_dispatch__:1795 2024-06-26T08:12:01.3321187Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-06-26T08:12:04.8317660Z 2024-06-26T08:12:06.0774137Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/vgg16-397923af.pth 2024-06-26T08:12:06.0949533Z 2024-06-26T08:12:06.0949855Z 2024-06-26T08:12:06.1953262Z 0% 0.00/528M [00:00 will be ignored 2024-06-26T08:29:39.5157801Z pass 2024-06-26T08:29:39.5586876Z TIMING: entire_frame_compile:17.52823 backend_compile:10.71544 2024-06-26T08:29:39.5588037Z STATS: call_* op count: 672 | FakeTensorMode.__torch_dispatch__:38333 | FakeTensor.__torch_dispatch__:2683 | ProxyTorchDispatchMode.__torch_dispatch__:13583 2024-06-26T08:29:39.5589189Z Dynamo produced 3 graphs covering 672 ops with 7 graph breaks (5 unique) 2024-06-26T08:29:43.2062251Z 2024-06-26T08:29:44.4016119Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:29:44.4016706Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:29:44.4017140Z cuda train alexnet 2024-06-26T08:29:46.4812449Z pass 2024-06-26T08:29:46.5024466Z TIMING: entire_frame_compile:0.75274 backend_compile:0.48514 2024-06-26T08:29:46.5026046Z STATS: call_* op count: 26 | FakeTensor.__torch_dispatch__:131 | FakeTensorMode.__torch_dispatch__:1991 | ProxyTorchDispatchMode.__torch_dispatch__:674 2024-06-26T08:29:46.5027669Z Dynamo produced 2 graphs covering 26 ops with 6 graph breaks (5 unique) 2024-06-26T08:29:49.8658941Z 2024-06-26T08:29:51.5872964Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:29:51.5873511Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:29:51.5873953Z cuda train basic_gnn_edgecnn 2024-06-26T08:29:54.2492101Z W0626 08:29:54.248000 140271646519936 torch/_logging/_internal.py:1040] [24/0] Profiler function will be ignored 2024-06-26T08:29:55.4267364Z pass 2024-06-26T08:29:55.4435272Z TIMING: entire_frame_compile:2.91977 backend_compile:1.19591 2024-06-26T08:29:55.4436346Z STATS: call_* op count: 103 | FakeTensorMode.__torch_dispatch__:4597 | FakeTensor.__torch_dispatch__:406 | ProxyTorchDispatchMode.__torch_dispatch__:1462 2024-06-26T08:29:55.4437692Z Dynamo produced 8 graphs covering 103 ops with 20 graph breaks (6 unique) 2024-06-26T08:29:58.9770276Z 2024-06-26T08:30:00.7400819Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:30:00.7401432Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:30:00.7402553Z cuda train basic_gnn_gcn 2024-06-26T08:30:02.3545896Z W0626 08:30:02.353000 140580774220416 torch/_logging/_internal.py:1040] [18/0] Profiler function will be ignored 2024-06-26T08:30:03.0265109Z pass 2024-06-26T08:30:03.0378620Z TIMING: entire_frame_compile:1.69563 backend_compile:0.77891 2024-06-26T08:30:03.0381049Z STATS: call_* op count: 99 | FakeTensorMode.__torch_dispatch__:2705 | FakeTensor.__torch_dispatch__:195 | ProxyTorchDispatchMode.__torch_dispatch__:866 2024-06-26T08:30:03.0383255Z Dynamo produced 10 graphs covering 99 ops with 13 graph breaks (6 unique) 2024-06-26T08:30:06.5721561Z 2024-06-26T08:30:08.2910358Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:30:08.2910891Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:30:08.2911408Z cuda train basic_gnn_gin 2024-06-26T08:30:09.8856482Z W0626 08:30:09.884000 139856933409408 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:30:11.0507839Z pass 2024-06-26T08:30:11.0604923Z TIMING: entire_frame_compile:2.1985 backend_compile:1.09349 2024-06-26T08:30:11.0606283Z STATS: call_* op count: 113 | FakeTensorMode.__torch_dispatch__:3951 | FakeTensor.__torch_dispatch__:259 | ProxyTorchDispatchMode.__torch_dispatch__:1395 2024-06-26T08:30:11.0607515Z Dynamo produced 3 graphs covering 113 ops with 7 graph breaks (5 unique) 2024-06-26T08:30:14.5510864Z 2024-06-26T08:30:16.3124461Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:30:16.3125084Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:30:16.3125523Z cuda train basic_gnn_sage 2024-06-26T08:30:17.7464210Z W0626 08:30:17.745000 139802532352640 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:30:18.6838953Z pass 2024-06-26T08:30:18.6903924Z TIMING: entire_frame_compile:1.8247 backend_compile:0.88541 2024-06-26T08:30:18.6905142Z STATS: call_* op count: 107 | FakeTensorMode.__torch_dispatch__:3189 | FakeTensor.__torch_dispatch__:202 | ProxyTorchDispatchMode.__torch_dispatch__:1102 2024-06-26T08:30:18.6906401Z Dynamo produced 3 graphs covering 107 ops with 7 graph breaks (5 unique) 2024-06-26T08:30:22.1884291Z 2024-06-26T08:30:22.6769928Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:30:22.6770394Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:30:22.6770845Z cuda train dcgan 2024-06-26T08:30:23.6625640Z pass 2024-06-26T08:30:23.6642826Z TIMING: entire_frame_compile:0.71866 backend_compile:0.4494 2024-06-26T08:30:23.6643901Z STATS: call_* op count: 17 | FakeTensor.__torch_dispatch__:103 | FakeTensorMode.__torch_dispatch__:1846 | ProxyTorchDispatchMode.__torch_dispatch__:439 2024-06-26T08:30:23.6645032Z Dynamo produced 2 graphs covering 17 ops with 6 graph breaks (5 unique) 2024-06-26T08:30:26.9776804Z 2024-06-26T08:30:30.3509716Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:30:30.3510314Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:30:30.3510744Z cuda train demucs 2024-06-26T08:31:27.0999804Z pass 2024-06-26T08:31:27.1249547Z TIMING: entire_frame_compile:2.09228 backend_compile:1.30249 2024-06-26T08:31:27.1251996Z STATS: call_* op count: 88 | FakeTensorMode.__torch_dispatch__:5932 | FakeTensor.__torch_dispatch__:441 | ProxyTorchDispatchMode.__torch_dispatch__:1789 2024-06-26T08:31:27.1254234Z Dynamo produced 6 graphs covering 88 ops with 9 graph breaks (6 unique) 2024-06-26T08:31:30.5150656Z 2024-06-26T08:31:32.6068759Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:31:32.6069221Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:31:32.6069661Z cuda train densenet121 2024-06-26T08:31:55.8625037Z pass 2024-06-26T08:31:55.8741048Z TIMING: entire_frame_compile:17.45658 backend_compile:13.01303 2024-06-26T08:31:55.8742150Z STATS: call_* op count: 435 | FakeTensor.__torch_dispatch__:2272 | FakeTensorMode.__torch_dispatch__:65412 | ProxyTorchDispatchMode.__torch_dispatch__:18924 2024-06-26T08:31:55.8743502Z Dynamo produced 2 graphs covering 435 ops with 6 graph breaks (5 unique) 2024-06-26T08:31:59.7399498Z 2024-06-26T08:32:04.2220555Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:04.2221026Z loading model: 0it [00:04, ?it/s] 2024-06-26T08:32:04.2221465Z cuda train detectron2_maskrcnn_r_50_c4 2024-06-26T08:32:04.2233667Z Traceback (most recent call last): 2024-06-26T08:32:04.2234662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:32:04.2235617Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:32:04.2236788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in forward_and_backward_pass 2024-06-26T08:32:04.2237885Z pred = mod(*cloned_inputs) 2024-06-26T08:32:04.2239378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:32:04.2240390Z return self._call_impl(*args, **kwargs) 2024-06-26T08:32:04.2241344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:32:04.2242648Z return forward_call(*args, **kwargs) 2024-06-26T08:32:04.2243607Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-06-26T08:32:04.2244682Z return self.inference(batched_inputs) 2024-06-26T08:32:04.2245792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference 2024-06-26T08:32:04.2246782Z results, _ = self.roi_heads(images, features, proposals, None) 2024-06-26T08:32:04.2247871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:32:04.2248812Z return self._call_impl(*args, **kwargs) 2024-06-26T08:32:04.2249731Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:32:04.2250567Z return forward_call(*args, **kwargs) 2024-06-26T08:32:04.2251543Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward 2024-06-26T08:32:04.2252460Z box_features = self._shared_roi_transform( 2024-06-26T08:32:04.2253542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform 2024-06-26T08:32:04.2254497Z x = self.pooler(features, boxes) 2024-06-26T08:32:04.2255451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:32:04.2256353Z return self._call_impl(*args, **kwargs) 2024-06-26T08:32:04.2257279Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:32:04.2258128Z return forward_call(*args, **kwargs) 2024-06-26T08:32:04.2259019Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward 2024-06-26T08:32:04.2259923Z return self.level_poolers[0](x[0], pooler_fmt_boxes) 2024-06-26T08:32:04.2260958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:32:04.2261851Z return self._call_impl(*args, **kwargs) 2024-06-26T08:32:04.2262765Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:32:04.2263599Z return forward_call(*args, **kwargs) 2024-06-26T08:32:04.2264634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward 2024-06-26T08:32:04.2265437Z return roi_align( 2024-06-26T08:32:04.2266247Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-06-26T08:32:04.2267365Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-06-26T08:32:04.2268559Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-06-26T08:32:04.2269645Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-06-26T08:32:04.2271033Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-06-26T08:32:04.2271908Z v1 = masked_index(y_low, x_low) 2024-06-26T08:32:04.2273096Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-06-26T08:32:04.2273925Z return input[ 2024-06-26T08:32:04.2276989Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 21890.87 GiB. GPU 0 has a total capacity of 21.98 GiB of which 16.54 GiB is free. Process 18626 has 5.42 GiB memory in use. Of the allocated memory 4.58 GiB is allocated by PyTorch, and 558.89 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-06-26T08:32:04.2279856Z 2024-06-26T08:32:04.2280153Z The above exception was the direct cause of the following exception: 2024-06-26T08:32:04.2280610Z 2024-06-26T08:32:04.2280757Z Traceback (most recent call last): 2024-06-26T08:32:04.2281403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:32:04.2282043Z ) = runner.load_model( 2024-06-26T08:32:04.2282703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:32:04.2283456Z self.validate_model(model, example_inputs) 2024-06-26T08:32:04.2284202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:32:04.2284953Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:32:04.2285422Z RuntimeError: Eager run failed 2024-06-26T08:32:04.2285678Z 2024-06-26T08:32:04.2285792Z eager_fail_to_run 2024-06-26T08:32:07.5207703Z 2024-06-26T08:32:15.7025689Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:15.7026400Z loading model: 0it [00:08, ?it/s] 2024-06-26T08:32:15.7027074Z cuda train dlrm 2024-06-26T08:32:17.4660904Z pass 2024-06-26T08:32:17.5070093Z TIMING: entire_frame_compile:1.11416 backend_compile:0.71348 2024-06-26T08:32:17.5071744Z STATS: call_* op count: 40 | FakeTensor.__torch_dispatch__:151 | FakeTensorMode.__torch_dispatch__:3115 | ProxyTorchDispatchMode.__torch_dispatch__:1110 2024-06-26T08:32:17.5073748Z Dynamo produced 2 graphs covering 40 ops with 6 graph breaks (5 unique) 2024-06-26T08:32:20.8488790Z 2024-06-26T08:32:22.2078189Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:22.2078660Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:32:22.2079094Z cuda train drq 2024-06-26T08:32:23.7282021Z W0626 08:32:23.727000 140382940795520 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:32:26.4391539Z pass 2024-06-26T08:32:26.4418484Z TIMING: entire_frame_compile:3.94223 backend_compile:2.06317 2024-06-26T08:32:26.4419586Z STATS: call_* op count: 137 | FakeTensorMode.__torch_dispatch__:7382 | FakeTensor.__torch_dispatch__:419 | ProxyTorchDispatchMode.__torch_dispatch__:2562 2024-06-26T08:32:26.4420704Z Dynamo produced 5 graphs covering 137 ops with 7 graph breaks (5 unique) 2024-06-26T08:32:29.9288963Z 2024-06-26T08:32:31.5081043Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:31.5081534Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:32:31.5083435Z cuda train fastNLP_Bert 2024-06-26T08:32:37.2918306Z W0626 08:32:37.291000 139792608031360 torch/_dynamo/variables/tensor.py:714] [3/0] Graph break from `Tensor.item()`, consider setting: 2024-06-26T08:32:37.2920050Z W0626 08:32:37.291000 139792608031360 torch/_dynamo/variables/tensor.py:714] [3/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-26T08:32:37.2921150Z W0626 08:32:37.291000 139792608031360 torch/_dynamo/variables/tensor.py:714] [3/0] or: 2024-06-26T08:32:37.2922209Z W0626 08:32:37.291000 139792608031360 torch/_dynamo/variables/tensor.py:714] [3/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-26T08:32:37.2923480Z W0626 08:32:37.291000 139792608031360 torch/_dynamo/variables/tensor.py:714] [3/0] to include these operations in the captured graph. 2024-06-26T08:32:37.2924554Z W0626 08:32:37.291000 139792608031360 torch/_dynamo/variables/tensor.py:714] [3/0] 2024-06-26T08:32:47.6223290Z pass 2024-06-26T08:32:47.7096166Z TIMING: entire_frame_compile:10.00958 backend_compile:7.02367 2024-06-26T08:32:47.7098637Z STATS: call_* op count: 448 | FakeTensorMode.__torch_dispatch__:29623 | ProxyTorchDispatchMode.__torch_dispatch__:13734 | FakeTensor.__torch_dispatch__:648 2024-06-26T08:32:47.7101246Z Dynamo produced 7 graphs covering 448 ops with 10 graph breaks (6 unique) 2024-06-26T08:32:51.3347604Z 2024-06-26T08:32:52.0958390Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:52.0958935Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:52.0959393Z cuda train functorch_dp_cifar10 2024-06-26T08:32:55.1560771Z W0626 08:32:55.155000 140711634166400 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:33:02.1566936Z pass 2024-06-26T08:33:02.1671736Z TIMING: entire_frame_compile:9.47069 backend_compile:5.72717 2024-06-26T08:33:02.1673266Z STATS: call_* op count: 277 | FakeTensor.__torch_dispatch__:1658 | FakeTensorMode.__torch_dispatch__:18091 | ProxyTorchDispatchMode.__torch_dispatch__:5405 2024-06-26T08:33:02.1674437Z Dynamo produced 3 graphs covering 277 ops with 7 graph breaks (5 unique) 2024-06-26T08:33:05.6744280Z 2024-06-26T08:33:06.1925179Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:33:06.1925671Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:33:06.1926118Z cuda train functorch_maml_omniglot 2024-06-26T08:33:07.0673659Z W0626 08:33:07.066000 140343842300544 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:33:08.5641437Z pass 2024-06-26T08:33:08.5641995Z TIMING: entire_frame_compile:2.0788 backend_compile:1.24081 2024-06-26T08:33:08.5643184Z STATS: call_* op count: 78 | FakeTensor.__torch_dispatch__:398 | FakeTensorMode.__torch_dispatch__:3886 | ProxyTorchDispatchMode.__torch_dispatch__:1104 2024-06-26T08:33:08.5644318Z Dynamo produced 3 graphs covering 78 ops with 7 graph breaks (5 unique) 2024-06-26T08:33:11.8945379Z 2024-06-26T08:33:13.1313051Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:33:13.1313657Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:33:13.1314194Z cuda train hf_Albert 2024-06-26T08:33:28.3219450Z W0626 08:33:28.320000 140549303030400 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:33:31.0867358Z pass 2024-06-26T08:33:31.1415836Z TIMING: entire_frame_compile:11.12513 backend_compile:7.4774 2024-06-26T08:33:31.1418086Z STATS: call_* op count: 542 | FakeTensorMode.__torch_dispatch__:36963 | FakeTensor.__torch_dispatch__:1841 | ProxyTorchDispatchMode.__torch_dispatch__:15170 2024-06-26T08:33:31.1420341Z Dynamo produced 3 graphs covering 542 ops with 6 graph breaks (5 unique) 2024-06-26T08:33:34.6858914Z 2024-06-26T08:33:38.3507920Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:33:38.3508400Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:33:38.3509223Z cuda train hf_Bart 2024-06-26T08:33:57.7989491Z W0626 08:33:57.798000 140559862469248 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:34:23.3426580Z pass 2024-06-26T08:34:23.3980181Z TIMING: entire_frame_compile:34.90355 backend_compile:22.16456 2024-06-26T08:34:23.3981340Z STATS: call_* op count: 1250 | FakeTensorMode.__torch_dispatch__:75544 | FakeTensor.__torch_dispatch__:6773 | ProxyTorchDispatchMode.__torch_dispatch__:26116 2024-06-26T08:34:23.3982455Z Dynamo produced 3 graphs covering 1250 ops with 6 graph breaks (5 unique) 2024-06-26T08:34:27.5654327Z 2024-06-26T08:34:29.9473411Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:34:29.9473906Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:34:29.9474343Z cuda train hf_Bert 2024-06-26T08:34:48.7992807Z W0626 08:34:48.798000 139971366564480 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:35:08.6309190Z pass 2024-06-26T08:35:08.6777692Z TIMING: entire_frame_compile:30.677 backend_compile:18.46211 2024-06-26T08:35:08.6780237Z STATS: call_* op count: 995 | FakeTensorMode.__torch_dispatch__:61040 | FakeTensor.__torch_dispatch__:5120 | ProxyTorchDispatchMode.__torch_dispatch__:21388 2024-06-26T08:35:08.6781807Z Dynamo produced 3 graphs covering 995 ops with 6 graph breaks (5 unique) 2024-06-26T08:35:12.6742872Z 2024-06-26T08:35:17.1313164Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:35:17.1313640Z loading model: 0it [00:04, ?it/s] 2024-06-26T08:35:17.1314079Z cuda train hf_Bert_large 2024-06-26T08:35:58.5075397Z W0626 08:35:58.506000 140583885546112 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:36:37.0779091Z pass 2024-06-26T08:36:37.2475319Z TIMING: entire_frame_compile:59.81785 backend_compile:35.45362 2024-06-26T08:36:37.2477578Z STATS: call_* op count: 1919 | FakeTensorMode.__torch_dispatch__:119012 | FakeTensor.__torch_dispatch__:9920 | ProxyTorchDispatchMode.__torch_dispatch__:41896 2024-06-26T08:36:37.2478762Z Dynamo produced 3 graphs covering 1919 ops with 6 graph breaks (5 unique) 2024-06-26T08:36:42.0141278Z 2024-06-26T08:36:45.3322799Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:36:45.3323281Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:36:45.3323732Z cuda train hf_BigBird 2024-06-26T08:36:56.5916431Z WARNING:common:fp64 golden ref were not generated for hf_BigBird. Setting accuracy check to cosine 2024-06-26T08:37:41.4271821Z W0626 08:37:41.426000 140543145132672 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:38:02.2186807Z pass 2024-06-26T08:38:02.2634476Z TIMING: entire_frame_compile:61.11939 backend_compile:41.78847 2024-06-26T08:38:02.2635778Z STATS: call_* op count: 3975 | FakeTensorMode.__torch_dispatch__:171028 | FakeTensor.__torch_dispatch__:3895 | ProxyTorchDispatchMode.__torch_dispatch__:67573 2024-06-26T08:38:02.2637135Z Dynamo produced 3 graphs covering 3975 ops with 6 graph breaks (5 unique) 2024-06-26T08:38:06.9181688Z 2024-06-26T08:38:08.8532340Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:38:08.8533023Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:38:08.8533469Z cuda train hf_DistilBert 2024-06-26T08:38:17.9657188Z W0626 08:38:17.964000 139947951071872 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:38:28.1696231Z pass 2024-06-26T08:38:28.1966178Z TIMING: entire_frame_compile:14.53815 backend_compile:9.27077 2024-06-26T08:38:28.1967325Z STATS: call_* op count: 540 | FakeTensor.__torch_dispatch__:2690 | FakeTensorMode.__torch_dispatch__:31905 | ProxyTorchDispatchMode.__torch_dispatch__:11112 2024-06-26T08:38:28.1968758Z Dynamo produced 3 graphs covering 540 ops with 6 graph breaks (5 unique) 2024-06-26T08:38:31.8063738Z 2024-06-26T08:38:34.8233289Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:38:34.8233769Z loading model: 0it [00:03, ?it/s] 2024-06-26T08:38:34.8234206Z cuda train hf_GPT2 2024-06-26T08:38:49.0501084Z W0626 08:38:49.049000 140068693901952 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:39:03.3549336Z pass 2024-06-26T08:39:03.4479897Z TIMING: entire_frame_compile:23.51591 backend_compile:14.38549 2024-06-26T08:39:03.4482696Z STATS: call_* op count: 1088 | FakeTensorMode.__torch_dispatch__:50341 | FakeTensor.__torch_dispatch__:3212 | ProxyTorchDispatchMode.__torch_dispatch__:17951 2024-06-26T08:39:03.4483975Z Dynamo produced 3 graphs covering 1088 ops with 6 graph breaks (5 unique) 2024-06-26T08:39:07.3106708Z 2024-06-26T08:39:19.7006668Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:39:19.7007285Z loading model: 0it [00:12, ?it/s] 2024-06-26T08:39:19.7007797Z cuda train hf_GPT2_large 2024-06-26T08:39:19.7008740Z pass_due_to_skip 2024-06-26T08:39:20.1980453Z TIMING: 2024-06-26T08:39:20.1980849Z STATS: call_* op count: 0 2024-06-26T08:39:20.1981395Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T08:39:23.4823908Z 2024-06-26T08:39:24.9455745Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:39:24.9456371Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:39:24.9458500Z cuda train hf_Reformer 2024-06-26T08:39:37.8423734Z W0626 08:39:37.841000 140602514387584 torch/_logging/_internal.py:1040] [32/0] Profiler function will be ignored 2024-06-26T08:39:45.9682344Z pass 2024-06-26T08:39:46.0354275Z TIMING: entire_frame_compile:12.98341 backend_compile:7.07572 2024-06-26T08:39:46.0356122Z STATS: call_* op count: 688 | FakeTensorMode.__torch_dispatch__:24613 | ProxyTorchDispatchMode.__torch_dispatch__:7563 | FakeTensor.__torch_dispatch__:1592 2024-06-26T08:39:46.0357298Z Dynamo produced 17 graphs covering 688 ops with 25 graph breaks (9 unique) 2024-06-26T08:39:49.7504482Z 2024-06-26T08:39:53.8020986Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:39:53.8021463Z loading model: 0it [00:04, ?it/s] 2024-06-26T08:39:53.8025307Z cuda train hf_T5_base 2024-06-26T08:39:56.9320115Z WARNING:common:fp64 golden ref were not generated for hf_T5_base. Setting accuracy check to cosine 2024-06-26T08:39:59.0411440Z ERROR:common: 2024-06-26T08:39:59.0412014Z Traceback (most recent call last): 2024-06-26T08:39:59.0413136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2564, in check_accuracy 2024-06-26T08:39:59.0414341Z correct_rerun_result = self.run_n_iterations( 2024-06-26T08:39:59.0415581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2359, in run_n_iterations 2024-06-26T08:39:59.0416820Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-06-26T08:39:59.0418195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 458, in forward_and_backward_pass 2024-06-26T08:39:59.0419450Z self.grad_scaler.scale(loss).backward() 2024-06-26T08:39:59.0420826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward 2024-06-26T08:39:59.0421957Z torch.autograd.backward( 2024-06-26T08:39:59.0423204Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 288, in backward 2024-06-26T08:39:59.0424370Z _engine_run_backward( 2024-06-26T08:39:59.0425714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 768, in _engine_run_backward 2024-06-26T08:39:59.0427413Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-06-26T08:39:59.0433299Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 192.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 88.44 MiB is free. Process 20264 has 21.88 GiB memory in use. Of the allocated memory 20.95 GiB is allocated by PyTorch, and 595.70 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-06-26T08:39:59.1268302Z eager_2nd_run_OOM 2024-06-26T08:39:59.1293271Z TIMING: 2024-06-26T08:39:59.1293712Z STATS: call_* op count: 0 2024-06-26T08:39:59.1294503Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T08:40:02.4537909Z 2024-06-26T08:40:12.1323683Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:40:12.1324157Z loading model: 0it [00:09, ?it/s] 2024-06-26T08:40:12.1324608Z cuda train hf_T5_large 2024-06-26T08:40:12.1327115Z pass_due_to_skip 2024-06-26T08:40:12.5991361Z TIMING: 2024-06-26T08:40:12.5991699Z STATS: call_* op count: 0 2024-06-26T08:40:12.5992244Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T08:40:15.8766526Z 2024-06-26T08:40:16.9196366Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:40:16.9196854Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:40:16.9197629Z cuda train hf_Whisper 2024-06-26T08:40:22.4757145Z W0626 08:40:22.474000 140359240528512 torch/_logging/_internal.py:1040] [5/0] Profiler function will be ignored 2024-06-26T08:40:29.5274971Z pass 2024-06-26T08:40:29.5454611Z TIMING: entire_frame_compile:9.72983 backend_compile:5.9718 2024-06-26T08:40:29.5455716Z STATS: call_* op count: 347 | FakeTensor.__torch_dispatch__:1908 | FakeTensorMode.__torch_dispatch__:20302 | ProxyTorchDispatchMode.__torch_dispatch__:6891 2024-06-26T08:40:29.5456843Z Dynamo produced 3 graphs covering 347 ops with 6 graph breaks (5 unique) 2024-06-26T08:40:33.1106099Z 2024-06-26T08:40:33.3804518Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:40:33.3805655Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:40:33.3806710Z cuda train hf_distil_whisper 2024-06-26T08:40:33.3811443Z Traceback (most recent call last): 2024-06-26T08:40:33.3813080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:40:33.3814743Z ) = runner.load_model( 2024-06-26T08:40:33.3815631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 320, in load_model 2024-06-26T08:40:33.3816488Z benchmark = benchmark_cls( 2024-06-26T08:40:33.3817479Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-06-26T08:40:33.3818288Z obj = type.__call__(cls, *args, **kwargs) 2024-06-26T08:40:33.3819204Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper/__init__.py", line 12, in __init__ 2024-06-26T08:40:33.3820212Z raise NotImplementedError("Training is not implemented.") 2024-06-26T08:40:33.3820808Z NotImplementedError: Training is not implemented. 2024-06-26T08:40:33.3821167Z 2024-06-26T08:40:33.3821286Z model_fail_to_load 2024-06-26T08:40:36.5593627Z 2024-06-26T08:40:36.9327710Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:40:36.9328177Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:40:36.9328649Z cuda train lennard_jones 2024-06-26T08:40:37.6480379Z W0626 08:40:37.647000 139921497272960 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:40:38.6741865Z pass 2024-06-26T08:40:38.6743053Z TIMING: entire_frame_compile:1.51282 backend_compile:0.87449 2024-06-26T08:40:38.6745239Z STATS: call_* op count: 61 | FakeTensor.__torch_dispatch__:298 | FakeTensorMode.__torch_dispatch__:2815 | ProxyTorchDispatchMode.__torch_dispatch__:921 2024-06-26T08:40:38.6746614Z Dynamo produced 3 graphs covering 61 ops with 7 graph breaks (5 unique) 2024-06-26T08:40:39.4119610Z accuracy pass_rate=81.82% 2024-06-26T08:40:39.4122629Z calls_captured gmean=0.00x mean=448.030x 2024-06-26T08:40:39.4125877Z unique_graphs gmean=0.00x mean=3.303x 2024-06-26T08:40:39.4129353Z graph_breaks gmean=0.00x mean=6.545x 2024-06-26T08:40:39.4132615Z unique_graph_breaks gmean=0.00x mean=4.333x 2024-06-26T08:40:39.4136178Z autograd_captures gmean=0.00x mean=0.000x 2024-06-26T08:40:39.4139399Z autograd_compiles gmean=0.00x mean=0.000x 2024-06-26T08:40:39.4142844Z cudagraph_skips gmean=0.00x mean=0.000x 2024-06-26T08:40:39.9690245Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy//aot_eager_torchbench_training.csv 2024-06-26T08:40:40.2683460Z torchrec_dlrm PASS 2024-06-26T08:40:40.2686429Z BERT_pytorch PASS 2024-06-26T08:40:40.2690858Z Background_Matting XFAIL 2024-06-26T08:40:40.2695382Z LearningToPaint PASS 2024-06-26T08:40:40.2699787Z Super_SloMo PASS 2024-06-26T08:40:40.2704283Z alexnet PASS 2024-06-26T08:40:40.2708764Z basic_gnn_edgecnn PASS 2024-06-26T08:40:40.2713414Z basic_gnn_gcn PASS 2024-06-26T08:40:40.2718048Z basic_gnn_gin PASS 2024-06-26T08:40:40.2722458Z basic_gnn_sage PASS 2024-06-26T08:40:40.2727061Z dcgan PASS 2024-06-26T08:40:40.2731406Z demucs PASS 2024-06-26T08:40:40.2735666Z densenet121 PASS 2024-06-26T08:40:40.2740140Z detectron2_maskrcnn_r_50_c4 XFAIL 2024-06-26T08:40:40.2744305Z dlrm PASS 2024-06-26T08:40:40.2748849Z drq PASS 2024-06-26T08:40:40.2753524Z fastNLP_Bert PASS 2024-06-26T08:40:40.2758094Z functorch_dp_cifar10 PASS 2024-06-26T08:40:40.2762196Z functorch_maml_omniglot PASS 2024-06-26T08:40:40.2766630Z hf_Albert PASS 2024-06-26T08:40:40.2770924Z hf_Bart PASS 2024-06-26T08:40:40.2775467Z hf_Bert PASS 2024-06-26T08:40:40.2779927Z hf_Bert_large PASS 2024-06-26T08:40:40.2784261Z hf_BigBird PASS 2024-06-26T08:40:40.2788662Z hf_DistilBert PASS 2024-06-26T08:40:40.2793389Z hf_GPT2 PASS 2024-06-26T08:40:40.2797814Z hf_GPT2_large XFAIL 2024-06-26T08:40:40.2802286Z hf_Reformer PASS 2024-06-26T08:40:40.2806549Z hf_T5_base XFAIL 2024-06-26T08:40:40.2811175Z hf_T5_large XFAIL 2024-06-26T08:40:40.2815377Z hf_Whisper PASS 2024-06-26T08:40:40.2819892Z hf_distil_whisper XFAIL 2024-06-26T08:40:40.2824147Z lennard_jones PASS 2024-06-26T08:40:40.3287481Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy//aot_eager_torchbench_training.csv 2024-06-26T08:40:40.6652747Z torchrec_dlrm PASS 2024-06-26T08:40:40.6656124Z BERT_pytorch PASS 2024-06-26T08:40:40.6660760Z Background_Matting PASS 2024-06-26T08:40:40.6665487Z LearningToPaint PASS 2024-06-26T08:40:40.6670529Z Super_SloMo PASS 2024-06-26T08:40:40.6676039Z alexnet PASS 2024-06-26T08:40:40.6680412Z basic_gnn_edgecnn PASS 2024-06-26T08:40:40.6685508Z basic_gnn_gcn PASS 2024-06-26T08:40:40.6690290Z basic_gnn_gin PASS 2024-06-26T08:40:40.6695015Z basic_gnn_sage PASS 2024-06-26T08:40:40.6700285Z dcgan PASS 2024-06-26T08:40:40.6704763Z demucs PASS 2024-06-26T08:40:40.6709707Z densenet121 PASS 2024-06-26T08:40:40.6715757Z detectron2_maskrcnn_r_50_c4 PASS 2024-06-26T08:40:40.6720171Z dlrm PASS 2024-06-26T08:40:40.6725121Z drq PASS 2024-06-26T08:40:40.6730063Z fastNLP_Bert PASS 2024-06-26T08:40:40.6735018Z functorch_dp_cifar10 PASS 2024-06-26T08:40:40.6739895Z functorch_maml_omniglot PASS 2024-06-26T08:40:40.6744814Z hf_Albert PASS 2024-06-26T08:40:40.6749698Z hf_Bart PASS 2024-06-26T08:40:40.6755459Z hf_Bert PASS 2024-06-26T08:40:40.6760479Z hf_Bert_large PASS 2024-06-26T08:40:40.6765337Z hf_BigBird PASS 2024-06-26T08:40:40.6770587Z hf_DistilBert PASS 2024-06-26T08:40:40.6775373Z hf_GPT2 PASS 2024-06-26T08:40:40.6780453Z hf_GPT2_large PASS 2024-06-26T08:40:40.6785704Z hf_Reformer PASS 2024-06-26T08:40:40.6790950Z hf_T5_base PASS 2024-06-26T08:40:40.6796041Z hf_T5_large PASS 2024-06-26T08:40:40.6800790Z hf_Whisper PASS 2024-06-26T08:40:40.6805544Z hf_distil_whisper PASS 2024-06-26T08:40:40.6810418Z lennard_jones PASS 2024-06-26T08:40:40.7286918Z + cleanup_workspace 2024-06-26T08:40:40.7288203Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-06-26T08:40:40.7289709Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-06-26T08:40:40.7290674Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-06-26T08:40:40.7291352Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-06-26T08:40:40.7292161Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-06-26T08:40:40.7293051Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-06-26T08:40:40.7293741Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-06-26T08:40:41.3744311Z ##[group]Run cat test/**/*_toprint.log || true 2024-06-26T08:40:41.3744822Z cat test/**/*_toprint.log || true 2024-06-26T08:40:41.3753160Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:40:41.3753664Z env: 2024-06-26T08:40:41.3753938Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:40:41.3754383Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:40:41.3755111Z DOCKER_CONTAINER_ID: 299b633493af89e9314f5bbb416079e7c4e43079b580c4c703a3fdf1a00e6a50 2024-06-26T08:40:41.3755747Z ##[endgroup] 2024-06-26T08:40:41.3817099Z cat: test/**/*_toprint.log: No such file or directory 2024-06-26T08:40:41.3868934Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-06-26T08:40:41.3869403Z kill "$MONITOR_SCRIPT_PID" 2024-06-26T08:40:41.3876426Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:40:41.3876927Z env: 2024-06-26T08:40:41.3877198Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:40:41.3877656Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:40:41.3878409Z DOCKER_CONTAINER_ID: 299b633493af89e9314f5bbb416079e7c4e43079b580c4c703a3fdf1a00e6a50 2024-06-26T08:40:41.3879074Z MONITOR_SCRIPT_PID: 10380 2024-06-26T08:40:41.3879425Z ##[endgroup] 2024-06-26T08:40:41.4064072Z Prepare all required actions 2024-06-26T08:40:41.4064517Z Getting action download info 2024-06-26T08:40:41.5648747Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-06-26T08:40:41.7008234Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-06-26T08:40:41.7008715Z with: 2024-06-26T08:40:41.7009261Z file-suffix: test-aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688916210 2024-06-26T08:40:41.7009941Z s3-bucket: gha-artifacts 2024-06-26T08:40:41.7010280Z env: 2024-06-26T08:40:41.7010556Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:40:41.7011012Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:40:41.7011751Z DOCKER_CONTAINER_ID: 299b633493af89e9314f5bbb416079e7c4e43079b580c4c703a3fdf1a00e6a50 2024-06-26T08:40:41.7012403Z ##[endgroup] 2024-06-26T08:40:41.7038301Z ##[group]Run # Remove any previous test jsons if they exist 2024-06-26T08:40:41.7039028Z # Remove any previous test jsons if they exist 2024-06-26T08:40:41.7039533Z rm -f test-jsons-*.zip 2024-06-26T08:40:41.7040061Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-06-26T08:40:41.7047966Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:40:41.7048464Z env: 2024-06-26T08:40:41.7048731Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:40:41.7049185Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:40:41.7049915Z DOCKER_CONTAINER_ID: 299b633493af89e9314f5bbb416079e7c4e43079b580c4c703a3fdf1a00e6a50 2024-06-26T08:40:41.7050815Z FILE_SUFFIX: test-aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688916210 2024-06-26T08:40:41.7051474Z ##[endgroup] 2024-06-26T08:40:41.7227948Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-06-26T08:40:41.7258035Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-06-26T08:40:41.7259008Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-06-26T08:40:41.7265070Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-06-26T08:40:41.7291883Z ##[group]Run # Remove any previous test reports if they exist 2024-06-26T08:40:41.7292536Z # Remove any previous test reports if they exist 2024-06-26T08:40:41.7293077Z rm -f test-reports-*.zip 2024-06-26T08:40:41.7293677Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-06-26T08:40:41.7301391Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:40:41.7301900Z env: 2024-06-26T08:40:41.7302188Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:40:41.7302650Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:40:41.7303380Z DOCKER_CONTAINER_ID: 299b633493af89e9314f5bbb416079e7c4e43079b580c4c703a3fdf1a00e6a50 2024-06-26T08:40:41.7304300Z FILE_SUFFIX: test-aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688916210 2024-06-26T08:40:41.7304960Z ##[endgroup] 2024-06-26T08:40:41.7459978Z adding: test/test-reports/inference_torchbench.csv (deflated 73%) 2024-06-26T08:40:41.7464533Z adding: test/test-reports/inference_torchbench_graph_breaks.csv (deflated 92%) 2024-06-26T08:40:41.7466778Z adding: test/test-reports/inference_torchbench_graph_break_deduped.csv (deflated 79%) 2024-06-26T08:40:41.7467728Z adding: test/test-reports/training_torchbench.csv (deflated 64%) 2024-06-26T08:40:41.7474394Z adding: test/test-reports/training_torchbench_graph_breaks.csv (deflated 97%) 2024-06-26T08:40:41.7475512Z adding: test/test-reports/training_torchbench_graph_break_deduped.csv (deflated 77%) 2024-06-26T08:40:41.7500515Z ##[group]Run # Remove any previous usage logs if they exist 2024-06-26T08:40:41.7501138Z # Remove any previous usage logs if they exist 2024-06-26T08:40:41.7501633Z rm -f logs-*.zip 2024-06-26T08:40:41.7502319Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-06-26T08:40:41.7503211Z # so check to see if the file exists first 2024-06-26T08:40:41.7503723Z if [ -f 'usage_log.txt' ]; then 2024-06-26T08:40:41.7504248Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-06-26T08:40:41.7504739Z fi 2024-06-26T08:40:41.7505101Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-06-26T08:40:41.7505680Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-06-26T08:40:41.7506175Z fi 2024-06-26T08:40:41.7514025Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:40:41.7514529Z env: 2024-06-26T08:40:41.7514857Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:40:41.7515366Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:40:41.7516084Z DOCKER_CONTAINER_ID: 299b633493af89e9314f5bbb416079e7c4e43079b580c4c703a3fdf1a00e6a50 2024-06-26T08:40:41.7516973Z FILE_SUFFIX: test-aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688916210 2024-06-26T08:40:41.7517616Z ##[endgroup] 2024-06-26T08:40:41.7590925Z adding: usage_log.txt (deflated 92%) 2024-06-26T08:40:41.7654973Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-06-26T08:40:41.7655687Z # Remove any previous debugging artifacts if they exist 2024-06-26T08:40:41.7656236Z rm -f debug-*.zip 2024-06-26T08:40:41.7656661Z if [ -d 'test/debug' ]; then 2024-06-26T08:40:41.7657157Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-06-26T08:40:41.7657630Z fi 2024-06-26T08:40:41.7664691Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:40:41.7665194Z env: 2024-06-26T08:40:41.7665463Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:40:41.7665914Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:40:41.7666696Z DOCKER_CONTAINER_ID: 299b633493af89e9314f5bbb416079e7c4e43079b580c4c703a3fdf1a00e6a50 2024-06-26T08:40:41.7667732Z FILE_SUFFIX: test-aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688916210 2024-06-26T08:40:41.7668371Z ##[endgroup] 2024-06-26T08:40:41.7741463Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:40:41.7741893Z with: 2024-06-26T08:40:41.7742171Z s3-bucket: gha-artifacts 2024-06-26T08:40:41.7742592Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:40:41.7743069Z retention-days: 14 2024-06-26T08:40:41.7743394Z if-no-files-found: warn 2024-06-26T08:40:41.7743749Z path: test-jsons-*.zip 2024-06-26T08:40:41.7744086Z name: artifact 2024-06-26T08:40:41.7744388Z region: us-east-1 2024-06-26T08:40:41.7744675Z env: 2024-06-26T08:40:41.7744946Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:40:41.7745395Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:40:41.7746120Z DOCKER_CONTAINER_ID: 299b633493af89e9314f5bbb416079e7c4e43079b580c4c703a3fdf1a00e6a50 2024-06-26T08:40:41.7746749Z ##[endgroup] 2024-06-26T08:40:42.0854574Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:40:42.0855322Z With the provided path, there will be 1 file uploaded 2024-06-26T08:40:42.0856085Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:40:42.0882983Z Starting upload of test-jsons-test-aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688916210.zip 2024-06-26T08:40:42.2381151Z Finished upload of test-jsons-test-aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688916210.zip 2024-06-26T08:40:42.2560431Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:40:42.2560884Z with: 2024-06-26T08:40:42.2561160Z s3-bucket: gha-artifacts 2024-06-26T08:40:42.2561589Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:40:42.2562059Z retention-days: 14 2024-06-26T08:40:42.2562399Z if-no-files-found: error 2024-06-26T08:40:42.2562754Z path: test-reports-*.zip 2024-06-26T08:40:42.2563101Z name: artifact 2024-06-26T08:40:42.2563420Z region: us-east-1 2024-06-26T08:40:42.2563718Z env: 2024-06-26T08:40:42.2563987Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:40:42.2564432Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:40:42.2565251Z DOCKER_CONTAINER_ID: 299b633493af89e9314f5bbb416079e7c4e43079b580c4c703a3fdf1a00e6a50 2024-06-26T08:40:42.2565889Z ##[endgroup] 2024-06-26T08:40:42.5644254Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:40:42.5644878Z With the provided path, there will be 1 file uploaded 2024-06-26T08:40:42.5645502Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:40:42.5671875Z Starting upload of test-reports-test-aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688916210.zip 2024-06-26T08:40:42.6809278Z Finished upload of test-reports-test-aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688916210.zip 2024-06-26T08:40:42.6947394Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:40:42.6947836Z with: 2024-06-26T08:40:42.6948121Z s3-bucket: gha-artifacts 2024-06-26T08:40:42.6948563Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:40:42.6949035Z retention-days: 14 2024-06-26T08:40:42.6949366Z if-no-files-found: ignore 2024-06-26T08:40:42.6949735Z path: logs-*.zip 2024-06-26T08:40:42.6950142Z name: artifact 2024-06-26T08:40:42.6950469Z region: us-east-1 2024-06-26T08:40:42.6950770Z env: 2024-06-26T08:40:42.6951052Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:40:42.6951500Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:40:42.6952212Z DOCKER_CONTAINER_ID: 299b633493af89e9314f5bbb416079e7c4e43079b580c4c703a3fdf1a00e6a50 2024-06-26T08:40:42.6953097Z ##[endgroup] 2024-06-26T08:40:43.0046274Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:40:43.0046919Z With the provided path, there will be 1 file uploaded 2024-06-26T08:40:43.0047536Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:40:43.0073623Z Starting upload of logs-test-aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688916210.zip 2024-06-26T08:40:43.1254930Z Finished upload of logs-test-aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_26688916210.zip 2024-06-26T08:40:43.1398999Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:40:43.1399445Z with: 2024-06-26T08:40:43.1399725Z s3-bucket: gha-artifacts 2024-06-26T08:40:43.1400152Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T08:40:43.1400627Z retention-days: 14 2024-06-26T08:40:43.1400958Z if-no-files-found: ignore 2024-06-26T08:40:43.1401317Z path: debug-*.zip 2024-06-26T08:40:43.1401632Z name: artifact 2024-06-26T08:40:43.1401930Z region: us-east-1 2024-06-26T08:40:43.1402228Z env: 2024-06-26T08:40:43.1402499Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:40:43.1402949Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:40:43.1403672Z DOCKER_CONTAINER_ID: 299b633493af89e9314f5bbb416079e7c4e43079b580c4c703a3fdf1a00e6a50 2024-06-26T08:40:43.1404308Z ##[endgroup] 2024-06-26T08:40:43.4434406Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-06-26T08:40:43.4569545Z ##[group]Run # shellcheck disable=SC2156 2024-06-26T08:40:43.4570030Z # shellcheck disable=SC2156 2024-06-26T08:40:43.4570829Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-06-26T08:40:43.4578783Z shell: /usr/bin/bash -e {0} 2024-06-26T08:40:43.4579138Z env: 2024-06-26T08:40:43.4579414Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:40:43.4579870Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:40:43.4580593Z DOCKER_CONTAINER_ID: 299b633493af89e9314f5bbb416079e7c4e43079b580c4c703a3fdf1a00e6a50 2024-06-26T08:40:43.4581230Z ##[endgroup] 2024-06-26T08:40:43.6227082Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-06-26T08:40:43.6227659Z with: 2024-06-26T08:40:43.6227913Z env: 2024-06-26T08:40:43.6228187Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:40:43.6228657Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:40:43.6229387Z DOCKER_CONTAINER_ID: 299b633493af89e9314f5bbb416079e7c4e43079b580c4c703a3fdf1a00e6a50 2024-06-26T08:40:43.6230024Z ##[endgroup] 2024-06-26T08:40:43.6247312Z ##[group]Run set -eou pipefail 2024-06-26T08:40:43.6247724Z set -eou pipefail 2024-06-26T08:40:43.6248079Z  2024-06-26T08:40:43.6248590Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-06-26T08:40:43.6249230Z for _ in $(seq 1440); do 2024-06-26T08:40:43.6249711Z  # Break if no ssh session exists anymore 2024-06-26T08:40:43.6250219Z  if [ "$(who)" = "" ]; then 2024-06-26T08:40:43.6250627Z  break 2024-06-26T08:40:43.6250964Z  fi 2024-06-26T08:40:43.6251257Z  echo "." 2024-06-26T08:40:43.6251582Z  sleep 5 2024-06-26T08:40:43.6251896Z done 2024-06-26T08:40:43.6259205Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:40:43.6259715Z env: 2024-06-26T08:40:43.6259994Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:40:43.6260452Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:40:43.6261171Z DOCKER_CONTAINER_ID: 299b633493af89e9314f5bbb416079e7c4e43079b580c4c703a3fdf1a00e6a50 2024-06-26T08:40:43.6261802Z ##[endgroup] 2024-06-26T08:40:43.6281932Z Holding runner for 2 hours until all ssh sessions have logged out 2024-06-26T08:40:43.6317443Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T08:40:43.6318219Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T08:40:43.6319016Z # shellcheck disable=SC2046 2024-06-26T08:40:43.6319491Z docker stop $(docker ps -q) || true 2024-06-26T08:40:43.6319966Z # Prune all of the docker images 2024-06-26T08:40:43.6320420Z docker system prune -af 2024-06-26T08:40:43.6327277Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:40:43.6327970Z env: 2024-06-26T08:40:43.6328241Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:40:43.6328699Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:40:43.6329435Z DOCKER_CONTAINER_ID: 299b633493af89e9314f5bbb416079e7c4e43079b580c4c703a3fdf1a00e6a50 2024-06-26T08:40:43.6330073Z ##[endgroup] 2024-06-26T08:40:43.9993460Z 299b633493af 2024-06-26T08:40:46.3164298Z Deleted Containers: 2024-06-26T08:40:46.3164867Z 299b633493af89e9314f5bbb416079e7c4e43079b580c4c703a3fdf1a00e6a50 2024-06-26T08:40:46.3165295Z 2024-06-26T08:40:50.2159164Z Deleted Images: 2024-06-26T08:40:50.2161645Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T08:40:50.2164385Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T08:40:50.2166500Z deleted: sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd 2024-06-26T08:40:50.2167797Z deleted: sha256:754ead7012f0b7691885a9660f02e80be6cee8c5665486fb995c1fb06adb6e92 2024-06-26T08:40:50.2168932Z deleted: sha256:fbe4496d8830e11be86d8f222522e7956f32f85bfc06b05ea53961f34a58af74 2024-06-26T08:40:50.2169842Z deleted: sha256:1c83814aa1a1a7e0cc998e3479bdab8e8766b975560cbf00690acf39ed899a80 2024-06-26T08:40:50.2170937Z deleted: sha256:e9ea8e61942c7f8ae4aab97b8d102b01d6bb4b6cf2a605808cd34130a17d8a0f 2024-06-26T08:40:50.2171912Z deleted: sha256:acc5fc32588cd95e082a6ce35e48d0f303ae8f96e3ab65c4e375b85e1f3c4a92 2024-06-26T08:40:50.2172753Z deleted: sha256:a93e53400904d29e4185d3be942ea34f19572ea438962788ae4e9b3d9bf4c498 2024-06-26T08:40:50.2173798Z deleted: sha256:c97092b88b5e4c77c68bbc4b51afab64e38719af1ff58013ec49a696cdd40a6e 2024-06-26T08:40:50.2174632Z deleted: sha256:b132889c90d67ffa53655846603e3fcc2372c23f04dbdbc54690a39d304eb89b 2024-06-26T08:40:50.2175444Z deleted: sha256:915d54081ed607710624abe3f2490114c745858831f4fc13396bd3b4b9a121bc 2024-06-26T08:40:50.2176259Z deleted: sha256:bfa8d446c498c87f535aef102695564de1b22c755e6c63944ae2f5073a2b31ec 2024-06-26T08:40:50.2177094Z deleted: sha256:e71460fb363c29e70bbe357d680aaa7ae694f094ef2f2713609158ad90611744 2024-06-26T08:40:50.2177920Z deleted: sha256:84079bb1b4ee1d0119ae3d3e701de460a1324912297c675425c9a36d30fcdb0f 2024-06-26T08:40:50.2178748Z deleted: sha256:d86ac3ca148d4758cb03b493fdd49109d8f1a2c7c85c4b755493c3c4529553e5 2024-06-26T08:40:50.2179631Z deleted: sha256:532b3ab97c44a673a0d5f62ee46fa6a83248a84de8877754a612355be56ed0a3 2024-06-26T08:40:50.2180458Z deleted: sha256:617e2a35f25ae24500c88c0c9b8475f0687fbcbed3d200013b4c99c6d8941aff 2024-06-26T08:40:50.2181301Z deleted: sha256:b88120bcd5a26a660c09c5df13176e0ef098fb2c8cdb57d74ab0796a3d1194d6 2024-06-26T08:40:50.2182121Z deleted: sha256:c016b2e7764735243205fb76fa9a273bb3dd4bf05532288e3a613ce022131ac3 2024-06-26T08:40:50.2182945Z deleted: sha256:bcaca462c390deddd2995de2d245a7495e7d2e1e94f0edad30a573c83d0588fd 2024-06-26T08:40:50.2183791Z deleted: sha256:9e260eba003d5c25d7b410e5c4c3093845d9e46611896840901c689d7fbf8ee6 2024-06-26T08:40:50.2184605Z deleted: sha256:3e62b86f8b911a9b59c62d55f063e71683ae6a01c22418db5d27930712adaad6 2024-06-26T08:40:50.2185427Z deleted: sha256:905920ccb1d88449b8dde686c34a4b2a8b6f59e78ac44916b189ea3593b9fbbb 2024-06-26T08:40:50.2186236Z deleted: sha256:961201237e03acd5061406238dd7a3c1343ac888d7583212cd0ab5ae1b4f0540 2024-06-26T08:40:50.2187054Z deleted: sha256:70047a2aac62159fa28fb4dbd4567ec2ffbab68fe046967e09acb0eb3e5cb9cb 2024-06-26T08:40:50.2187896Z deleted: sha256:1ec680ad781436ae65fa987f5fa1ed032ecd5c82a6e35eea6707de501f62c0f4 2024-06-26T08:40:50.2188736Z deleted: sha256:ef859623a50e85abf6e6c1dbd6188aa765a66cc7b2bd618a1cbe4825b7b2bd91 2024-06-26T08:40:50.2189623Z deleted: sha256:404ade5cacff59773b93ab438bfa64cd3a171d07f6f168bdc4a5d4bfa0b43526 2024-06-26T08:40:50.2190625Z deleted: sha256:5b27de6ee3d9f2f18217bf18dec725f7b4954c0e48c85d35752634c22d62209b 2024-06-26T08:40:50.2191621Z deleted: sha256:498e39c575b3fa069acaac7e46b91b4e54ae3a05f2ada5c4347091f346f25d4c 2024-06-26T08:40:50.2192749Z deleted: sha256:1405919d2694d65669b1f4c842282021afe89208ce86570e96464903733da2d0 2024-06-26T08:40:50.2193580Z deleted: sha256:d5b41e5d70896c0472fc3e6da0516a17d89cd6635dc75010ecddc31ce9690fad 2024-06-26T08:40:50.2194420Z deleted: sha256:18e9379b62b9d0bb40293d28a652eafaee66a499f66f0776bf14e37a513101b7 2024-06-26T08:40:50.2195264Z deleted: sha256:a781f80ed00ff2926a699fd01a309bb0b4fb7d5655eb652c205bb9c7f811035f 2024-06-26T08:40:50.2196096Z deleted: sha256:c0430973b7cf759075330ee29c79bfd1312c91e6033d9dd883b4b1c29743a171 2024-06-26T08:40:50.2196936Z deleted: sha256:e733b3d66efb9342e5a283376c0e1ddffa47a42fecc26e61ecaf0a3472885d97 2024-06-26T08:40:50.2197768Z deleted: sha256:8929c0b2047f74b84c82c39eff9077a92ae1a828cde00c9ab78fb3e0ed5e0283 2024-06-26T08:40:50.2198596Z deleted: sha256:f5b56155fb58a0a60fe6391d3581cf91610ed69d4f0a321257c19944da296a52 2024-06-26T08:40:50.2199429Z deleted: sha256:b54d2cb355238d1aa91e58df93064447b80f53706d1ca62823acc834f5ac54fe 2024-06-26T08:40:50.2200260Z deleted: sha256:e7311285af9eeddb51aa5629aa98386e89f1456ff32843b6aee6443247edf66c 2024-06-26T08:40:50.2201084Z deleted: sha256:67657ceec347da8dc8085d5731633efeac115ba677fa4b7c5966c4260f7abab0 2024-06-26T08:40:50.2201917Z deleted: sha256:b496e50409dbf414930fc9b4437aa0aa4ce27bc7840e050ae459aaffcf9aca86 2024-06-26T08:40:50.2202742Z deleted: sha256:8f5276170cc4a60d36ad7b964a502306b29546254cedcbcb101ea9123497d2c0 2024-06-26T08:40:50.2203552Z deleted: sha256:53d524d1564745e96a7e515462171b73184d9f1ff5454b32fcfbbca39d361467 2024-06-26T08:40:50.2204377Z deleted: sha256:2acbadb00d69b6aca575f04baba4ea26ae2b5c2d85f09945004ad93a78a8eb75 2024-06-26T08:40:50.2205352Z deleted: sha256:02665efeff878aeb603c1edb2e21cb700c171c025d64f5aae1b2b20b12f3964a 2024-06-26T08:40:50.2206209Z deleted: sha256:19bd23e9a51dfe883a8b8d2ff6d66ff1b3476b8912856406bb79b81762aa6d4b 2024-06-26T08:40:50.2207043Z deleted: sha256:73aa8637bf9299d6298dabd858480c274a62117388f2ca1d5e9eb15c6d1e4155 2024-06-26T08:40:50.2207866Z deleted: sha256:5f8968406305f348b1caebc0e035187d6e9965ace7b9d5f2f21114d37c9fe2e1 2024-06-26T08:40:50.2208700Z deleted: sha256:d827104914db0c503cf5c81f13e2456bf4efce42d4cf1f8d5d70838207278bee 2024-06-26T08:40:50.2209582Z deleted: sha256:867b219724f2f1aaf56f42e3e9f03a5d64bdca8a6a72407efd10328aeea5d510 2024-06-26T08:40:50.2210423Z deleted: sha256:b9ff2a7cceb9237b26bcb17728edf8a558eec5a76f63626e84f80541dfc0f1d4 2024-06-26T08:40:50.2211251Z deleted: sha256:be73f509106819007ce2b0f5a1301ac56ae583cb6c1fb69bee93efa0840368f1 2024-06-26T08:40:50.2212089Z deleted: sha256:2a11c07c915c3fdbb4e73096dea73ea950c970d50fe1a5764060bd62b7b7cf2c 2024-06-26T08:40:50.2212933Z deleted: sha256:5e9989622e2abc1c72f8184343507b45d75e4ff971edb68b96cbcd75ae484ff5 2024-06-26T08:40:50.2213782Z deleted: sha256:05ce642bd9cf5a9205bff107df28b1d6cef7d5f6ab3ea944d28c032ec61270df 2024-06-26T08:40:50.2214817Z deleted: sha256:017702617f6c07b73f2088678f4710b794cfe897a213a5cf122a0e2519eae0ac 2024-06-26T08:40:50.2215871Z deleted: sha256:44b3b37bb1ca39dc5cf356de41a23a5c12f605a1992813e408087fb2b0ad6d1c 2024-06-26T08:40:50.2216702Z deleted: sha256:8c387982e9f27db357a1d149dc149cabc0bd04668ad6956387a5072682dada67 2024-06-26T08:40:50.2217526Z deleted: sha256:10d415dca4e3fbb88a747b40317f3c6333c041ba8f73855f67fb9fe981fae5ee 2024-06-26T08:40:50.2218351Z deleted: sha256:6c65469bf0f53dcd625faefe363b8f3136c08388a6cf64d9158e8576e4ee89b1 2024-06-26T08:40:50.2219217Z deleted: sha256:027db5d8b76deeefbd8c34b485ab7ff14ed2a3e6800425c1ce0d77c2e9fdd4ab 2024-06-26T08:40:50.2220066Z deleted: sha256:24941234ad28cf5c8174d7cf18326c10b4e17f2c2137672ea0322df461597dd7 2024-06-26T08:40:50.2220905Z deleted: sha256:240ea2eebfa8519d6c2d88f4c99e66ce4e038d000f93ce8d03726c01d81180ef 2024-06-26T08:40:50.2221733Z deleted: sha256:09e823cb391a69f6a195e3ca8a41624b5f8b34eee14c6eb36277de814a91966e 2024-06-26T08:40:50.2222533Z deleted: sha256:730479d695169a6593bf0ac16b377491dce65175a6d158e012422367d04577fd 2024-06-26T08:40:50.2223459Z deleted: sha256:b0a30f9f7bd5459ff662453b9346a7ba8ea8db1191528345c1953eab4aa2f54e 2024-06-26T08:40:50.2224280Z deleted: sha256:75cefa87effaa3ad361c74642839923101f0f9982f814f3c9e2ed4ad52f2a008 2024-06-26T08:40:50.2225167Z deleted: sha256:6f97db87ba9512e615e871c6982000602982e1e5a5be334a56e7292251e84e00 2024-06-26T08:40:50.2226081Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-06-26T08:40:50.2226915Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-06-26T08:40:50.2227724Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-06-26T08:40:50.2228560Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-06-26T08:40:50.2229406Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-06-26T08:40:50.2230352Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-06-26T08:40:50.2231198Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-06-26T08:40:50.2246907Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-06-26T08:40:50.2247767Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-06-26T08:40:50.2248614Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-06-26T08:40:50.2249430Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-06-26T08:40:50.2249922Z 2024-06-26T08:40:50.2250067Z Total reclaimed space: 40.92GB 2024-06-26T08:40:50.2313388Z Post job cleanup. 2024-06-26T08:40:50.2361066Z Post job cleanup. 2024-06-26T08:40:50.3183435Z [command]/usr/bin/git version 2024-06-26T08:40:50.3241003Z git version 2.40.1 2024-06-26T08:40:50.3280403Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/585ec0d7-a3e5-4452-ad25-b0946a0234c6' before making global git config changes 2024-06-26T08:40:50.3281649Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T08:40:50.3286121Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T08:40:50.3313052Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T08:40:50.3334965Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T08:40:50.3547830Z Entering 'android/libs/fbjni' 2024-06-26T08:40:50.3583674Z Entering 'third_party/FP16' 2024-06-26T08:40:50.3622557Z Entering 'third_party/FXdiv' 2024-06-26T08:40:50.3659619Z Entering 'third_party/NNPACK' 2024-06-26T08:40:50.3694457Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T08:40:50.3730915Z Entering 'third_party/XNNPACK' 2024-06-26T08:40:50.3774737Z Entering 'third_party/benchmark' 2024-06-26T08:40:50.3810113Z Entering 'third_party/cpp-httplib' 2024-06-26T08:40:50.3847693Z Entering 'third_party/cpuinfo' 2024-06-26T08:40:50.3883237Z Entering 'third_party/cudnn_frontend' 2024-06-26T08:40:50.3921052Z Entering 'third_party/cutlass' 2024-06-26T08:40:50.3959783Z Entering 'third_party/eigen' 2024-06-26T08:40:50.3997909Z Entering 'third_party/fbgemm' 2024-06-26T08:40:50.4034492Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T08:40:50.4066773Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T08:40:50.4103809Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T08:40:50.4145182Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T08:40:50.4181849Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T08:40:50.4216242Z Entering 'third_party/flatbuffers' 2024-06-26T08:40:50.4256620Z Entering 'third_party/fmt' 2024-06-26T08:40:50.4289273Z Entering 'third_party/foxi' 2024-06-26T08:40:50.4323139Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T08:40:50.4358006Z Entering 'third_party/gloo' 2024-06-26T08:40:50.4389920Z Entering 'third_party/googletest' 2024-06-26T08:40:50.4423944Z Entering 'third_party/ideep' 2024-06-26T08:40:50.4455243Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T08:40:50.4498597Z Entering 'third_party/ittapi' 2024-06-26T08:40:50.4536481Z Entering 'third_party/kineto' 2024-06-26T08:40:50.4573707Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T08:40:50.4608910Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T08:40:50.4646923Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T08:40:50.4683193Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T08:40:50.4717293Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T08:40:50.4747373Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T08:40:50.4785809Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T08:40:50.4821212Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T08:40:50.4857463Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T08:40:50.4894571Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T08:40:50.4932093Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T08:40:50.4964478Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T08:40:50.4996936Z Entering 'third_party/mimalloc' 2024-06-26T08:40:50.5028877Z Entering 'third_party/nccl/nccl' 2024-06-26T08:40:50.5060956Z Entering 'third_party/nlohmann' 2024-06-26T08:40:50.5093626Z Entering 'third_party/onnx' 2024-06-26T08:40:50.5138247Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T08:40:50.5170880Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T08:40:50.5203962Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T08:40:50.5237349Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T08:40:50.5268421Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T08:40:50.5301228Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T08:40:50.5332054Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T08:40:50.5364089Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T08:40:50.5395450Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T08:40:50.5426258Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T08:40:50.5457297Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T08:40:50.5490614Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T08:40:50.5523330Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T08:40:50.5569909Z Entering 'third_party/pocketfft' 2024-06-26T08:40:50.5601835Z Entering 'third_party/protobuf' 2024-06-26T08:40:50.5637254Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T08:40:50.5668815Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T08:40:50.5701599Z Entering 'third_party/psimd' 2024-06-26T08:40:50.5734052Z Entering 'third_party/pthreadpool' 2024-06-26T08:40:50.5770040Z Entering 'third_party/pybind11' 2024-06-26T08:40:50.5802674Z Entering 'third_party/python-peachpy' 2024-06-26T08:40:50.5838094Z Entering 'third_party/sleef' 2024-06-26T08:40:50.5874335Z Entering 'third_party/tensorpipe' 2024-06-26T08:40:50.5908426Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T08:40:50.5944206Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T08:40:50.5979110Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T08:40:50.6013036Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T08:40:50.6045673Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T08:40:50.6093105Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T08:40:50.6115402Z http.https://github.com/.extraheader 2024-06-26T08:40:50.6124403Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-06-26T08:40:50.6152949Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T08:40:50.6361858Z Entering 'android/libs/fbjni' 2024-06-26T08:40:50.6382684Z http.https://github.com/.extraheader 2024-06-26T08:40:50.6408895Z Entering 'third_party/FP16' 2024-06-26T08:40:50.6431918Z http.https://github.com/.extraheader 2024-06-26T08:40:50.6457225Z Entering 'third_party/FXdiv' 2024-06-26T08:40:50.6478980Z http.https://github.com/.extraheader 2024-06-26T08:40:50.6502385Z Entering 'third_party/NNPACK' 2024-06-26T08:40:50.6525449Z http.https://github.com/.extraheader 2024-06-26T08:40:50.6549468Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T08:40:50.6571384Z http.https://github.com/.extraheader 2024-06-26T08:40:50.6594735Z Entering 'third_party/XNNPACK' 2024-06-26T08:40:50.6615646Z http.https://github.com/.extraheader 2024-06-26T08:40:50.6649967Z Entering 'third_party/benchmark' 2024-06-26T08:40:50.6671949Z http.https://github.com/.extraheader 2024-06-26T08:40:50.6698330Z Entering 'third_party/cpp-httplib' 2024-06-26T08:40:50.6719683Z http.https://github.com/.extraheader 2024-06-26T08:40:50.6744637Z Entering 'third_party/cpuinfo' 2024-06-26T08:40:50.6767874Z http.https://github.com/.extraheader 2024-06-26T08:40:50.6790944Z Entering 'third_party/cudnn_frontend' 2024-06-26T08:40:50.6812472Z http.https://github.com/.extraheader 2024-06-26T08:40:50.6837053Z Entering 'third_party/cutlass' 2024-06-26T08:40:50.6857101Z http.https://github.com/.extraheader 2024-06-26T08:40:50.6887987Z Entering 'third_party/eigen' 2024-06-26T08:40:50.6909972Z http.https://github.com/.extraheader 2024-06-26T08:40:50.6936170Z Entering 'third_party/fbgemm' 2024-06-26T08:40:50.6957281Z http.https://github.com/.extraheader 2024-06-26T08:40:50.6979043Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T08:40:50.7000541Z http.https://github.com/.extraheader 2024-06-26T08:40:50.7025204Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T08:40:50.7047339Z http.https://github.com/.extraheader 2024-06-26T08:40:50.7073720Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T08:40:50.7095041Z http.https://github.com/.extraheader 2024-06-26T08:40:50.7123648Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T08:40:50.7144380Z http.https://github.com/.extraheader 2024-06-26T08:40:50.7169644Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T08:40:50.7191363Z http.https://github.com/.extraheader 2024-06-26T08:40:50.7215672Z Entering 'third_party/flatbuffers' 2024-06-26T08:40:50.7239147Z http.https://github.com/.extraheader 2024-06-26T08:40:50.7264578Z Entering 'third_party/fmt' 2024-06-26T08:40:50.7286061Z http.https://github.com/.extraheader 2024-06-26T08:40:50.7308812Z Entering 'third_party/foxi' 2024-06-26T08:40:50.7331072Z http.https://github.com/.extraheader 2024-06-26T08:40:50.7357272Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T08:40:50.7378002Z http.https://github.com/.extraheader 2024-06-26T08:40:50.7400403Z Entering 'third_party/gloo' 2024-06-26T08:40:50.7422206Z http.https://github.com/.extraheader 2024-06-26T08:40:50.7445170Z Entering 'third_party/googletest' 2024-06-26T08:40:50.7466431Z http.https://github.com/.extraheader 2024-06-26T08:40:50.7491638Z Entering 'third_party/ideep' 2024-06-26T08:40:50.7514087Z http.https://github.com/.extraheader 2024-06-26T08:40:50.7537041Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T08:40:50.7558802Z http.https://github.com/.extraheader 2024-06-26T08:40:50.7585547Z Entering 'third_party/ittapi' 2024-06-26T08:40:50.7608207Z http.https://github.com/.extraheader 2024-06-26T08:40:50.7631790Z Entering 'third_party/kineto' 2024-06-26T08:40:50.7653575Z http.https://github.com/.extraheader 2024-06-26T08:40:50.7675937Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T08:40:50.7696797Z http.https://github.com/.extraheader 2024-06-26T08:40:50.7719577Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T08:40:50.7741178Z http.https://github.com/.extraheader 2024-06-26T08:40:50.7766992Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T08:40:50.7788808Z http.https://github.com/.extraheader 2024-06-26T08:40:50.7812738Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T08:40:50.7835627Z http.https://github.com/.extraheader 2024-06-26T08:40:50.7858730Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T08:40:50.7880944Z http.https://github.com/.extraheader 2024-06-26T08:40:50.7901767Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T08:40:50.7923493Z http.https://github.com/.extraheader 2024-06-26T08:40:50.7948506Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T08:40:50.7970531Z http.https://github.com/.extraheader 2024-06-26T08:40:50.7996786Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T08:40:50.8018005Z http.https://github.com/.extraheader 2024-06-26T08:40:50.8041065Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T08:40:50.8061822Z http.https://github.com/.extraheader 2024-06-26T08:40:50.8085245Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T08:40:50.8107006Z http.https://github.com/.extraheader 2024-06-26T08:40:50.8133594Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T08:40:50.8156318Z http.https://github.com/.extraheader 2024-06-26T08:40:50.8178952Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T08:40:50.8200950Z http.https://github.com/.extraheader 2024-06-26T08:40:50.8227088Z Entering 'third_party/mimalloc' 2024-06-26T08:40:50.8248369Z http.https://github.com/.extraheader 2024-06-26T08:40:50.8270676Z Entering 'third_party/nccl/nccl' 2024-06-26T08:40:50.8294693Z http.https://github.com/.extraheader 2024-06-26T08:40:50.8319023Z Entering 'third_party/nlohmann' 2024-06-26T08:40:50.8340778Z http.https://github.com/.extraheader 2024-06-26T08:40:50.8366304Z Entering 'third_party/onnx' 2024-06-26T08:40:50.8389358Z http.https://github.com/.extraheader 2024-06-26T08:40:50.8426333Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T08:40:50.8450290Z http.https://github.com/.extraheader 2024-06-26T08:40:50.8473780Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T08:40:50.8496131Z http.https://github.com/.extraheader 2024-06-26T08:40:50.8522056Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T08:40:50.8544033Z http.https://github.com/.extraheader 2024-06-26T08:40:50.8568210Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T08:40:50.8589204Z http.https://github.com/.extraheader 2024-06-26T08:40:50.8613770Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T08:40:50.8634721Z http.https://github.com/.extraheader 2024-06-26T08:40:50.8659408Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T08:40:50.8680260Z http.https://github.com/.extraheader 2024-06-26T08:40:50.8705181Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T08:40:50.8727935Z http.https://github.com/.extraheader 2024-06-26T08:40:50.8753033Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T08:40:50.8774331Z http.https://github.com/.extraheader 2024-06-26T08:40:50.8797772Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T08:40:50.8819180Z http.https://github.com/.extraheader 2024-06-26T08:40:50.8842659Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T08:40:50.8863759Z http.https://github.com/.extraheader 2024-06-26T08:40:50.8886361Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T08:40:50.8907553Z http.https://github.com/.extraheader 2024-06-26T08:40:50.8933405Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T08:40:50.8955441Z http.https://github.com/.extraheader 2024-06-26T08:40:50.8981275Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T08:40:50.9001903Z http.https://github.com/.extraheader 2024-06-26T08:40:50.9038823Z Entering 'third_party/pocketfft' 2024-06-26T08:40:50.9061763Z http.https://github.com/.extraheader 2024-06-26T08:40:50.9087909Z Entering 'third_party/protobuf' 2024-06-26T08:40:50.9109444Z http.https://github.com/.extraheader 2024-06-26T08:40:50.9136453Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T08:40:50.9158151Z http.https://github.com/.extraheader 2024-06-26T08:40:50.9181225Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T08:40:50.9203532Z http.https://github.com/.extraheader 2024-06-26T08:40:50.9229243Z Entering 'third_party/psimd' 2024-06-26T08:40:50.9251642Z http.https://github.com/.extraheader 2024-06-26T08:40:50.9275549Z Entering 'third_party/pthreadpool' 2024-06-26T08:40:50.9297822Z http.https://github.com/.extraheader 2024-06-26T08:40:50.9320993Z Entering 'third_party/pybind11' 2024-06-26T08:40:50.9342608Z http.https://github.com/.extraheader 2024-06-26T08:40:50.9367865Z Entering 'third_party/python-peachpy' 2024-06-26T08:40:50.9388660Z http.https://github.com/.extraheader 2024-06-26T08:40:50.9411221Z Entering 'third_party/sleef' 2024-06-26T08:40:50.9433072Z http.https://github.com/.extraheader 2024-06-26T08:40:50.9455474Z Entering 'third_party/tensorpipe' 2024-06-26T08:40:50.9478490Z http.https://github.com/.extraheader 2024-06-26T08:40:50.9500261Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T08:40:50.9522189Z http.https://github.com/.extraheader 2024-06-26T08:40:50.9546799Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T08:40:50.9569412Z http.https://github.com/.extraheader 2024-06-26T08:40:50.9593507Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T08:40:50.9615871Z http.https://github.com/.extraheader 2024-06-26T08:40:50.9638446Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T08:40:50.9660069Z http.https://github.com/.extraheader 2024-06-26T08:40:50.9681704Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T08:40:50.9703559Z http.https://github.com/.extraheader 2024-06-26T08:40:50.9803838Z A job completed hook has been configured by the self-hosted runner administrator 2024-06-26T08:40:50.9821138Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T08:40:50.9827741Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:40:50.9828256Z ##[endgroup] 2024-06-26T08:40:52.0800426Z Cleaning up orphan processes