2024-08-20T22:05:17.6068775Z Current runner version: '2.319.1' 2024-08-20T22:05:17.6075042Z Runner name: 'i-029e43603fd06f026' 2024-08-20T22:05:17.6075858Z Runner group name: 'Default' 2024-08-20T22:05:17.6076806Z Machine name: 'ip-10-0-49-173' 2024-08-20T22:05:17.6094045Z Testing runner upgrade compatibility 2024-08-20T22:05:17.7698214Z ##[group]GITHUB_TOKEN Permissions 2024-08-20T22:05:17.7700482Z Actions: read 2024-08-20T22:05:17.7701044Z Attestations: read 2024-08-20T22:05:17.7701663Z Checks: read 2024-08-20T22:05:17.7702158Z Contents: read 2024-08-20T22:05:17.7702653Z Deployments: read 2024-08-20T22:05:17.7703235Z Discussions: read 2024-08-20T22:05:17.7703747Z Issues: read 2024-08-20T22:05:17.7704218Z Metadata: read 2024-08-20T22:05:17.7704773Z Packages: read 2024-08-20T22:05:17.7705291Z Pages: read 2024-08-20T22:05:17.7705785Z PullRequests: read 2024-08-20T22:05:17.7706438Z RepositoryProjects: read 2024-08-20T22:05:17.7707056Z SecurityEvents: read 2024-08-20T22:05:17.7707601Z Statuses: read 2024-08-20T22:05:17.7708335Z ##[endgroup] 2024-08-20T22:05:17.7711752Z Secret source: Actions 2024-08-20T22:05:17.7712612Z Prepare workflow directory 2024-08-20T22:05:17.9084347Z Prepare all required actions 2024-08-20T22:05:17.9253317Z Getting action download info 2024-08-20T22:05:18.1445405Z Download action repository 'pytorch/test-infra@main' (SHA:0c3a2634aaa2f638c8f640e743f03d696ce1191f) 2024-08-20T22:05:18.7203761Z Download action repository 'pytorch/pytorch@main' (SHA:1ae5d5bb62141d7c9b1b0b66c66a462b4e10b1f2) 2024-08-20T22:05:22.7282621Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-20T22:05:22.9149412Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-20T22:05:23.1491917Z Getting action download info 2024-08-20T22:05:23.2684248Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-20T22:05:23.4497346Z Getting action download info 2024-08-20T22:05:23.5760796Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-20T22:05:23.7370696Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/133712 (40ec5f6ddd9787aca0449b24128343ff4c4a88b3) 2024-08-20T22:05:23.7372847Z ##[group] Inputs 2024-08-20T22:05:23.7373320Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:05:23.7380436Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:05:23.7388420Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:05:23.7389604Z sync-tag: 2024-08-20T22:05:23.7390598Z timeout-minutes: 240 2024-08-20T22:05:23.7390942Z use-gha: 2024-08-20T22:05:23.7391234Z dashboard-tag: 2024-08-20T22:05:23.7391569Z s3-bucket: gha-artifacts 2024-08-20T22:05:23.7391925Z aws-role-to-assume: 2024-08-20T22:05:23.7392267Z ##[endgroup] 2024-08-20T22:05:23.7393349Z Complete job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:05:23.8056373Z A job started hook has been configured by the self-hosted runner administrator 2024-08-20T22:05:23.8197222Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-20T22:05:23.8209499Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:05:23.8210049Z ##[endgroup] 2024-08-20T22:05:25.3635637Z Runner Type: linux.g5.4xlarge.nvidia.gpu 2024-08-20T22:05:25.3636149Z Instance Type: g5.4xlarge 2024-08-20T22:05:25.3636836Z AMI Name: al2023-ami-2023.5.20240701.0-kernel-6.1-x86_64 2024-08-20T22:05:25.3637386Z AMI ID: ami-06c68f701d8090592 2024-08-20T22:05:31.1454689Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-20T22:05:31.1455259Z with: 2024-08-20T22:05:31.1456154Z github-secret: *** 2024-08-20T22:05:31.1457133Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:05:31.1458148Z activate-with-label: false 2024-08-20T22:05:31.1458527Z label: with-ssh 2024-08-20T22:05:31.1458864Z remove-existing-keys: true 2024-08-20T22:05:31.1459237Z fail-silently: true 2024-08-20T22:05:31.1459562Z env: 2024-08-20T22:05:31.1460060Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:05:31.1460436Z ##[endgroup] 2024-08-20T22:05:31.2354802Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-20T22:05:31.2357946Z ciflow reference detected, attempting to extract PR number 2024-08-20T22:05:31.5009764Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-08-20T22:05:31.5724902Z No SSH keys found for user pytorch-bot[bot] 2024-08-20T22:05:31.5725530Z Grabbing public ssh keys from https://github.com/XuehaiPan.keys 2024-08-20T22:05:31.6651179Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-20T22:05:31.6666577Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-20T22:05:31.6691865Z Login using: ssh ec2-user@ec2-3-87-88-140.compute-1.amazonaws.com 2024-08-20T22:05:31.6692635Z All testing is done inside the container, to start an interactive session run: 2024-08-20T22:05:31.6693434Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:05:31.6822888Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-20T22:05:31.6823446Z with: 2024-08-20T22:05:31.6823728Z submodules: recursive 2024-08-20T22:05:31.6824066Z fetch-depth: 0 2024-08-20T22:05:31.6824361Z env: 2024-08-20T22:05:31.6824637Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:05:31.6824985Z ##[endgroup] 2024-08-20T22:05:31.7013296Z ##[group]Run retry () { 2024-08-20T22:05:31.7013698Z retry () { 2024-08-20T22:05:31.7014218Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-20T22:05:31.7014772Z } 2024-08-20T22:05:31.7015074Z echo "${GITHUB_WORKSPACE}" 2024-08-20T22:05:31.7015504Z if [ -z "${NO_SUDO}" ]; then 2024-08-20T22:05:31.7015988Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:05:31.7016447Z else 2024-08-20T22:05:31.7016788Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:05:31.7017217Z fi 2024-08-20T22:05:31.7017805Z mkdir "${GITHUB_WORKSPACE}" 2024-08-20T22:05:31.7027809Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:05:31.7028326Z env: 2024-08-20T22:05:31.7028609Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:05:31.7028952Z NO_SUDO: 2024-08-20T22:05:31.7029231Z ##[endgroup] 2024-08-20T22:05:31.7060463Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:05:31.8636367Z ##[group]Run malfet/checkout@silent-checkout 2024-08-20T22:05:31.8636802Z with: 2024-08-20T22:05:31.8637133Z ref: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:05:31.8637579Z fetch-depth: 0 2024-08-20T22:05:31.8637891Z submodules: recursive 2024-08-20T22:05:31.8638226Z quiet-checkout: true 2024-08-20T22:05:31.8638577Z repository: pytorch/pytorch 2024-08-20T22:05:31.8639061Z token: *** 2024-08-20T22:05:31.8639355Z ssh-strict: true 2024-08-20T22:05:31.8639679Z persist-credentials: true 2024-08-20T22:05:31.8640040Z clean: true 2024-08-20T22:05:31.8640396Z sparse-checkout-cone-mode: true 2024-08-20T22:05:31.8640788Z lfs: false 2024-08-20T22:05:31.8641089Z set-safe-directory: true 2024-08-20T22:05:31.8641433Z env: 2024-08-20T22:05:31.8641711Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:05:31.8642168Z ##[endgroup] 2024-08-20T22:05:31.9624032Z Syncing repository: pytorch/pytorch 2024-08-20T22:05:31.9625590Z ##[group]Getting Git version info 2024-08-20T22:05:31.9626402Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:05:31.9627257Z [command]/usr/bin/git version 2024-08-20T22:05:31.9627619Z git version 2.40.1 2024-08-20T22:05:31.9631548Z ##[endgroup] 2024-08-20T22:05:31.9645258Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/939fe433-b0b3-4fc8-a249-8f4eb60cacbe/.gitconfig' 2024-08-20T22:05:31.9652018Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/939fe433-b0b3-4fc8-a249-8f4eb60cacbe' before making global git config changes 2024-08-20T22:05:31.9653311Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T22:05:31.9656574Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:05:31.9712137Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:05:31.9716160Z ##[group]Initializing the repository 2024-08-20T22:05:31.9718802Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:05:31.9763846Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-20T22:05:31.9764760Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-20T22:05:31.9765596Z hint: of your new repositories, which will suppress this warning, call: 2024-08-20T22:05:31.9766174Z hint: 2024-08-20T22:05:31.9766637Z hint: git config --global init.defaultBranch 2024-08-20T22:05:31.9767125Z hint: 2024-08-20T22:05:31.9767988Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-20T22:05:31.9768938Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-20T22:05:31.9769697Z hint: 2024-08-20T22:05:31.9770046Z hint: git branch -m 2024-08-20T22:05:31.9770811Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-20T22:05:31.9777427Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-20T22:05:31.9820541Z ##[endgroup] 2024-08-20T22:05:31.9821142Z ##[group]Disabling automatic garbage collection 2024-08-20T22:05:31.9823468Z [command]/usr/bin/git config --local gc.auto 0 2024-08-20T22:05:31.9865076Z ##[endgroup] 2024-08-20T22:05:31.9865597Z ##[group]Setting up auth 2024-08-20T22:05:31.9870613Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T22:05:31.9913765Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T22:05:32.0278557Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T22:05:32.0320345Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T22:05:32.0680471Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:05:32.0744515Z ##[endgroup] 2024-08-20T22:05:32.0745092Z ##[group]Fetching the repository 2024-08-20T22:05:32.0750763Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-20T22:05:35.0492471Z remote: Enumerating objects: 1008509 2024-08-20T22:05:35.0493040Z remote: Enumerating objects: 1010208, done. 2024-08-20T22:05:35.0493871Z remote: Counting objects: 0% (1/1699) 2024-08-20T22:05:35.0496988Z remote: Counting objects: 1% (17/1699) 2024-08-20T22:05:35.0498174Z remote: Counting objects: 2% (34/1699) 2024-08-20T22:05:35.0498868Z remote: Counting objects: 3% (51/1699) 2024-08-20T22:05:35.0499430Z remote: Counting objects: 4% (68/1699) 2024-08-20T22:05:35.0499929Z remote: Counting objects: 5% (85/1699) 2024-08-20T22:05:35.0500435Z remote: Counting objects: 6% (102/1699) 2024-08-20T22:05:35.0500987Z remote: Counting objects: 7% (119/1699) 2024-08-20T22:05:35.0501676Z remote: Counting objects: 8% (136/1699) 2024-08-20T22:05:35.0502380Z remote: Counting objects: 9% (153/1699) 2024-08-20T22:05:35.0503041Z remote: Counting objects: 10% (170/1699) 2024-08-20T22:05:35.0503689Z remote: Counting objects: 11% (187/1699) 2024-08-20T22:05:35.0504274Z remote: Counting objects: 12% (204/1699) 2024-08-20T22:05:35.0504780Z remote: Counting objects: 13% (221/1699) 2024-08-20T22:05:35.0505272Z remote: Counting objects: 14% (238/1699) 2024-08-20T22:05:35.0505755Z remote: Counting objects: 15% (255/1699) 2024-08-20T22:05:35.0506248Z remote: Counting objects: 16% (272/1699) 2024-08-20T22:05:35.0506741Z remote: Counting objects: 17% (289/1699) 2024-08-20T22:05:35.0507229Z remote: Counting objects: 18% (306/1699) 2024-08-20T22:05:35.0507714Z remote: Counting objects: 19% (323/1699) 2024-08-20T22:05:35.0508211Z remote: Counting objects: 20% (340/1699) 2024-08-20T22:05:35.0508706Z remote: Counting objects: 21% (357/1699) 2024-08-20T22:05:35.0509199Z remote: Counting objects: 22% (374/1699) 2024-08-20T22:05:35.0509685Z remote: Counting objects: 23% (391/1699) 2024-08-20T22:05:35.0510184Z remote: Counting objects: 24% (408/1699) 2024-08-20T22:05:35.0510679Z remote: Counting objects: 25% (425/1699) 2024-08-20T22:05:35.0511577Z remote: Counting objects: 26% (442/1699) 2024-08-20T22:05:35.0512120Z remote: Counting objects: 27% (459/1699) 2024-08-20T22:05:35.0512777Z remote: Counting objects: 28% (476/1699) 2024-08-20T22:05:35.0513438Z remote: Counting objects: 29% (493/1699) 2024-08-20T22:05:35.0514081Z remote: Counting objects: 30% (510/1699) 2024-08-20T22:05:35.0514724Z remote: Counting objects: 31% (527/1699) 2024-08-20T22:05:35.0515372Z remote: Counting objects: 32% (544/1699) 2024-08-20T22:05:35.0515974Z remote: Counting objects: 33% (561/1699) 2024-08-20T22:05:35.0516677Z remote: Counting objects: 34% (578/1699) 2024-08-20T22:05:35.0517432Z remote: Counting objects: 35% (595/1699) 2024-08-20T22:05:35.0518145Z remote: Counting objects: 36% (612/1699) 2024-08-20T22:05:35.0518645Z remote: Counting objects: 37% (629/1699) 2024-08-20T22:05:35.0519135Z remote: Counting objects: 38% (646/1699) 2024-08-20T22:05:35.0519847Z remote: Counting objects: 39% (663/1699) 2024-08-20T22:05:35.0520346Z remote: Counting objects: 40% (680/1699) 2024-08-20T22:05:35.0520893Z remote: Counting objects: 41% (697/1699) 2024-08-20T22:05:35.0521391Z remote: Counting objects: 42% (714/1699) 2024-08-20T22:05:35.0521879Z remote: Counting objects: 43% (731/1699) 2024-08-20T22:05:35.0522374Z remote: Counting objects: 44% (748/1699) 2024-08-20T22:05:35.0522899Z remote: Counting objects: 45% (765/1699) 2024-08-20T22:05:35.0523515Z remote: Counting objects: 46% (782/1699) 2024-08-20T22:05:35.0524005Z remote: Counting objects: 47% (799/1699) 2024-08-20T22:05:35.0524499Z remote: Counting objects: 48% (816/1699) 2024-08-20T22:05:35.0524998Z remote: Counting objects: 49% (833/1699) 2024-08-20T22:05:35.0525493Z remote: Counting objects: 50% (850/1699) 2024-08-20T22:05:35.0525990Z remote: Counting objects: 51% (867/1699) 2024-08-20T22:05:35.0526496Z remote: Counting objects: 52% (884/1699) 2024-08-20T22:05:35.0526993Z remote: Counting objects: 53% (901/1699) 2024-08-20T22:05:35.0527486Z remote: Counting objects: 54% (918/1699) 2024-08-20T22:05:35.0527974Z remote: Counting objects: 55% (935/1699) 2024-08-20T22:05:35.0528471Z remote: Counting objects: 56% (952/1699) 2024-08-20T22:05:35.0528967Z remote: Counting objects: 57% (969/1699) 2024-08-20T22:05:35.0529460Z remote: Counting objects: 58% (986/1699) 2024-08-20T22:05:35.0529957Z remote: Counting objects: 59% (1003/1699) 2024-08-20T22:05:35.0530478Z remote: Counting objects: 60% (1020/1699) 2024-08-20T22:05:35.0530994Z remote: Counting objects: 61% (1037/1699) 2024-08-20T22:05:35.0531507Z remote: Counting objects: 62% (1054/1699) 2024-08-20T22:05:35.0532031Z remote: Counting objects: 63% (1071/1699) 2024-08-20T22:05:35.0532541Z remote: Counting objects: 64% (1088/1699) 2024-08-20T22:05:35.0533056Z remote: Counting objects: 65% (1105/1699) 2024-08-20T22:05:35.0533564Z remote: Counting objects: 66% (1122/1699) 2024-08-20T22:05:35.0534071Z remote: Counting objects: 67% (1139/1699) 2024-08-20T22:05:35.0534575Z remote: Counting objects: 68% (1156/1699) 2024-08-20T22:05:35.0535082Z remote: Counting objects: 69% (1173/1699) 2024-08-20T22:05:35.0535588Z remote: Counting objects: 70% (1190/1699) 2024-08-20T22:05:35.0536095Z remote: Counting objects: 71% (1207/1699) 2024-08-20T22:05:35.0536597Z remote: Counting objects: 72% (1224/1699) 2024-08-20T22:05:35.0537102Z remote: Counting objects: 73% (1241/1699) 2024-08-20T22:05:35.0537607Z remote: Counting objects: 74% (1258/1699) 2024-08-20T22:05:35.0538112Z remote: Counting objects: 75% (1275/1699) 2024-08-20T22:05:35.0538614Z remote: Counting objects: 76% (1292/1699) 2024-08-20T22:05:35.0539215Z remote: Counting objects: 77% (1309/1699) 2024-08-20T22:05:35.0539727Z remote: Counting objects: 78% (1326/1699) 2024-08-20T22:05:35.0540233Z remote: Counting objects: 79% (1343/1699) 2024-08-20T22:05:35.0540734Z remote: Counting objects: 80% (1360/1699) 2024-08-20T22:05:35.0541240Z remote: Counting objects: 81% (1377/1699) 2024-08-20T22:05:35.0541746Z remote: Counting objects: 82% (1394/1699) 2024-08-20T22:05:35.0542251Z remote: Counting objects: 83% (1411/1699) 2024-08-20T22:05:35.0542754Z remote: Counting objects: 84% (1428/1699) 2024-08-20T22:05:35.0543261Z remote: Counting objects: 85% (1445/1699) 2024-08-20T22:05:35.0543764Z remote: Counting objects: 86% (1462/1699) 2024-08-20T22:05:35.0544271Z remote: Counting objects: 87% (1479/1699) 2024-08-20T22:05:35.0544772Z remote: Counting objects: 88% (1496/1699) 2024-08-20T22:05:35.0545277Z remote: Counting objects: 89% (1513/1699) 2024-08-20T22:05:35.0545880Z remote: Counting objects: 90% (1530/1699) 2024-08-20T22:05:35.0546386Z remote: Counting objects: 91% (1547/1699) 2024-08-20T22:05:35.0546889Z remote: Counting objects: 92% (1564/1699) 2024-08-20T22:05:35.0547397Z remote: Counting objects: 93% (1581/1699) 2024-08-20T22:05:35.0547902Z remote: Counting objects: 94% (1598/1699) 2024-08-20T22:05:35.0548409Z remote: Counting objects: 95% (1615/1699) 2024-08-20T22:05:35.0548911Z remote: Counting objects: 96% (1632/1699) 2024-08-20T22:05:35.0549421Z remote: Counting objects: 97% (1649/1699) 2024-08-20T22:05:35.0549929Z remote: Counting objects: 98% (1666/1699) 2024-08-20T22:05:35.0550435Z remote: Counting objects: 99% (1683/1699) 2024-08-20T22:05:35.0550986Z remote: Counting objects: 100% (1699/1699) 2024-08-20T22:05:35.0551532Z remote: Counting objects: 100% (1699/1699), done. 2024-08-20T22:05:35.0679170Z remote: Compressing objects: 0% (1/826) 2024-08-20T22:05:35.0961549Z remote: Compressing objects: 1% (9/826) 2024-08-20T22:05:35.1248597Z remote: Compressing objects: 2% (17/826) 2024-08-20T22:05:35.1743701Z remote: Compressing objects: 3% (25/826) 2024-08-20T22:05:35.2557111Z remote: Compressing objects: 4% (34/826) 2024-08-20T22:05:35.3012675Z remote: Compressing objects: 5% (42/826) 2024-08-20T22:05:35.3659305Z remote: Compressing objects: 6% (50/826) 2024-08-20T22:05:35.4073698Z remote: Compressing objects: 7% (58/826) 2024-08-20T22:05:35.4340000Z remote: Compressing objects: 8% (67/826) 2024-08-20T22:05:35.4554235Z remote: Compressing objects: 9% (75/826) 2024-08-20T22:05:35.4792629Z remote: Compressing objects: 10% (83/826) 2024-08-20T22:05:35.5000012Z remote: Compressing objects: 11% (91/826) 2024-08-20T22:05:35.5194278Z remote: Compressing objects: 12% (100/826) 2024-08-20T22:05:35.5343063Z remote: Compressing objects: 13% (108/826) 2024-08-20T22:05:35.5458619Z remote: Compressing objects: 14% (116/826) 2024-08-20T22:05:35.5542755Z remote: Compressing objects: 15% (124/826) 2024-08-20T22:05:35.5633470Z remote: Compressing objects: 16% (133/826) 2024-08-20T22:05:35.5691506Z remote: Compressing objects: 17% (141/826) 2024-08-20T22:05:35.5733323Z remote: Compressing objects: 18% (149/826) 2024-08-20T22:05:35.5781436Z remote: Compressing objects: 19% (157/826) 2024-08-20T22:05:35.5811560Z remote: Compressing objects: 20% (166/826) 2024-08-20T22:05:35.5832694Z remote: Compressing objects: 21% (174/826) 2024-08-20T22:05:35.5836661Z remote: Compressing objects: 22% (182/826) 2024-08-20T22:05:35.5841291Z remote: Compressing objects: 23% (190/826) 2024-08-20T22:05:35.5849729Z remote: Compressing objects: 24% (199/826) 2024-08-20T22:05:35.5854555Z remote: Compressing objects: 25% (207/826) 2024-08-20T22:05:35.5864842Z remote: Compressing objects: 26% (215/826) 2024-08-20T22:05:35.5873354Z remote: Compressing objects: 27% (224/826) 2024-08-20T22:05:35.5899186Z remote: Compressing objects: 28% (232/826) 2024-08-20T22:05:35.5905591Z remote: Compressing objects: 29% (240/826) 2024-08-20T22:05:35.5917903Z remote: Compressing objects: 30% (248/826) 2024-08-20T22:05:35.5927310Z remote: Compressing objects: 31% (257/826) 2024-08-20T22:05:35.5932847Z remote: Compressing objects: 32% (265/826) 2024-08-20T22:05:35.5941622Z remote: Compressing objects: 33% (273/826) 2024-08-20T22:05:35.5948773Z remote: Compressing objects: 34% (281/826) 2024-08-20T22:05:35.5954873Z remote: Compressing objects: 35% (290/826) 2024-08-20T22:05:35.5959531Z remote: Compressing objects: 36% (298/826) 2024-08-20T22:05:35.5969537Z remote: Compressing objects: 37% (306/826) 2024-08-20T22:05:35.5976121Z remote: Compressing objects: 38% (314/826) 2024-08-20T22:05:35.5982500Z remote: Compressing objects: 39% (323/826) 2024-08-20T22:05:35.5987966Z remote: Compressing objects: 40% (331/826) 2024-08-20T22:05:35.5992877Z remote: Compressing objects: 41% (339/826) 2024-08-20T22:05:35.6003019Z remote: Compressing objects: 42% (347/826) 2024-08-20T22:05:35.6006615Z remote: Compressing objects: 43% (356/826) 2024-08-20T22:05:35.6013526Z remote: Compressing objects: 44% (364/826) 2024-08-20T22:05:35.6018035Z remote: Compressing objects: 45% (372/826) 2024-08-20T22:05:35.6022930Z remote: Compressing objects: 46% (380/826) 2024-08-20T22:05:35.6025735Z remote: Compressing objects: 47% (389/826) 2024-08-20T22:05:35.6028651Z remote: Compressing objects: 48% (397/826) 2024-08-20T22:05:35.6032127Z remote: Compressing objects: 49% (405/826) 2024-08-20T22:05:35.6034491Z remote: Compressing objects: 50% (413/826) 2024-08-20T22:05:35.6037600Z remote: Compressing objects: 51% (422/826) 2024-08-20T22:05:35.6039572Z remote: Compressing objects: 52% (430/826) 2024-08-20T22:05:35.6042251Z remote: Compressing objects: 53% (438/826) 2024-08-20T22:05:35.6043580Z remote: Compressing objects: 54% (447/826) 2024-08-20T22:05:35.6044645Z remote: Compressing objects: 55% (455/826) 2024-08-20T22:05:35.6045606Z remote: Compressing objects: 56% (463/826) 2024-08-20T22:05:35.6046915Z remote: Compressing objects: 57% (471/826) 2024-08-20T22:05:35.6047768Z remote: Compressing objects: 58% (480/826) 2024-08-20T22:05:35.6048572Z remote: Compressing objects: 59% (488/826) 2024-08-20T22:05:35.6049373Z remote: Compressing objects: 60% (496/826) 2024-08-20T22:05:35.6049911Z remote: Compressing objects: 61% (504/826) 2024-08-20T22:05:35.6050434Z remote: Compressing objects: 62% (513/826) 2024-08-20T22:05:35.6050968Z remote: Compressing objects: 63% (521/826) 2024-08-20T22:05:35.6052671Z remote: Compressing objects: 64% (529/826) 2024-08-20T22:05:35.6059352Z remote: Compressing objects: 65% (537/826) 2024-08-20T22:05:35.6064294Z remote: Compressing objects: 66% (546/826) 2024-08-20T22:05:35.6068561Z remote: Compressing objects: 67% (554/826) 2024-08-20T22:05:35.6072628Z remote: Compressing objects: 68% (562/826) 2024-08-20T22:05:35.6076984Z remote: Compressing objects: 69% (570/826) 2024-08-20T22:05:35.6081536Z remote: Compressing objects: 70% (579/826) 2024-08-20T22:05:35.6084706Z remote: Compressing objects: 71% (587/826) 2024-08-20T22:05:35.6087464Z remote: Compressing objects: 72% (595/826) 2024-08-20T22:05:35.6091443Z remote: Compressing objects: 73% (603/826) 2024-08-20T22:05:35.6094436Z remote: Compressing objects: 74% (612/826) 2024-08-20T22:05:35.6097223Z remote: Compressing objects: 75% (620/826) 2024-08-20T22:05:35.6099744Z remote: Compressing objects: 76% (628/826) 2024-08-20T22:05:35.6102544Z remote: Compressing objects: 77% (637/826) 2024-08-20T22:05:35.6104070Z remote: Compressing objects: 78% (645/826) 2024-08-20T22:05:35.6105483Z remote: Compressing objects: 79% (653/826) 2024-08-20T22:05:35.6107833Z remote: Compressing objects: 80% (661/826) 2024-08-20T22:05:35.6109295Z remote: Compressing objects: 81% (670/826) 2024-08-20T22:05:35.6111539Z remote: Compressing objects: 82% (678/826) 2024-08-20T22:05:35.6112685Z remote: Compressing objects: 83% (686/826) 2024-08-20T22:05:35.6114597Z remote: Compressing objects: 84% (694/826) 2024-08-20T22:05:35.6116076Z remote: Compressing objects: 85% (703/826) 2024-08-20T22:05:35.6117737Z remote: Compressing objects: 86% (711/826) 2024-08-20T22:05:35.6118912Z remote: Compressing objects: 87% (719/826) 2024-08-20T22:05:35.6121936Z remote: Compressing objects: 88% (727/826) 2024-08-20T22:05:35.6123414Z remote: Compressing objects: 89% (736/826) 2024-08-20T22:05:35.6125733Z remote: Compressing objects: 90% (744/826) 2024-08-20T22:05:35.6126479Z remote: Compressing objects: 91% (752/826) 2024-08-20T22:05:35.6127262Z remote: Compressing objects: 92% (760/826) 2024-08-20T22:05:35.6127935Z remote: Compressing objects: 93% (769/826) 2024-08-20T22:05:35.6128610Z remote: Compressing objects: 94% (777/826) 2024-08-20T22:05:35.6129982Z remote: Compressing objects: 95% (785/826) 2024-08-20T22:05:35.6132048Z remote: Compressing objects: 96% (793/826) 2024-08-20T22:05:35.6132596Z remote: Compressing objects: 97% (802/826) 2024-08-20T22:05:35.6133920Z remote: Compressing objects: 98% (810/826) 2024-08-20T22:05:35.6134596Z remote: Compressing objects: 99% (818/826) 2024-08-20T22:05:35.6135567Z remote: Compressing objects: 100% (826/826) 2024-08-20T22:05:35.6136145Z remote: Compressing objects: 100% (826/826), done. 2024-08-20T22:05:58.8784830Z remote: Total 1010208 (delta 1106), reused 1335 (delta 870), pack-reused 1008509 (from 1) 2024-08-20T22:06:22.4213782Z [command]/usr/bin/git rev-parse --verify --quiet 40ec5f6ddd9787aca0449b24128343ff4c4a88b3^{object} 2024-08-20T22:06:22.4249986Z 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:06:22.4255889Z ##[endgroup] 2024-08-20T22:06:22.4259125Z ##[group]Determining the checkout info 2024-08-20T22:06:22.4260158Z ##[endgroup] 2024-08-20T22:06:22.4262549Z ##[group]Checking out the ref 2024-08-20T22:06:22.4263345Z [command]/usr/bin/git checkout --quiet --force 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:06:24.1616467Z ##[endgroup] 2024-08-20T22:06:24.1619693Z ##[group]Setting up auth for fetching submodules 2024-08-20T22:06:24.1620690Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:06:24.1681744Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-20T22:06:24.1721526Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-20T22:06:24.1760850Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-20T22:06:24.1798569Z ##[endgroup] 2024-08-20T22:06:24.1799106Z ##[group]Fetching submodules 2024-08-20T22:06:24.1802242Z [command]/usr/bin/git submodule sync --recursive 2024-08-20T22:06:24.2186189Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-20T22:06:24.2553786Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-20T22:06:24.2555442Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-20T22:06:24.2558803Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-20T22:06:24.2563216Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-20T22:06:24.2566561Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-20T22:06:24.2569415Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-20T22:06:24.2572889Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-20T22:06:24.2576428Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-20T22:06:24.2579867Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-20T22:06:24.2583754Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-20T22:06:24.2587472Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-20T22:06:24.2591191Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-20T22:06:24.2595041Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-20T22:06:24.2599043Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-20T22:06:24.2602950Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-20T22:06:24.2607540Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:06:24.2614806Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-20T22:06:24.2619037Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-20T22:06:24.2623067Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-20T22:06:24.2627515Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-20T22:06:24.2631878Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-20T22:06:24.2636460Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-20T22:06:24.2640935Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-20T22:06:24.2645746Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-20T22:06:24.2650315Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-20T22:06:24.2655375Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-20T22:06:24.2660285Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-20T22:06:24.2665733Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-20T22:06:24.2670503Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-20T22:06:24.2675617Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-20T22:06:24.2680746Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-20T22:06:24.2686066Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-20T22:06:24.2691122Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-20T22:06:24.2699709Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-20T22:06:24.2733300Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-20T22:06:24.5852285Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-20T22:06:24.7760880Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-20T22:06:24.9496036Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-20T22:06:25.1849916Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-20T22:06:27.1112412Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-20T22:06:38.5897053Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-20T22:06:39.0039419Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-20T22:06:39.4546871Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-20T22:06:40.0886378Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-20T22:06:41.3476150Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-20T22:06:43.5213373Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-20T22:06:49.4244165Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-20T22:06:50.9652248Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-20T22:06:52.7342990Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-20T22:06:54.0553204Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-20T22:06:54.4587904Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-20T22:06:54.7952201Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-20T22:06:55.9601038Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-20T22:06:56.3612556Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-20T22:06:56.6124639Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-20T22:06:58.1833425Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-20T22:06:58.9392576Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-20T22:06:59.5908067Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-20T22:07:06.0576585Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-20T22:07:08.0918468Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-20T22:07:12.3853173Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-20T22:07:12.6356871Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-20T22:07:22.1748410Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-20T22:07:22.3697867Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-20T22:07:22.5789399Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-20T22:07:23.4947337Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-20T22:07:23.8325512Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-20T22:07:24.4659348Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-20T22:07:24.8795662Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-20T22:07:24.8944830Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-20T22:07:24.9062705Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-20T22:07:24.9370548Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-20T22:07:24.9813853Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-20T22:07:26.1609285Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-20T22:07:26.1899504Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-20T22:07:26.2428712Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-20T22:07:26.3554851Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-20T22:07:26.3957717Z Submodule path 'third_party/cudnn_frontend': checked out '23511ba176243f27b3b275da1fb3814ea805a171' 2024-08-20T22:07:27.0219000Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-20T22:07:27.3076107Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-20T22:07:27.3996064Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-20T22:07:27.4017586Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:07:27.4021043Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:07:27.4024529Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:07:27.4028152Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:07:27.4031892Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:07:27.4063512Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-20T22:07:28.6320586Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-20T22:07:29.2327546Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-20T22:07:31.4051275Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-20T22:07:32.4626644Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-20T22:07:32.8198608Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-20T22:07:32.9296563Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-20T22:07:33.4215680Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-20T22:07:33.4910278Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-20T22:07:33.5057328Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-20T22:07:33.6576944Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-20T22:07:33.7039148Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-20T22:07:33.7499714Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-20T22:07:33.7824190Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-20T22:07:33.8362653Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:07:33.8520793Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-20T22:07:33.8540520Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-20T22:07:33.8576400Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-20T22:07:47.0293734Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-20T22:07:47.0512135Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-20T22:07:47.1506364Z Submodule path 'third_party/kineto': checked out 'd9753139d181b9ff42872465aac0e5d3018be415' 2024-08-20T22:07:47.1525615Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:07:47.1528841Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:07:47.1532466Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:07:47.1565076Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-20T22:07:47.6858165Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-20T22:07:48.8866475Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-20T22:07:50.0313364Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-20T22:07:50.0333316Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:07:50.0336703Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:07:50.0340265Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:07:50.0343962Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:07:50.0347674Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:07:50.0351597Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:07:50.0355465Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:07:50.0359513Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:07:50.0391001Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-20T22:07:50.8967130Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-20T22:07:51.2540571Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-20T22:07:52.5063442Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-20T22:07:52.7967032Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-20T22:07:53.3054493Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-20T22:07:54.3874328Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-20T22:08:00.8742590Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-20T22:08:01.2806145Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-20T22:08:01.3036325Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-20T22:08:01.3466239Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-20T22:08:01.3631262Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-20T22:08:01.3650622Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:01.3681998Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-20T22:08:01.6432482Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-20T22:08:01.6651146Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-20T22:08:01.7127592Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-20T22:08:01.8346257Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-20T22:08:01.8547119Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-20T22:08:01.9000497Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-20T22:08:01.9662146Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-20T22:08:02.0118473Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-20T22:08:02.0417266Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-20T22:08:02.1668407Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-20T22:08:02.6774522Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-20T22:08:02.6809665Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:02.6813006Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:02.6845051Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-20T22:08:03.1858114Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-20T22:08:04.1578841Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-20T22:08:04.1984162Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-20T22:08:04.2886032Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-20T22:08:04.2909259Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:04.2912541Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:04.2915982Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:04.2919431Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:04.2923386Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:04.2927329Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:04.2930925Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:04.2934649Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:04.2964812Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-20T22:08:04.8175531Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-20T22:08:05.9148341Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-20T22:08:06.2543674Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-20T22:08:12.3584273Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-20T22:08:12.6215969Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-20T22:08:12.8102841Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-20T22:08:13.0998024Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-20T22:08:20.0192811Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-20T22:08:20.0658343Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-20T22:08:20.0844377Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-20T22:08:20.2078853Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-20T22:08:20.2244813Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-20T22:08:20.2431425Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-20T22:08:20.2632195Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-20T22:08:20.2651232Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:20.2654811Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:20.2684993Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-20T22:08:22.2011606Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-20T22:08:23.5452753Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-20T22:08:23.5988507Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:08:24.2436472Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-20T22:08:24.2579513Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-20T22:08:24.5736654Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-20T22:08:24.5761407Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:24.5765003Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:24.5796179Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-20T22:08:25.0628445Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-20T22:08:26.1398526Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-20T22:08:26.2215300Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-20T22:08:26.2334098Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-20T22:08:26.2485593Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-20T22:08:26.2926440Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-20T22:08:26.3263940Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-20T22:08:26.3740833Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-20T22:08:26.4083561Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-20T22:08:26.4103711Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:26.4107199Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:26.4110339Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:26.4113927Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:26.4145240Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-20T22:08:27.4913710Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-20T22:08:27.7220722Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-20T22:08:28.9527599Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-20T22:08:29.9368377Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-20T22:08:29.9558274Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-20T22:08:30.0334197Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-20T22:08:30.0680100Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-20T22:08:30.0698721Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:30.0729854Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-20T22:08:30.2940288Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-20T22:08:30.2990482Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-20T22:08:30.3356932Z Entering 'android/libs/fbjni' 2024-08-20T22:08:30.3409319Z Entering 'third_party/FP16' 2024-08-20T22:08:30.3462537Z Entering 'third_party/FXdiv' 2024-08-20T22:08:30.3514276Z Entering 'third_party/NNPACK' 2024-08-20T22:08:30.3566203Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:30.3618073Z Entering 'third_party/XNNPACK' 2024-08-20T22:08:30.3686484Z Entering 'third_party/benchmark' 2024-08-20T22:08:30.3742176Z Entering 'third_party/cpp-httplib' 2024-08-20T22:08:30.3794550Z Entering 'third_party/cpuinfo' 2024-08-20T22:08:30.3845881Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:08:30.3902090Z Entering 'third_party/cutlass' 2024-08-20T22:08:30.3960772Z Entering 'third_party/eigen' 2024-08-20T22:08:30.4014257Z Entering 'third_party/fbgemm' 2024-08-20T22:08:30.4066698Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:08:30.4116372Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:08:30.4167106Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:08:30.4223703Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:08:30.4273584Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:08:30.4326065Z Entering 'third_party/flatbuffers' 2024-08-20T22:08:30.4380895Z Entering 'third_party/fmt' 2024-08-20T22:08:30.4432368Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:30.4487665Z Entering 'third_party/gloo' 2024-08-20T22:08:30.4538917Z Entering 'third_party/googletest' 2024-08-20T22:08:30.4591494Z Entering 'third_party/ideep' 2024-08-20T22:08:30.4642621Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:08:30.4703063Z Entering 'third_party/ittapi' 2024-08-20T22:08:30.4753461Z Entering 'third_party/kineto' 2024-08-20T22:08:30.4805841Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:08:30.4854891Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:08:30.4907442Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:08:30.4964058Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:08:30.5015129Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:08:30.5066968Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:30.5121248Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:08:30.5173104Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:08:30.5223993Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:08:30.5277622Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:08:30.5335668Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:08:30.5387263Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:08:30.5438675Z Entering 'third_party/mimalloc' 2024-08-20T22:08:30.5495463Z Entering 'third_party/nccl/nccl' 2024-08-20T22:08:30.5546773Z Entering 'third_party/nlohmann' 2024-08-20T22:08:30.5605263Z Entering 'third_party/onnx' 2024-08-20T22:08:30.5670898Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:30.5722567Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:30.5776985Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:08:30.5829451Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:30.5879757Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:30.5930020Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:30.5980798Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:30.6033366Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:30.6084917Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:30.6134965Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:30.6184424Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:30.6236371Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:30.6291371Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:30.6367219Z Entering 'third_party/pocketfft' 2024-08-20T22:08:30.6421746Z Entering 'third_party/protobuf' 2024-08-20T22:08:30.6475730Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:30.6526635Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:30.6580816Z Entering 'third_party/psimd' 2024-08-20T22:08:30.6632384Z Entering 'third_party/pthreadpool' 2024-08-20T22:08:30.6682713Z Entering 'third_party/pybind11' 2024-08-20T22:08:30.6733772Z Entering 'third_party/python-peachpy' 2024-08-20T22:08:30.6785062Z Entering 'third_party/sleef' 2024-08-20T22:08:30.6834885Z Entering 'third_party/tensorpipe' 2024-08-20T22:08:30.6886716Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:30.6936768Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:30.6986706Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:30.7036616Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:30.7087330Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:30.7166064Z ##[endgroup] 2024-08-20T22:08:30.7166931Z ##[group]Persisting credentials for submodules 2024-08-20T22:08:30.7171711Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-20T22:08:30.7532910Z Entering 'android/libs/fbjni' 2024-08-20T22:08:30.7600959Z Entering 'third_party/FP16' 2024-08-20T22:08:30.7667267Z Entering 'third_party/FXdiv' 2024-08-20T22:08:30.7733457Z Entering 'third_party/NNPACK' 2024-08-20T22:08:30.7801343Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:30.7872579Z Entering 'third_party/XNNPACK' 2024-08-20T22:08:30.7953569Z Entering 'third_party/benchmark' 2024-08-20T22:08:30.8021364Z Entering 'third_party/cpp-httplib' 2024-08-20T22:08:30.8088097Z Entering 'third_party/cpuinfo' 2024-08-20T22:08:30.8157982Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:08:30.8225271Z Entering 'third_party/cutlass' 2024-08-20T22:08:30.8304209Z Entering 'third_party/eigen' 2024-08-20T22:08:30.8374270Z Entering 'third_party/fbgemm' 2024-08-20T22:08:30.8440671Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:08:30.8509007Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:08:30.8575598Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:08:30.8647212Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:08:30.8712512Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:08:30.8780149Z Entering 'third_party/flatbuffers' 2024-08-20T22:08:30.8848786Z Entering 'third_party/fmt' 2024-08-20T22:08:30.8915127Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:30.8982056Z Entering 'third_party/gloo' 2024-08-20T22:08:30.9048371Z Entering 'third_party/googletest' 2024-08-20T22:08:30.9115258Z Entering 'third_party/ideep' 2024-08-20T22:08:30.9179789Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:08:30.9252743Z Entering 'third_party/ittapi' 2024-08-20T22:08:30.9320031Z Entering 'third_party/kineto' 2024-08-20T22:08:30.9385560Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:08:30.9450120Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:08:30.9518389Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:08:30.9590769Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:08:30.9661404Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:08:30.9729779Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:30.9801715Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:08:30.9874265Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:08:30.9941977Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:08:31.0009940Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:08:31.0084428Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:08:31.0150348Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:08:31.0219965Z Entering 'third_party/mimalloc' 2024-08-20T22:08:31.0290357Z Entering 'third_party/nccl/nccl' 2024-08-20T22:08:31.0356891Z Entering 'third_party/nlohmann' 2024-08-20T22:08:31.0431527Z Entering 'third_party/onnx' 2024-08-20T22:08:31.0515608Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:31.0587916Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:31.0659754Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:08:31.0727283Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:31.0792970Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:31.0858046Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:31.0924066Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:31.0992916Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:31.1058022Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:31.1125841Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:31.1191244Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:31.1259686Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:31.1329788Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:31.1417714Z Entering 'third_party/pocketfft' 2024-08-20T22:08:31.1485330Z Entering 'third_party/protobuf' 2024-08-20T22:08:31.1553323Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:31.1619991Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:31.1691256Z Entering 'third_party/psimd' 2024-08-20T22:08:31.1758029Z Entering 'third_party/pthreadpool' 2024-08-20T22:08:31.1830934Z Entering 'third_party/pybind11' 2024-08-20T22:08:31.1897149Z Entering 'third_party/python-peachpy' 2024-08-20T22:08:31.1968452Z Entering 'third_party/sleef' 2024-08-20T22:08:31.2034795Z Entering 'third_party/tensorpipe' 2024-08-20T22:08:31.2100601Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:31.2166124Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:31.2231478Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:31.2297808Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:31.2360793Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:31.2459695Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-20T22:08:31.2822553Z Entering 'android/libs/fbjni' 2024-08-20T22:08:31.2885687Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-20T22:08:31.2907276Z Entering 'third_party/FP16' 2024-08-20T22:08:31.2973937Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-20T22:08:31.2994932Z Entering 'third_party/FXdiv' 2024-08-20T22:08:31.3056566Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-20T22:08:31.3078395Z Entering 'third_party/NNPACK' 2024-08-20T22:08:31.3143565Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-20T22:08:31.3165082Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:31.3226236Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-20T22:08:31.3247962Z Entering 'third_party/XNNPACK' 2024-08-20T22:08:31.3314049Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-20T22:08:31.3350050Z Entering 'third_party/benchmark' 2024-08-20T22:08:31.3411802Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:08:31.3434119Z Entering 'third_party/cpp-httplib' 2024-08-20T22:08:31.3495329Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-20T22:08:31.3516589Z Entering 'third_party/cpuinfo' 2024-08-20T22:08:31.3582347Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:08:31.3603890Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:08:31.3666766Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-20T22:08:31.3688326Z Entering 'third_party/cutlass' 2024-08-20T22:08:31.3750311Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:08:31.3779125Z Entering 'third_party/eigen' 2024-08-20T22:08:31.3840237Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-20T22:08:31.3865880Z Entering 'third_party/fbgemm' 2024-08-20T22:08:31.3927344Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-20T22:08:31.3946802Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:08:31.4008724Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-20T22:08:31.4029546Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:08:31.4090804Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:08:31.4111754Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:08:31.4175722Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:08:31.4202965Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:08:31.4264273Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:31.4285008Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:08:31.4345664Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-20T22:08:31.4370174Z Entering 'third_party/flatbuffers' 2024-08-20T22:08:31.4431721Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-20T22:08:31.4455271Z Entering 'third_party/fmt' 2024-08-20T22:08:31.4521447Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:08:31.4543172Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:31.4604583Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-20T22:08:31.4625887Z Entering 'third_party/gloo' 2024-08-20T22:08:31.4687028Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-20T22:08:31.4708614Z Entering 'third_party/googletest' 2024-08-20T22:08:31.4770688Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:31.4792530Z Entering 'third_party/ideep' 2024-08-20T22:08:31.4853977Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-20T22:08:31.4873066Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:08:31.4935073Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-20T22:08:31.4964760Z Entering 'third_party/ittapi' 2024-08-20T22:08:31.5027772Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-20T22:08:31.5048474Z Entering 'third_party/kineto' 2024-08-20T22:08:31.5111073Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-20T22:08:31.5130722Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:08:31.5193341Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-20T22:08:31.5212595Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:08:31.5274671Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-20T22:08:31.5296586Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:08:31.5358635Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-20T22:08:31.5379189Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:08:31.5440989Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:08:31.5464611Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:08:31.5526789Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-20T22:08:31.5544972Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:31.5608508Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-20T22:08:31.5633082Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:08:31.5693737Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-20T22:08:31.5714239Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:08:31.5776318Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:31.5796769Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:08:31.5858909Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-20T22:08:31.5881735Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:08:31.5948409Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-20T22:08:31.5971908Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:08:31.6032868Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-20T22:08:31.6052955Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:08:31.6113695Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-20T22:08:31.6136406Z Entering 'third_party/mimalloc' 2024-08-20T22:08:31.6199119Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-20T22:08:31.6222363Z Entering 'third_party/nccl/nccl' 2024-08-20T22:08:31.6284054Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-20T22:08:31.6305485Z Entering 'third_party/nlohmann' 2024-08-20T22:08:31.6366441Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-20T22:08:31.6389210Z Entering 'third_party/onnx' 2024-08-20T22:08:31.6454495Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-20T22:08:31.6489233Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:31.6552582Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:08:31.6573112Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:31.6635168Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:08:31.6662934Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:08:31.6724543Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-20T22:08:31.6745240Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:31.6805874Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:08:31.6825713Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:31.6888228Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:31.6908297Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:31.6968847Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-20T22:08:31.6988681Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:31.7048490Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-20T22:08:31.7070737Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:31.7131130Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-20T22:08:31.7150563Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:31.7210689Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-20T22:08:31.7230401Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:31.7291490Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-20T22:08:31.7310550Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:31.7371800Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-20T22:08:31.7393734Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:31.7454912Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-20T22:08:31.7477959Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:31.7538711Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-20T22:08:31.7582978Z Entering 'third_party/pocketfft' 2024-08-20T22:08:31.7643790Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-20T22:08:31.7665297Z Entering 'third_party/protobuf' 2024-08-20T22:08:31.7726241Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-20T22:08:31.7748635Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:31.7811301Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:08:31.7830545Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:31.7891877Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:31.7917095Z Entering 'third_party/psimd' 2024-08-20T22:08:31.7983200Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-20T22:08:31.8004680Z Entering 'third_party/pthreadpool' 2024-08-20T22:08:31.8065939Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-20T22:08:31.8087221Z Entering 'third_party/pybind11' 2024-08-20T22:08:31.8148154Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:08:31.8170483Z Entering 'third_party/python-peachpy' 2024-08-20T22:08:31.8231159Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-20T22:08:31.8252209Z Entering 'third_party/sleef' 2024-08-20T22:08:31.8313125Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-20T22:08:31.8334196Z Entering 'third_party/tensorpipe' 2024-08-20T22:08:31.8398718Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-20T22:08:31.8419768Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:31.8481210Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:31.8501758Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:31.8563390Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-20T22:08:31.8583390Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:31.8643011Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-20T22:08:31.8663875Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:31.8724194Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:08:31.8742594Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:31.8805955Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-20T22:08:31.9922592Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-20T22:08:32.0289869Z Entering 'android/libs/fbjni' 2024-08-20T22:08:32.0340716Z Entering 'third_party/FP16' 2024-08-20T22:08:32.0393516Z Entering 'third_party/FXdiv' 2024-08-20T22:08:32.0445678Z Entering 'third_party/NNPACK' 2024-08-20T22:08:32.0503365Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:32.0555304Z Entering 'third_party/XNNPACK' 2024-08-20T22:08:32.0625274Z Entering 'third_party/benchmark' 2024-08-20T22:08:32.0677857Z Entering 'third_party/cpp-httplib' 2024-08-20T22:08:32.0729305Z Entering 'third_party/cpuinfo' 2024-08-20T22:08:32.0781997Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:08:32.0833461Z Entering 'third_party/cutlass' 2024-08-20T22:08:32.0895342Z Entering 'third_party/eigen' 2024-08-20T22:08:32.0948395Z Entering 'third_party/fbgemm' 2024-08-20T22:08:32.1004776Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:08:32.1055945Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:08:32.1106673Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:08:32.1163993Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:08:32.1211958Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:08:32.1263715Z Entering 'third_party/flatbuffers' 2024-08-20T22:08:32.1318560Z Entering 'third_party/fmt' 2024-08-20T22:08:32.1371355Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:32.1423133Z Entering 'third_party/gloo' 2024-08-20T22:08:32.1475253Z Entering 'third_party/googletest' 2024-08-20T22:08:32.1527274Z Entering 'third_party/ideep' 2024-08-20T22:08:32.1581801Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:08:32.1643510Z Entering 'third_party/ittapi' 2024-08-20T22:08:32.1700383Z Entering 'third_party/kineto' 2024-08-20T22:08:32.1751328Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:08:32.1802456Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:08:32.1857125Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:08:32.1913622Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:08:32.1965367Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:08:32.2014638Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:32.2069344Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:08:32.2121289Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:08:32.2175683Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:08:32.2226405Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:08:32.2280284Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:08:32.2330421Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:08:32.2383842Z Entering 'third_party/mimalloc' 2024-08-20T22:08:32.2435902Z Entering 'third_party/nccl/nccl' 2024-08-20T22:08:32.2493859Z Entering 'third_party/nlohmann' 2024-08-20T22:08:32.2546867Z Entering 'third_party/onnx' 2024-08-20T22:08:32.2616999Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:32.2669376Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:32.2725562Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:08:32.2780478Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:32.2829638Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:32.2880279Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:32.2934085Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:32.2984898Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:32.3034484Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:32.3084896Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:32.3132124Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:32.3184291Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:32.3237431Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:32.3307972Z Entering 'third_party/pocketfft' 2024-08-20T22:08:32.3358967Z Entering 'third_party/protobuf' 2024-08-20T22:08:32.3412034Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:32.3461546Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:32.3516012Z Entering 'third_party/psimd' 2024-08-20T22:08:32.3567812Z Entering 'third_party/pthreadpool' 2024-08-20T22:08:32.3622674Z Entering 'third_party/pybind11' 2024-08-20T22:08:32.3673785Z Entering 'third_party/python-peachpy' 2024-08-20T22:08:32.3724245Z Entering 'third_party/sleef' 2024-08-20T22:08:32.3776745Z Entering 'third_party/tensorpipe' 2024-08-20T22:08:32.3826571Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:32.3876807Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:32.3926476Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:32.3976923Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:32.4024517Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:32.4099992Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-20T22:08:32.4467961Z Entering 'android/libs/fbjni' 2024-08-20T22:08:32.4518785Z Entering 'third_party/FP16' 2024-08-20T22:08:32.4571329Z Entering 'third_party/FXdiv' 2024-08-20T22:08:32.4623357Z Entering 'third_party/NNPACK' 2024-08-20T22:08:32.4675790Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:32.4728105Z Entering 'third_party/XNNPACK' 2024-08-20T22:08:32.4799732Z Entering 'third_party/benchmark' 2024-08-20T22:08:32.4853095Z Entering 'third_party/cpp-httplib' 2024-08-20T22:08:32.4906624Z Entering 'third_party/cpuinfo' 2024-08-20T22:08:32.4958818Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:08:32.5010830Z Entering 'third_party/cutlass' 2024-08-20T22:08:32.5070325Z Entering 'third_party/eigen' 2024-08-20T22:08:32.5124654Z Entering 'third_party/fbgemm' 2024-08-20T22:08:32.5175390Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:08:32.5226144Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:08:32.5276705Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:08:32.5333499Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:08:32.5384750Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:08:32.5437337Z Entering 'third_party/flatbuffers' 2024-08-20T22:08:32.5491763Z Entering 'third_party/fmt' 2024-08-20T22:08:32.5542727Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:32.5594719Z Entering 'third_party/gloo' 2024-08-20T22:08:32.5645935Z Entering 'third_party/googletest' 2024-08-20T22:08:32.5697535Z Entering 'third_party/ideep' 2024-08-20T22:08:32.5746599Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:08:32.5809527Z Entering 'third_party/ittapi' 2024-08-20T22:08:32.5862300Z Entering 'third_party/kineto' 2024-08-20T22:08:32.5913987Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:08:32.5963512Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:08:32.6017432Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:08:32.6069248Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:08:32.6120745Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:08:32.6170590Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:32.6224685Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:08:32.6276439Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:08:32.6327868Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:08:32.6385555Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:08:32.6438828Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:08:32.6489598Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:08:32.6542377Z Entering 'third_party/mimalloc' 2024-08-20T22:08:32.6595750Z Entering 'third_party/nccl/nccl' 2024-08-20T22:08:32.6652488Z Entering 'third_party/nlohmann' 2024-08-20T22:08:32.6705835Z Entering 'third_party/onnx' 2024-08-20T22:08:32.6772179Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:32.6823395Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:32.6880056Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:08:32.6933966Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:32.6984631Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:32.7034626Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:32.7086149Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:32.7137091Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:32.7187287Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:32.7241439Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:32.7291045Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:32.7343750Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:32.7397262Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:32.7468048Z Entering 'third_party/pocketfft' 2024-08-20T22:08:32.7520362Z Entering 'third_party/protobuf' 2024-08-20T22:08:32.7575029Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:32.7624750Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:32.7678652Z Entering 'third_party/psimd' 2024-08-20T22:08:32.7731036Z Entering 'third_party/pthreadpool' 2024-08-20T22:08:32.7784024Z Entering 'third_party/pybind11' 2024-08-20T22:08:32.7836578Z Entering 'third_party/python-peachpy' 2024-08-20T22:08:32.7889314Z Entering 'third_party/sleef' 2024-08-20T22:08:32.7940624Z Entering 'third_party/tensorpipe' 2024-08-20T22:08:32.7991578Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:32.8041482Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:32.8096196Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:32.8145613Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:32.8194947Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:32.8267940Z ##[endgroup] 2024-08-20T22:08:32.8320364Z [command]/usr/bin/git log -1 --format='%H' 2024-08-20T22:08:32.8357478Z '40ec5f6ddd9787aca0449b24128343ff4c4a88b3' 2024-08-20T22:08:32.8575885Z Prepare all required actions 2024-08-20T22:08:32.8576351Z Getting action download info 2024-08-20T22:08:33.0025643Z ##[group]Run ./.github/actions/setup-linux 2024-08-20T22:08:33.0026089Z env: 2024-08-20T22:08:33.0026383Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:33.0026734Z ##[endgroup] 2024-08-20T22:08:33.0088714Z ##[group]Run set -euo pipefail 2024-08-20T22:08:33.0089175Z set -euo pipefail 2024-08-20T22:08:33.0089593Z function get_ec2_metadata() { 2024-08-20T22:08:33.0090160Z  # Pulled from instance metadata endpoint for EC2 2024-08-20T22:08:33.0091100Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-20T22:08:33.0091881Z  category=$1 2024-08-20T22:08:33.0092412Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-20T22:08:33.0093045Z  runner_name_str=i-029e43603fd06f026 2024-08-20T22:08:33.0093548Z  if [[ -f /.inarc ]]; then 2024-08-20T22:08:33.0094050Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-20T22:08:33.0094615Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-20T22:08:33.0095326Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-20T22:08:33.0095932Z  else 2024-08-20T22:08:33.0096431Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-20T22:08:33.0097004Z  fi 2024-08-20T22:08:33.0097290Z } 2024-08-20T22:08:33.0097652Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-20T22:08:33.0098239Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-20T22:08:33.0098893Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-20T22:08:33.0099467Z echo "system info $(uname -a)" 2024-08-20T22:08:33.0110352Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:33.0110861Z env: 2024-08-20T22:08:33.0111142Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:33.0111491Z ##[endgroup] 2024-08-20T22:08:33.0207194Z ami-id: ami-06c68f701d8090592 2024-08-20T22:08:33.0270583Z instance-id: i-029e43603fd06f026 2024-08-20T22:08:33.0328961Z instance-type: g5.4xlarge 2024-08-20T22:08:33.0342927Z system info Linux ip-10-0-49-173.ec2.internal 6.1.94-99.176.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-20T22:08:33.0367611Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:08:33.0368538Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:08:33.0377832Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:33.0378332Z env: 2024-08-20T22:08:33.0378617Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:33.0378968Z ##[endgroup] 2024-08-20T22:08:33.0455066Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-20T22:08:33.0455686Z if systemctl is-active --quiet docker; then 2024-08-20T22:08:33.0456234Z  echo "Docker daemon is running..."; 2024-08-20T22:08:33.0456851Z else 2024-08-20T22:08:33.0457345Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-20T22:08:33.0457933Z fi 2024-08-20T22:08:33.0466581Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:33.0467079Z env: 2024-08-20T22:08:33.0467362Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:33.0467710Z ##[endgroup] 2024-08-20T22:08:33.0558892Z Docker daemon is running... 2024-08-20T22:08:33.0608615Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:08:33.0609186Z with: 2024-08-20T22:08:33.0609465Z shell: bash 2024-08-20T22:08:33.0609777Z timeout_minutes: 5 2024-08-20T22:08:33.0610105Z max_attempts: 3 2024-08-20T22:08:33.0610432Z retry_wait_seconds: 30 2024-08-20T22:08:33.0613596Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-08-20T22:08:33.0616658Z polling_interval_seconds: 1 2024-08-20T22:08:33.0617050Z warning_on_retry: true 2024-08-20T22:08:33.0617415Z continue_on_error: false 2024-08-20T22:08:33.0617755Z env: 2024-08-20T22:08:33.0618040Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:33.0618413Z AWS_RETRY_MODE: standard 2024-08-20T22:08:33.0618827Z AWS_MAX_ATTEMPTS: 5 2024-08-20T22:08:33.0619175Z AWS_DEFAULT_REGION: us-east-1 2024-08-20T22:08:33.0619562Z ##[endgroup] 2024-08-20T22:08:34.4061364Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:08:34.4062242Z Configure a credential helper to remove this warning. See 2024-08-20T22:08:34.4063105Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:08:34.4063630Z 2024-08-20T22:08:34.4063772Z Login Succeeded 2024-08-20T22:08:35.1176803Z Command completed after 1 attempt(s). 2024-08-20T22:08:35.1263456Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:08:35.1264188Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:08:35.1264858Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:08:35.1273927Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:35.1274426Z env: 2024-08-20T22:08:35.1274711Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:35.1275062Z ##[endgroup] 2024-08-20T22:08:35.1402949Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:08:35.1403823Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:08:35.1404445Z # shellcheck disable=SC2046 2024-08-20T22:08:35.1404924Z docker stop $(docker ps -q) || true 2024-08-20T22:08:35.1405420Z # Prune all of the docker images 2024-08-20T22:08:35.1405889Z docker system prune -af 2024-08-20T22:08:35.1414964Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:35.1415465Z env: 2024-08-20T22:08:35.1415737Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:35.1416086Z ##[endgroup] 2024-08-20T22:08:35.1730682Z "docker stop" requires at least 1 argument. 2024-08-20T22:08:35.1731617Z See 'docker stop --help'. 2024-08-20T22:08:35.1731858Z 2024-08-20T22:08:35.1732081Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-20T22:08:35.1732464Z 2024-08-20T22:08:35.1734913Z Stop one or more running containers 2024-08-20T22:08:41.3820628Z Deleted Images: 2024-08-20T22:08:41.3821530Z untagged: pytorch/manylinux-builder:cuda12.4 2024-08-20T22:08:41.3822764Z untagged: pytorch/manylinux-builder@sha256:7725463652a1c2549bc65956a5bb994a17e810fddafd661178f5a6dbd219e8b2 2024-08-20T22:08:41.3824355Z deleted: sha256:efaee664cd962834357b8bf241b8c4feac79ea2dbc636135704bc64bc5724447 2024-08-20T22:08:41.3825399Z deleted: sha256:1cc9f3c0d46a56331445c8cd28ce10f621b35d27f4bfeff493224e8adec41a58 2024-08-20T22:08:41.3826244Z deleted: sha256:c123174a23f907c9db3a54db5239be691c3f34b24cae50c3a70c34d613759294 2024-08-20T22:08:41.3827102Z deleted: sha256:1331679934bc883dffe4cf503929eeddb81fb84096f49c6db28feb1f061b04ff 2024-08-20T22:08:41.3827947Z deleted: sha256:bcfc7bfd2e88b2a744955c7f4ab1f8d431f5cb3e0e8c14c86e9c26132175137d 2024-08-20T22:08:41.3828803Z deleted: sha256:87602668b0b8ca45c1313a4515b28d958cb631e42201dea028ae79844e7b39c0 2024-08-20T22:08:41.3829781Z deleted: sha256:b2d291b2b5b81a97ce4249bf487c2bdf701b4bab27b16d4532f43138c360d826 2024-08-20T22:08:41.3836348Z deleted: sha256:9a5c6ccc25a086f18b54de2c5e3c5ef8bf3db84376e2815202620fe4eb612da8 2024-08-20T22:08:41.3837261Z deleted: sha256:6dd1c3c49490597a5ceef26c087408451a1ec49f4e761beba349a9c22fe75a22 2024-08-20T22:08:41.3838098Z deleted: sha256:651a979337413f82708e36195f04bf8afc6ad530652f4536af008d59ef63fc1e 2024-08-20T22:08:41.3838961Z deleted: sha256:23f7691823044cb1b0e5e110856ebf01fbf8a06909f4a9ef5eacab45da2fe3f6 2024-08-20T22:08:41.3839797Z deleted: sha256:11491d1e279edd3633545d2f7d8a41f0e08d243b14591f63669c92ae0dc40202 2024-08-20T22:08:41.3840615Z deleted: sha256:667784a6cd7b2f98661a158a82541cc7aa0b6a6df1262c6c5811932051171585 2024-08-20T22:08:41.3841452Z deleted: sha256:91622dc153e8cdd1a7adc76725c18a01b4507a2cc80c0fc785d3f12a42c4bad3 2024-08-20T22:08:41.3842304Z deleted: sha256:0d1432dba2ae1fddaff582e8833cb735cd7d544de0761ff263d6a20c6e433c00 2024-08-20T22:08:41.3843206Z deleted: sha256:24d299370986b800148c865a0294eaa0ffb60e3dc8aa796037c15ba69b2f77e9 2024-08-20T22:08:41.3844038Z deleted: sha256:6365fced154bcf86d1175ca69ebd8984faa809817e25ad658fb154ca6cf94eb7 2024-08-20T22:08:41.3844878Z deleted: sha256:d5f96861f34d35b7c8e387fc5cb291dd437d350c8ff23c7ecc4ff42119c668ac 2024-08-20T22:08:41.3845719Z deleted: sha256:8ec1443495054baf1684da24b64c2a06b95db924f726734ddff109edd8faca20 2024-08-20T22:08:41.3846572Z deleted: sha256:8bece3790bad83ee529ccec08dd83d147ae1e01dad832f7df1d04a24aec4f1a2 2024-08-20T22:08:41.3847418Z deleted: sha256:a0624a0073a62e85607b4be4e4c2040d78d20c4ccb779de75bea1831e89a37da 2024-08-20T22:08:41.3848264Z deleted: sha256:cbb001ad780888e5a12efbc2d7583a2b828a4c94b744584e68ce7d1aef891967 2024-08-20T22:08:41.3849113Z deleted: sha256:ef737025fcf99e19b7cd389d4f4558bf3e9185dce6cf25d1088b2b2b6f11f970 2024-08-20T22:08:41.3849939Z deleted: sha256:016882d6d6c81f3b4399221a4a358a11a477a8441147b6552e55f44e23c2c6e1 2024-08-20T22:08:41.3850764Z deleted: sha256:1bfde69bb99337ec3c49c4231311c5f39d69ad7c6775d076f3d3f8a3afa93b4f 2024-08-20T22:08:41.3851619Z deleted: sha256:8caec6ba792e50111d3a0b54bbf67cf3cd106c1d3cc1bbf23f9668aa9724e7a5 2024-08-20T22:08:41.3852467Z deleted: sha256:d08c39913a3001e41697b0550ed7516e15825e43bdfa1c7f8d89f3d9a0524b6b 2024-08-20T22:08:41.3853294Z deleted: sha256:e17360b6ff03d3b73b31c356984176d27c83e3640b255a273534d0d648462032 2024-08-20T22:08:41.3854118Z deleted: sha256:2466196f153e7742867c5faaf84d14fadf812e2609093eebecbe4c6ad4ee0fd9 2024-08-20T22:08:41.3854953Z deleted: sha256:cf5a2ca15e073de5aff9a9591bb89af446f5c8d8539c33148c62d60db73ead37 2024-08-20T22:08:41.3855807Z deleted: sha256:1ac53e5f98c88af8b594781f12bbb416e15bf1a48cd0342d20c93b6538c06d04 2024-08-20T22:08:41.3856647Z deleted: sha256:66039151bed04d6596bb1850af7d5da1e7e5f2213f3b5af8e06202fbf9e79535 2024-08-20T22:08:41.3857492Z deleted: sha256:8cb15b5c841450240e4c96c9ca0a6ff7da76ee8875fa8dc94a70dd6d7e93d1f6 2024-08-20T22:08:41.3858334Z deleted: sha256:af6c56f31add2d54a56e29693cc949e758f12ec348e3a44cc54fe8ee79d16eba 2024-08-20T22:08:41.3859174Z deleted: sha256:d5d341c3c243770ea9b70c4674439e91ebcfa4b73e26ce67e0469718ea770ffa 2024-08-20T22:08:41.3860187Z deleted: sha256:0f54ac065eac806c885fa3294cd1792d17f7956abb7720ba68d98ec89c9c7dc0 2024-08-20T22:08:41.3861031Z deleted: sha256:6bf3daa861b1fd1361da0b38250b9133760235fd48c27fefff94dbae487cb41f 2024-08-20T22:08:41.3861962Z untagged: pytorch/manylinux-builder:cuda12.1 2024-08-20T22:08:41.3862893Z untagged: pytorch/manylinux-builder@sha256:1ceaa79868583367287b01793bb482334e53f31ed3974a78cd271b02247df451 2024-08-20T22:08:41.3863950Z deleted: sha256:f4fc3a9f2fd055e5b72e4feda5bbbd8c6e79ef22b6071d45945eb8ec64042f41 2024-08-20T22:08:41.3864811Z deleted: sha256:70032a09fef4ce88f555f69ad94ca1943d316b6f3b82da8dc7c2821690e88eed 2024-08-20T22:08:41.3865658Z deleted: sha256:a56cfbfa766fde9a9ee399928a535662d561613c3e54f9bf7d42a22557e05941 2024-08-20T22:08:41.3866488Z deleted: sha256:26c2625d88a3265ee12a5179363793d84e44778efa3b0565f571020d370121a9 2024-08-20T22:08:41.3867314Z deleted: sha256:00f9eb5fad579947541dd876721cd7e111bcbd38ca057b4c9c7c5d5130b45561 2024-08-20T22:08:41.3868290Z deleted: sha256:4bce2aee5079b7a763b5022d7617acfa9dc05c721b799345424836fff1bec24f 2024-08-20T22:08:41.3869132Z deleted: sha256:f572e0d9d5a866e08e3e512558dc935b53506c01916dcf5eb41c562cafca97c4 2024-08-20T22:08:41.3869978Z deleted: sha256:276ca07dabc1b6ec9a5577d3fe2fa1b0912f2c14eb1135fbfd5e883dadf96a55 2024-08-20T22:08:41.3870831Z deleted: sha256:413d9646e2cdd6bb7d142b7e2b44e80b8e55b0ea3101a06d2ae462bd98d41b53 2024-08-20T22:08:41.3871686Z deleted: sha256:eb7e8fa961c4c9ce0d409c41e2db44a5abf11078c04dcd727065808670502776 2024-08-20T22:08:41.3872534Z deleted: sha256:f052aaceea687692e6797adac86f83463ca13c7acb13399f3fe60671d914d837 2024-08-20T22:08:41.3873377Z deleted: sha256:95abcbbd6fe3402acb416254069895c430be984f650e0beb61e0b96dc9d4e3a1 2024-08-20T22:08:41.3874221Z deleted: sha256:1f1575a0dc7eafbe0bc24349a498188dc66b0c7afe0a92ad3177982dc53d1d7f 2024-08-20T22:08:41.3875070Z deleted: sha256:e8516cf5f8eaea7208825a0e74bc6ccc2425b2eeb87bf835fba068f096435065 2024-08-20T22:08:41.3875914Z deleted: sha256:50e753ec0ac354845824b7647dcb5c80fc1ecdf20246a62ef10afe70d2cba914 2024-08-20T22:08:41.3876744Z deleted: sha256:9a3a953a80dac55ead243068e4801c591294ce2467a7f2ab37acf37e7b50623d 2024-08-20T22:08:41.3877586Z deleted: sha256:8b589e158e3986bebcf7bfa8daf21e07ea295d0129753a23b59dc6016212d07a 2024-08-20T22:08:41.3878410Z deleted: sha256:9780567f56682e7ccb6519d6a582a4233a94a473e8319fdce963a938e8348437 2024-08-20T22:08:41.3879230Z deleted: sha256:83f38e31da6d72b6c7a862161962449d865590ecee2da9f77ce4a146acc998b0 2024-08-20T22:08:41.3880051Z deleted: sha256:b23603a0fe1921a9b378cd4f49db9da250b4f8026d73c027f355725059ea9ea8 2024-08-20T22:08:41.3880872Z deleted: sha256:6fc31f41ca816c5630174d83c904113d618ef54ee97610240e0f5f74a7b05ddb 2024-08-20T22:08:41.3881691Z deleted: sha256:b1fc1302c8cd0f46187e82d20d88b8202fea499b545d365200cdf56252b58512 2024-08-20T22:08:41.3882531Z deleted: sha256:53b918ca72cbb58e7562e0c9dc2babc7518b268d0ec7536af7f2606e1dff95cd 2024-08-20T22:08:41.3883476Z deleted: sha256:4913bc3e68cdc276b1bdc93abfb9d4fe26c4edc0e02f58b976716f71c7cff383 2024-08-20T22:08:41.3884325Z deleted: sha256:12c37601a9c0fafe2eace184744245d4aacfb250075ccfea1318049fac59673c 2024-08-20T22:08:41.3885162Z deleted: sha256:76c1229c2e89d5a621f482e8ad46f1ab304af8101f6dc0c94e7a2d968d0e5dd1 2024-08-20T22:08:41.3886005Z deleted: sha256:8a0b862a9ff3652d53bc44d54db195bccd91824ae70c438bf13eeb29a262dae6 2024-08-20T22:08:41.3886856Z deleted: sha256:36bbacb75b1fc6aa5f9dd973767ed867786a74144cb10b11b3b5459b5a36453c 2024-08-20T22:08:41.3887698Z deleted: sha256:601fbf08be902110317ee896327aedc5f0d4f0bb2c68e3be8a9afab4d7e375ac 2024-08-20T22:08:41.3888546Z deleted: sha256:ebfd2b021b5e26b3ac1c81ab75ee84007f8ae82fc5329d1bb797589046ed7e3a 2024-08-20T22:08:41.3889392Z deleted: sha256:7f450dd7de6a0c0234061a4d6d27513dbd48d3af305d48ac844be6f213c9f484 2024-08-20T22:08:41.3890248Z deleted: sha256:0afb7f31e1ac43cfddfdc4034a31fa4ededc1d057ba90aa605105d5004d09ff6 2024-08-20T22:08:41.3891087Z deleted: sha256:83933310864f0db0b6271f6484fe2a2ec4c4aa107dc676707aab1a86e84e78b8 2024-08-20T22:08:41.3891907Z deleted: sha256:5a8dd28488303431ac01ac5ef13992e7bf21d61c6e94702046235c8e1892c01c 2024-08-20T22:08:41.3892737Z deleted: sha256:0d1932c19f3321c16b8b0febc6905cf104fe67e6cc7901d220de7f2899e7336c 2024-08-20T22:08:41.3893670Z deleted: sha256:701fa06264450ce0aa66e58d35aad330e7bd68b0fbac8949d8e2d7587340a139 2024-08-20T22:08:41.3894542Z deleted: sha256:f89a6b18745ab169f86e3cfa97cd85d114b480b991fc3d9c5be888a9c29bed91 2024-08-20T22:08:41.3895363Z deleted: sha256:174f5685490326fc0a1c0f5570b8663732189b327007e47ff13d2ca59673db02 2024-08-20T22:08:41.3895844Z 2024-08-20T22:08:41.4553904Z Total reclaimed space: 23.37GB 2024-08-20T22:08:41.4608497Z ##[group]Run set +e 2024-08-20T22:08:41.4608983Z set +e 2024-08-20T22:08:41.4609422Z set -x 2024-08-20T22:08:41.4609861Z  2024-08-20T22:08:41.4610332Z PT_DOMAIN=download.pytorch.org 2024-08-20T22:08:41.4611517Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-20T22:08:41.4613450Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-20T22:08:41.4614655Z # one is returned at random 2024-08-20T22:08:41.4615501Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-20T22:08:41.4616271Z  2024-08-20T22:08:41.4616731Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:08:41.4617659Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-20T22:08:41.4618826Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-20T22:08:41.4619639Z  2024-08-20T22:08:41.4620124Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:08:41.4620937Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-20T22:08:41.4621687Z  exit 1 2024-08-20T22:08:41.4622130Z  fi 2024-08-20T22:08:41.4622529Z fi 2024-08-20T22:08:41.4622919Z  2024-08-20T22:08:41.4623442Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-20T22:08:41.4624218Z  # Clean up any old records first 2024-08-20T22:08:41.4625030Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-20T22:08:41.4625704Z fi 2024-08-20T22:08:41.4626105Z  2024-08-20T22:08:41.4626715Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-20T22:08:41.4627524Z cat /etc/hosts 2024-08-20T22:08:41.4639313Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:41.4640047Z env: 2024-08-20T22:08:41.4640453Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:41.4640965Z ##[endgroup] 2024-08-20T22:08:41.4675679Z + PT_DOMAIN=download.pytorch.org 2024-08-20T22:08:41.4682193Z ++ dig -4 +short download.pytorch.org 2024-08-20T22:08:41.4682816Z ++ tail -n1 2024-08-20T22:08:41.4986532Z + RESOLVED_IP=18.160.10.36 2024-08-20T22:08:41.4987207Z + '[' -z 18.160.10.36 ']' 2024-08-20T22:08:41.4987734Z + grep -r download.pytorch.org /etc/hosts 2024-08-20T22:08:41.5005897Z + echo '18.160.10.36 download.pytorch.org' 2024-08-20T22:08:41.5006748Z + sudo tee -a /etc/hosts 2024-08-20T22:08:41.6129996Z 18.160.10.36 download.pytorch.org 2024-08-20T22:08:41.6152372Z + cat /etc/hosts 2024-08-20T22:08:41.6162775Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-20T22:08:41.6175582Z ::1 localhost6 localhost6.localdomain6 2024-08-20T22:08:41.6176452Z 18.160.10.36 download.pytorch.org 2024-08-20T22:08:41.6334422Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-20T22:08:41.6335035Z with: 2024-08-20T22:08:41.6336146Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:41.6337384Z docker-build-dir: .ci/docker 2024-08-20T22:08:41.6337788Z working-directory: . 2024-08-20T22:08:41.6338267Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:41.6338799Z force-push: false 2024-08-20T22:08:41.6339113Z env: 2024-08-20T22:08:41.6339385Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:41.6339924Z ##[endgroup] 2024-08-20T22:08:41.6360482Z ##[group]Run set -ex 2024-08-20T22:08:41.6361052Z set -ex 2024-08-20T22:08:41.6361475Z  2024-08-20T22:08:41.6362143Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-20T22:08:41.6363401Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-20T22:08:41.6364390Z # job could then download the pre-built image as usual 2024-08-20T22:08:41.6365202Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-20T22:08:41.6365989Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:41.6366755Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:41.6367432Z  2024-08-20T22:08:41.6368036Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-20T22:08:41.6368762Z  exit 0 2024-08-20T22:08:41.6369189Z else 2024-08-20T22:08:41.6369650Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:41.6370204Z fi 2024-08-20T22:08:41.6370597Z  2024-08-20T22:08:41.6371158Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-20T22:08:41.6372167Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-20T22:08:41.6373050Z  # use it as it is, but first let's extract the tag 2024-08-20T22:08:41.6373817Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-20T22:08:41.6374706Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:41.6375505Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:41.6376154Z else 2024-08-20T22:08:41.6376728Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-20T22:08:41.6377496Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:41.6378514Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:41.6379409Z fi 2024-08-20T22:08:41.6389227Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:41.6389824Z env: 2024-08-20T22:08:41.6390218Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:41.6390719Z REPO_NAME: pytorch 2024-08-20T22:08:41.6391927Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:41.6393266Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:08:41.6393915Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:41.6394569Z ##[endgroup] 2024-08-20T22:08:41.6427224Z + [[ ! -d .ci/docker ]] 2024-08-20T22:08:41.6427758Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-20T22:08:41.6428358Z + echo skip=false 2024-08-20T22:08:41.6430492Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-20T22:08:41.6435130Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:41.6436518Z ++ awk -F '[:,]' '{print $2}' 2024-08-20T22:08:41.6460712Z + DOCKER_TAG=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:41.6461544Z + echo docker-tag=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:41.6463499Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:41.6496853Z ##[group]Run set +e 2024-08-20T22:08:41.6497621Z set +e 2024-08-20T22:08:41.6498024Z set -x 2024-08-20T22:08:41.6498459Z  2024-08-20T22:08:41.6498824Z login() { 2024-08-20T22:08:41.6499587Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:08:41.6500449Z } 2024-08-20T22:08:41.6500848Z  2024-08-20T22:08:41.6501190Z retry () { 2024-08-20T22:08:41.6501729Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:08:41.6502290Z } 2024-08-20T22:08:41.6502653Z  2024-08-20T22:08:41.6503172Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:08:41.6503678Z  2024-08-20T22:08:41.6504243Z # Check if image already exists, if it does then skip building it 2024-08-20T22:08:41.6505119Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-20T22:08:41.6505686Z  exit 0 2024-08-20T22:08:41.6506091Z fi 2024-08-20T22:08:41.6506554Z  2024-08-20T22:08:41.6507154Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-20T22:08:41.6508036Z # be empty. The default action would be to continue rebuild the image 2024-08-20T22:08:41.6508955Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-20T22:08:41.6509726Z  # if we're on the base branch then use the parent commit 2024-08-20T22:08:41.6510509Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-20T22:08:41.6511042Z else 2024-08-20T22:08:41.6511608Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-20T22:08:41.6512424Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-20T22:08:41.6513084Z fi 2024-08-20T22:08:41.6513425Z  2024-08-20T22:08:41.6513864Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-20T22:08:41.6514507Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:41.6515082Z  2024-08-20T22:08:41.6515795Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-20T22:08:41.6516698Z  exit 0 2024-08-20T22:08:41.6517111Z fi 2024-08-20T22:08:41.6517601Z  2024-08-20T22:08:41.6518095Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-20T22:08:41.6519178Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-20T22:08:41.6520112Z  exit 1 2024-08-20T22:08:41.6520519Z fi 2024-08-20T22:08:41.6520850Z  2024-08-20T22:08:41.6521472Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-20T22:08:41.6522537Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-20T22:08:41.6523623Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-20T22:08:41.6524698Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-20T22:08:41.6525920Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-20T22:08:41.6526896Z fi 2024-08-20T22:08:41.6527354Z  2024-08-20T22:08:41.6527764Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:41.6536306Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:41.6536899Z env: 2024-08-20T22:08:41.6537273Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:41.6537789Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:08:41.6538355Z BASE_REVISION: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:08:41.6539771Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:41.6541277Z DOCKER_TAG: f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:41.6542066Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:41.6542663Z ##[endgroup] 2024-08-20T22:08:41.6572187Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:41.6573034Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:41.6574952Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:08:41.6576490Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:42.1929863Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:08:42.1930916Z Configure a credential helper to remove this warning. See 2024-08-20T22:08:42.1932037Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:08:42.1932599Z 2024-08-20T22:08:42.1932793Z Login Succeeded 2024-08-20T22:08:42.1954604Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:42.4162499Z { 2024-08-20T22:08:42.4162956Z "schemaVersion": 2, 2024-08-20T22:08:42.4163987Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-20T22:08:42.4164994Z "config": { 2024-08-20T22:08:42.4166414Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-20T22:08:42.4167280Z "size": 48711, 2024-08-20T22:08:42.4168405Z "digest": "sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb" 2024-08-20T22:08:42.4169658Z }, 2024-08-20T22:08:42.4170173Z "layers": [ 2024-08-20T22:08:42.4171433Z { 2024-08-20T22:08:42.4173056Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4173931Z "size": 28580681, 2024-08-20T22:08:42.4174934Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-08-20T22:08:42.4175925Z }, 2024-08-20T22:08:42.4176531Z { 2024-08-20T22:08:42.4177222Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4178139Z "size": 7943451, 2024-08-20T22:08:42.4179022Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-08-20T22:08:42.4180007Z }, 2024-08-20T22:08:42.4180463Z { 2024-08-20T22:08:42.4181248Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4181998Z "size": 55728572, 2024-08-20T22:08:42.4182611Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-08-20T22:08:42.4183394Z }, 2024-08-20T22:08:42.4183749Z { 2024-08-20T22:08:42.4184319Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4185051Z "size": 186, 2024-08-20T22:08:42.4185699Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-08-20T22:08:42.4186714Z }, 2024-08-20T22:08:42.4187119Z { 2024-08-20T22:08:42.4187618Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4188296Z "size": 6886, 2024-08-20T22:08:42.4189218Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-08-20T22:08:42.4189956Z }, 2024-08-20T22:08:42.4190608Z { 2024-08-20T22:08:42.4191210Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4191893Z "size": 1329001756, 2024-08-20T22:08:42.4192533Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-08-20T22:08:42.4193314Z }, 2024-08-20T22:08:42.4193780Z { 2024-08-20T22:08:42.4194587Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4195218Z "size": 62501, 2024-08-20T22:08:42.4195885Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-08-20T22:08:42.4197267Z }, 2024-08-20T22:08:42.4197785Z { 2024-08-20T22:08:42.4198787Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4199487Z "size": 1684, 2024-08-20T22:08:42.4200122Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-08-20T22:08:42.4200836Z }, 2024-08-20T22:08:42.4201230Z { 2024-08-20T22:08:42.4201778Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4202474Z "size": 1523, 2024-08-20T22:08:42.4203239Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-08-20T22:08:42.4203971Z }, 2024-08-20T22:08:42.4204445Z { 2024-08-20T22:08:42.4204996Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4205615Z "size": 2528295403, 2024-08-20T22:08:42.4206350Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-08-20T22:08:42.4207064Z }, 2024-08-20T22:08:42.4207370Z { 2024-08-20T22:08:42.4207982Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4208654Z "size": 86016, 2024-08-20T22:08:42.4209313Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-08-20T22:08:42.4210067Z }, 2024-08-20T22:08:42.4210417Z { 2024-08-20T22:08:42.4210961Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4211770Z "size": 1822, 2024-08-20T22:08:42.4212386Z "digest": "sha256:ac23c5fcc1ef310e9cd0d138b180281188e20a3e28c51cd14751622e3c7d92ea" 2024-08-20T22:08:42.4213135Z }, 2024-08-20T22:08:42.4213532Z { 2024-08-20T22:08:42.4214068Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4214717Z "size": 246761563, 2024-08-20T22:08:42.4215448Z "digest": "sha256:ca426ffc9d2bd051ad1ff5c4f02fd5d9be161e70953cc37882c815080c708f4d" 2024-08-20T22:08:42.4216173Z }, 2024-08-20T22:08:42.4216508Z { 2024-08-20T22:08:42.4217097Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4217761Z "size": 544, 2024-08-20T22:08:42.4218550Z "digest": "sha256:4373aa743caf6660c671295b837048c8350a2165f2336dc3b7b6c96462579e50" 2024-08-20T22:08:42.4219217Z }, 2024-08-20T22:08:42.4219587Z { 2024-08-20T22:08:42.4220170Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4220836Z "size": 1258, 2024-08-20T22:08:42.4221457Z "digest": "sha256:75b2abb16c685854c017d0cf75c626526c956734c656d9e78fdd4520317e9b3c" 2024-08-20T22:08:42.4222214Z }, 2024-08-20T22:08:42.4222572Z { 2024-08-20T22:08:42.4223091Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4223800Z "size": 484, 2024-08-20T22:08:42.4224475Z "digest": "sha256:ef1e3ef1e85f0124270b8506721219150e230184181f8207d64463bf5c67e656" 2024-08-20T22:08:42.4225195Z }, 2024-08-20T22:08:42.4225623Z { 2024-08-20T22:08:42.4226184Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4226846Z "size": 91711766, 2024-08-20T22:08:42.4227550Z "digest": "sha256:0bf61114d37b2d09065aa6852b1e78a32580a1ff011dd5d23f03b0f5c4e710af" 2024-08-20T22:08:42.4228248Z }, 2024-08-20T22:08:42.4228602Z { 2024-08-20T22:08:42.4229178Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4229861Z "size": 3390, 2024-08-20T22:08:42.4230580Z "digest": "sha256:2df849f72ba4258df10c9c45ace21a66d93b73196e78e96f12c13098822f60e2" 2024-08-20T22:08:42.4231374Z }, 2024-08-20T22:08:42.4231716Z { 2024-08-20T22:08:42.4232282Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4233013Z "size": 1909, 2024-08-20T22:08:42.4233646Z "digest": "sha256:34c47d63c5de27418739aa048b4dd781c3006d65691e9b903c95304d0509d89c" 2024-08-20T22:08:42.4234355Z }, 2024-08-20T22:08:42.4234731Z { 2024-08-20T22:08:42.4235267Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4236017Z "size": 701, 2024-08-20T22:08:42.4236713Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:08:42.4237487Z }, 2024-08-20T22:08:42.4237854Z { 2024-08-20T22:08:42.4238439Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4239196Z "size": 2864254879, 2024-08-20T22:08:42.4239841Z "digest": "sha256:6efb96fc87ea1bca54876d85331e5585e28f325a1d60ee1aef7811eabb74f933" 2024-08-20T22:08:42.4240631Z }, 2024-08-20T22:08:42.4240992Z { 2024-08-20T22:08:42.4241483Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4242188Z "size": 380, 2024-08-20T22:08:42.4242847Z "digest": "sha256:b119c4fc6c58d9e72744201ff4850b34184a94eb3566031b7d36e87ecce950a6" 2024-08-20T22:08:42.4243743Z }, 2024-08-20T22:08:42.4244066Z { 2024-08-20T22:08:42.4244612Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4245321Z "size": 12900, 2024-08-20T22:08:42.4246074Z "digest": "sha256:33b639226eeefa41d7bb26a8ecc20d271ceef45571dabfd80aaee3634926381b" 2024-08-20T22:08:42.4246766Z }, 2024-08-20T22:08:42.4247151Z { 2024-08-20T22:08:42.4247706Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4248322Z "size": 803, 2024-08-20T22:08:42.4249010Z "digest": "sha256:d3547bd43146930aa830d54562c98aa678e355cf5f84527e9717eb72fffe8ca3" 2024-08-20T22:08:42.4249747Z }, 2024-08-20T22:08:42.4250095Z { 2024-08-20T22:08:42.4250637Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4251318Z "size": 106, 2024-08-20T22:08:42.4251959Z "digest": "sha256:5645520256fa6d7f2270608eaed53849a131d03fbf4c5d4472c9cfde1a439e47" 2024-08-20T22:08:42.4252741Z }, 2024-08-20T22:08:42.4253117Z { 2024-08-20T22:08:42.4253643Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4254365Z "size": 503, 2024-08-20T22:08:42.4254992Z "digest": "sha256:829ed7d91e3818d0e11834c71551117c7aeec0a51d798e93d641faa0c02582cb" 2024-08-20T22:08:42.4255673Z }, 2024-08-20T22:08:42.4256100Z { 2024-08-20T22:08:42.4256643Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4257271Z "size": 121477267, 2024-08-20T22:08:42.4258003Z "digest": "sha256:385d0421d67c533d628985c453a186028a61f6ad4209ae519ec3d4ecea661d3e" 2024-08-20T22:08:42.4258706Z }, 2024-08-20T22:08:42.4259078Z { 2024-08-20T22:08:42.4259687Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4260690Z "size": 109, 2024-08-20T22:08:42.4261360Z "digest": "sha256:7cbfce999d76851e07dd71056a6d852629d789ccfd46be44cc405b451fb8bb58" 2024-08-20T22:08:42.4262145Z }, 2024-08-20T22:08:42.4262455Z { 2024-08-20T22:08:42.4263001Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4263701Z "size": 490, 2024-08-20T22:08:42.4264289Z "digest": "sha256:6658e558ebe496f0d628947534b0ea1e44808f84f2dfd8a9a704e8aa5329d791" 2024-08-20T22:08:42.4265017Z }, 2024-08-20T22:08:42.4265434Z { 2024-08-20T22:08:42.4265982Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4266591Z "size": 297, 2024-08-20T22:08:42.4267328Z "digest": "sha256:33444a5a7e01a3010fd1cc16b32a66892a08600d7f700747d23faf3c36624f3a" 2024-08-20T22:08:42.4268043Z }, 2024-08-20T22:08:42.4268586Z { 2024-08-20T22:08:42.4269114Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4269759Z "size": 103, 2024-08-20T22:08:42.4270443Z "digest": "sha256:7c47122f4e09b47e891b4ae9e499a35a4eb46faacae87d0e946615adda6e0706" 2024-08-20T22:08:42.4271172Z }, 2024-08-20T22:08:42.4271475Z { 2024-08-20T22:08:42.4272039Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4272708Z "size": 1473, 2024-08-20T22:08:42.4273313Z "digest": "sha256:1811ff4d3556ac255604f346d879528aff2aadd7652c2d39baf4fe6eaa60aec3" 2024-08-20T22:08:42.4274143Z }, 2024-08-20T22:08:42.4274507Z { 2024-08-20T22:08:42.4275159Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4275852Z "size": 594729516, 2024-08-20T22:08:42.4276510Z "digest": "sha256:7754ee068b33f760d9c320c26a1a3935247ace0c8c463a976105c1586a43fb1c" 2024-08-20T22:08:42.4277210Z }, 2024-08-20T22:08:42.4277620Z { 2024-08-20T22:08:42.4278120Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4278792Z "size": 164, 2024-08-20T22:08:42.4279469Z "digest": "sha256:0479fbac3361713c075bf788e778117917dcacdd1ff71ed79a789dd2016a1714" 2024-08-20T22:08:42.4280138Z }, 2024-08-20T22:08:42.4280602Z { 2024-08-20T22:08:42.4281176Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4281822Z "size": 564, 2024-08-20T22:08:42.4282433Z "digest": "sha256:f1d6874dee9c34d154278d2b6935d04447f07fa89d25d4a09398234040d8a7e8" 2024-08-20T22:08:42.4283279Z }, 2024-08-20T22:08:42.4283625Z { 2024-08-20T22:08:42.4284177Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4284846Z "size": 35874203, 2024-08-20T22:08:42.4285529Z "digest": "sha256:afc429e87b593a64f3e0a1be51e09e966ea1aa71e6e8a0bec4edc9f2f7bc1c05" 2024-08-20T22:08:42.4286241Z }, 2024-08-20T22:08:42.4286651Z { 2024-08-20T22:08:42.4287174Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4287890Z "size": 104, 2024-08-20T22:08:42.4288554Z "digest": "sha256:019620471c2592cd22437fe8ed1a94c784114cf7445c5c2ab36dd0bae727fc66" 2024-08-20T22:08:42.4289239Z }, 2024-08-20T22:08:42.4289570Z { 2024-08-20T22:08:42.4290136Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4290796Z "size": 425, 2024-08-20T22:08:42.4291408Z "digest": "sha256:27aeb1b2cd5e28d8f0e54fc1090baf88abc29ebf5fce088b493fcae99c9550d1" 2024-08-20T22:08:42.4292196Z }, 2024-08-20T22:08:42.4292534Z { 2024-08-20T22:08:42.4293101Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4293739Z "size": 20262183, 2024-08-20T22:08:42.4294453Z "digest": "sha256:829c0e19f4b7797e9120a86172fabfd710f9b3971ffd714a616931a3fadc7192" 2024-08-20T22:08:42.4295193Z }, 2024-08-20T22:08:42.4295522Z { 2024-08-20T22:08:42.4296049Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4296752Z "size": 438, 2024-08-20T22:08:42.4297400Z "digest": "sha256:560cd6122a3868e7e82e75eb103d182edf6512114d61ccdfae099af00cb16b4c" 2024-08-20T22:08:42.4298071Z }, 2024-08-20T22:08:42.4298480Z { 2024-08-20T22:08:42.4299010Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4299663Z "size": 701, 2024-08-20T22:08:42.4300321Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:08:42.4301029Z }, 2024-08-20T22:08:42.4301435Z { 2024-08-20T22:08:42.4302033Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4302650Z "size": 141, 2024-08-20T22:08:42.4303303Z "digest": "sha256:5a4eb19d72198a01f3babd4711c40fcf7c6e5cb9d5cfedc507b0061408d47097" 2024-08-20T22:08:42.4304089Z }, 2024-08-20T22:08:42.4304390Z { 2024-08-20T22:08:42.4304939Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4305634Z "size": 135, 2024-08-20T22:08:42.4306352Z "digest": "sha256:da6a838a66c72b5de9a5590c233779880ea901c1264940a1c618825d2501ffd1" 2024-08-20T22:08:42.4307040Z }, 2024-08-20T22:08:42.4307433Z { 2024-08-20T22:08:42.4308051Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4308701Z "size": 4838068375, 2024-08-20T22:08:42.4309381Z "digest": "sha256:cbb79a7053537bed2d80b77198ea552b985563b11d3f8616593c6ab575a3a090" 2024-08-20T22:08:42.4310107Z }, 2024-08-20T22:08:42.4310443Z { 2024-08-20T22:08:42.4310976Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4311648Z "size": 194, 2024-08-20T22:08:42.4312274Z "digest": "sha256:4d4fa6802ab3e6b36d8d4adb650046e9370c4a16a19876970559ef8a312ca855" 2024-08-20T22:08:42.4313139Z }, 2024-08-20T22:08:42.4313441Z { 2024-08-20T22:08:42.4313963Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4314767Z "size": 564, 2024-08-20T22:08:42.4315397Z "digest": "sha256:0f085d4fa279366fcd52d303b7ad8abddc5962a8872174b8109f3f5d5ca5f2cf" 2024-08-20T22:08:42.4316059Z }, 2024-08-20T22:08:42.4316461Z { 2024-08-20T22:08:42.4316978Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4317689Z "size": 43162921, 2024-08-20T22:08:42.4318304Z "digest": "sha256:4a0412373305517489d891a4749e6d2b80b4aeb7df3b8adca0ed0abb2f3e5300" 2024-08-20T22:08:42.4319004Z }, 2024-08-20T22:08:42.4319417Z { 2024-08-20T22:08:42.4319900Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4320550Z "size": 106, 2024-08-20T22:08:42.4321237Z "digest": "sha256:302c8c1779444e54919f83e91c6d5e1fdea653fc899be802b3a2a34d34d99d5d" 2024-08-20T22:08:42.4322019Z }, 2024-08-20T22:08:42.4322327Z { 2024-08-20T22:08:42.4322918Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4323666Z "size": 1380, 2024-08-20T22:08:42.4324318Z "digest": "sha256:02970347dff622ee45629bc28788716d00528464f71f88ec7b74ae5f44a486e2" 2024-08-20T22:08:42.4325035Z }, 2024-08-20T22:08:42.4325384Z { 2024-08-20T22:08:42.4325941Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4326603Z "size": 701, 2024-08-20T22:08:42.4327313Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:08:42.4328025Z }, 2024-08-20T22:08:42.4328489Z { 2024-08-20T22:08:42.4328980Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4329648Z "size": 139, 2024-08-20T22:08:42.4330339Z "digest": "sha256:916c5e8580164926a3aec87e88a5d1c2d02756e49fdeed4198e1e6304efa2fd3" 2024-08-20T22:08:42.4331068Z }, 2024-08-20T22:08:42.4331365Z { 2024-08-20T22:08:42.4331928Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4332592Z "size": 118, 2024-08-20T22:08:42.4333257Z "digest": "sha256:c42b3602a1fb4df5111224650f15adffb025979ae9c614807379ea84e189533b" 2024-08-20T22:08:42.4333931Z }, 2024-08-20T22:08:42.4334306Z { 2024-08-20T22:08:42.4334951Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4335571Z "size": 4324672728, 2024-08-20T22:08:42.4336268Z "digest": "sha256:11fad972ef6cb9d2dc236504b0ee5b6e5ea996e4576ece0ed3262fbb9ee2de60" 2024-08-20T22:08:42.4337046Z }, 2024-08-20T22:08:42.4337423Z { 2024-08-20T22:08:42.4337921Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4338614Z "size": 174, 2024-08-20T22:08:42.4339265Z "digest": "sha256:f957c16244819132b478272aaf24231699c4fcbd495c229a574f94daab8d6b86" 2024-08-20T22:08:42.4339969Z }, 2024-08-20T22:08:42.4340323Z { 2024-08-20T22:08:42.4340879Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4341537Z "size": 907, 2024-08-20T22:08:42.4342321Z "digest": "sha256:768c31dca016b80bbdc973c8ee818034bdbd17b7ea9bdadbf0ec45ea51005720" 2024-08-20T22:08:42.4343017Z }, 2024-08-20T22:08:42.4343469Z { 2024-08-20T22:08:42.4344086Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4344765Z "size": 701, 2024-08-20T22:08:42.4345413Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:08:42.4346201Z }, 2024-08-20T22:08:42.4346556Z { 2024-08-20T22:08:42.4347076Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4347796Z "size": 134, 2024-08-20T22:08:42.4348454Z "digest": "sha256:8c32d0ecebbe2054c96a3a5c1f3d6c8b87c2e826686b369e5242cd55d0f01c2d" 2024-08-20T22:08:42.4349280Z }, 2024-08-20T22:08:42.4349640Z { 2024-08-20T22:08:42.4350160Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4350932Z "size": 32, 2024-08-20T22:08:42.4351590Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:08:42.4352326Z }, 2024-08-20T22:08:42.4352661Z { 2024-08-20T22:08:42.4353246Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4353883Z "size": 158, 2024-08-20T22:08:42.4354565Z "digest": "sha256:78ee77eddef23f20afc78e2924f4467041cadb754634ee3df431c68211cf676d" 2024-08-20T22:08:42.4355318Z }, 2024-08-20T22:08:42.4355757Z { 2024-08-20T22:08:42.4356246Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4356959Z "size": 1841, 2024-08-20T22:08:42.4357592Z "digest": "sha256:37513e9b43d7123e7ddee09100f62a72b57af7b3833eaecd515e247e2f63e13f" 2024-08-20T22:08:42.4358352Z }, 2024-08-20T22:08:42.4358686Z { 2024-08-20T22:08:42.4359218Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4360211Z "size": 7529776, 2024-08-20T22:08:42.4360887Z "digest": "sha256:a7a5392e2bf6b73d9be292cc22a99ad4637636c15733beb7afbe6403b5ba21a8" 2024-08-20T22:08:42.4361611Z }, 2024-08-20T22:08:42.4362001Z { 2024-08-20T22:08:42.4362551Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4363321Z "size": 163, 2024-08-20T22:08:42.4364023Z "digest": "sha256:b10de02255c021b2366b85bf12ddf593aac89f88964201c455b71a145bc5ca2e" 2024-08-20T22:08:42.4364744Z }, 2024-08-20T22:08:42.4365084Z { 2024-08-20T22:08:42.4365645Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4366306Z "size": 7943, 2024-08-20T22:08:42.4366959Z "digest": "sha256:e56eeff2db93ffe5002a2bb764795b7333321c6802cc28bef0801a619fe6a07e" 2024-08-20T22:08:42.4367751Z }, 2024-08-20T22:08:42.4368071Z { 2024-08-20T22:08:42.4368597Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4369319Z "size": 8068, 2024-08-20T22:08:42.4369999Z "digest": "sha256:70ac3e38d0fd1f0f46ad710c3fc485fd01406606054e8cf84a63af1570bc9f3a" 2024-08-20T22:08:42.4370722Z }, 2024-08-20T22:08:42.4371121Z { 2024-08-20T22:08:42.4371643Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4558975Z "size": 302, 2024-08-20T22:08:42.4559656Z "digest": "sha256:eb09cbf3e4168749cc8605e3188ed35070f6037d13930e443c38d8941734488f" 2024-08-20T22:08:42.4560608Z }, 2024-08-20T22:08:42.4560866Z { 2024-08-20T22:08:42.4561305Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4561865Z "size": 7629638, 2024-08-20T22:08:42.4562401Z "digest": "sha256:ee1273153675e6543b58f5be50321abee5e91f7351701d5d501a6837eaf676ef" 2024-08-20T22:08:42.4563011Z }, 2024-08-20T22:08:42.4563308Z { 2024-08-20T22:08:42.4563747Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4564323Z "size": 108, 2024-08-20T22:08:42.4564904Z "digest": "sha256:e22979f9e0eedfa16f98eb4fa26f30a50c237f5ffd2f6e5b898f2b908058496d" 2024-08-20T22:08:42.4565516Z }, 2024-08-20T22:08:42.4565760Z { 2024-08-20T22:08:42.4566187Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4566744Z "size": 54145661, 2024-08-20T22:08:42.4567734Z "digest": "sha256:6c1f0aabce7c84f9b0b9ab3b96325778dba1c1465b7463c5b4c6be1cf52b74db" 2024-08-20T22:08:42.4568363Z }, 2024-08-20T22:08:42.4568610Z { 2024-08-20T22:08:42.4569039Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4569584Z "size": 473, 2024-08-20T22:08:42.4570123Z "digest": "sha256:7ebe49c3a23e71b289ce012b48dad35791a2f2aa7fcee2326a64d763b0d14df8" 2024-08-20T22:08:42.4570737Z }, 2024-08-20T22:08:42.4570975Z { 2024-08-20T22:08:42.4571402Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4571954Z "size": 1374858766, 2024-08-20T22:08:42.4572515Z "digest": "sha256:5d115b29b5d53e5418762bc41ae975dd245bcffc3d058b8902c58146d505cb37" 2024-08-20T22:08:42.4573282Z }, 2024-08-20T22:08:42.4573527Z { 2024-08-20T22:08:42.4573949Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4574495Z "size": 106, 2024-08-20T22:08:42.4575040Z "digest": "sha256:7f2d7a04fbe65bbe1e895138e0cfeeef53ad926c890bab8e7e132ad1e655e48f" 2024-08-20T22:08:42.4575667Z }, 2024-08-20T22:08:42.4575913Z { 2024-08-20T22:08:42.4576330Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4576882Z "size": 559, 2024-08-20T22:08:42.4577403Z "digest": "sha256:d79fd7d526989e554455eda7d459f2c88444030f61710101e52ec8c7a49b7b79" 2024-08-20T22:08:42.4578008Z }, 2024-08-20T22:08:42.4578244Z { 2024-08-20T22:08:42.4578673Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4579222Z "size": 46248488, 2024-08-20T22:08:42.4579771Z "digest": "sha256:9752d6fbbcab4e7403f26777c5b65911320de6389c81b833a47ba45c4f3354b4" 2024-08-20T22:08:42.4580379Z }, 2024-08-20T22:08:42.4580621Z { 2024-08-20T22:08:42.4581047Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4581594Z "size": 111, 2024-08-20T22:08:42.4582121Z "digest": "sha256:83808b2eff4d71ed46db7c29f66fe918f328405c25b314225dfad01b4868d1f4" 2024-08-20T22:08:42.4582729Z }, 2024-08-20T22:08:42.4582972Z { 2024-08-20T22:08:42.4583393Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4583946Z "size": 32, 2024-08-20T22:08:42.4584538Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:08:42.4585149Z }, 2024-08-20T22:08:42.4585383Z { 2024-08-20T22:08:42.4585811Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4586358Z "size": 32, 2024-08-20T22:08:42.4586885Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:08:42.4587491Z }, 2024-08-20T22:08:42.4587734Z { 2024-08-20T22:08:42.4588158Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4588698Z "size": 32, 2024-08-20T22:08:42.4589231Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:08:42.4589849Z }, 2024-08-20T22:08:42.4590089Z { 2024-08-20T22:08:42.4590509Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:42.4591054Z "size": 32, 2024-08-20T22:08:42.4591581Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:08:42.4592192Z } 2024-08-20T22:08:42.4592429Z ] 2024-08-20T22:08:42.4592670Z } 2024-08-20T22:08:42.4592943Z + exit 0 2024-08-20T22:08:42.4699864Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:08:42.4700353Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:08:42.4700891Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-20T22:08:42.4710270Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:42.4710773Z env: 2024-08-20T22:08:42.4711043Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:42.4712185Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:42.4713351Z ##[endgroup] 2024-08-20T22:08:42.4745033Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:42.4800109Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-20T22:08:42.4800693Z with: 2024-08-20T22:08:42.4801776Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:42.4803168Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:42.4803875Z env: 2024-08-20T22:08:42.4804152Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:42.4804500Z ##[endgroup] 2024-08-20T22:08:42.4822109Z ##[group]Run set -x 2024-08-20T22:08:42.4822457Z set -x 2024-08-20T22:08:42.4822763Z set +e 2024-08-20T22:08:42.4823056Z  2024-08-20T22:08:42.4823331Z login() { 2024-08-20T22:08:42.4824000Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:08:42.4824728Z } 2024-08-20T22:08:42.4825012Z  2024-08-20T22:08:42.4825314Z retry () { 2024-08-20T22:08:42.4825710Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:08:42.4826158Z } 2024-08-20T22:08:42.4826437Z  2024-08-20T22:08:42.4826754Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:08:42.4827169Z  2024-08-20T22:08:42.4827443Z set -e 2024-08-20T22:08:42.4827926Z # ignore output since only exit code is used for conditional 2024-08-20T22:08:42.4828649Z # only pull docker image if it's not available locally 2024-08-20T22:08:42.4829423Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-20T22:08:42.4830125Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-20T22:08:42.4830568Z fi 2024-08-20T22:08:42.4839413Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:42.4839909Z env: 2024-08-20T22:08:42.4840183Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:42.4841340Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:42.4842652Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:42.4843284Z ##[endgroup] 2024-08-20T22:08:42.4873021Z + set +e 2024-08-20T22:08:42.4873681Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:42.4874390Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:42.4877294Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:08:42.4878532Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:43.0127299Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:08:43.0128978Z Configure a credential helper to remove this warning. See 2024-08-20T22:08:43.0130860Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:08:43.0131950Z 2024-08-20T22:08:43.0132200Z Login Succeeded 2024-08-20T22:08:43.0150814Z + set -e 2024-08-20T22:08:43.0152272Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:43.0310547Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:43.0312764Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:43.2588463Z f6d216893d65c7b8ae43df4daaf247db808378e9: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-08-20T22:08:43.2589706Z 7a2c55901189: Pulling fs layer 2024-08-20T22:08:43.2590214Z 224fe954d725: Pulling fs layer 2024-08-20T22:08:43.2591023Z 75722010b82e: Pulling fs layer 2024-08-20T22:08:43.2591547Z d527cbbb87e3: Pulling fs layer 2024-08-20T22:08:43.2592045Z b57676e46aee: Pulling fs layer 2024-08-20T22:08:43.2592542Z a8c1e85b5e14: Pulling fs layer 2024-08-20T22:08:43.2593059Z a41a8d1c11c8: Pulling fs layer 2024-08-20T22:08:43.2593556Z 0c1227890755: Pulling fs layer 2024-08-20T22:08:43.2594042Z d8d1234baab3: Pulling fs layer 2024-08-20T22:08:43.2594543Z 7ed32bc8e469: Pulling fs layer 2024-08-20T22:08:43.2595336Z ec1e7978c1fe: Pulling fs layer 2024-08-20T22:08:43.2595837Z ac23c5fcc1ef: Pulling fs layer 2024-08-20T22:08:43.2596333Z ca426ffc9d2b: Pulling fs layer 2024-08-20T22:08:43.2596826Z 4373aa743caf: Pulling fs layer 2024-08-20T22:08:43.2597322Z 75b2abb16c68: Pulling fs layer 2024-08-20T22:08:43.2597861Z ef1e3ef1e85f: Pulling fs layer 2024-08-20T22:08:43.2598360Z 0bf61114d37b: Pulling fs layer 2024-08-20T22:08:43.2598859Z 2df849f72ba4: Pulling fs layer 2024-08-20T22:08:43.2599354Z 34c47d63c5de: Pulling fs layer 2024-08-20T22:08:43.2599862Z 1fef4bd19792: Pulling fs layer 2024-08-20T22:08:43.2600347Z 6efb96fc87ea: Pulling fs layer 2024-08-20T22:08:43.2600869Z b119c4fc6c58: Pulling fs layer 2024-08-20T22:08:43.2601373Z 33b639226eee: Pulling fs layer 2024-08-20T22:08:43.2601872Z d3547bd43146: Pulling fs layer 2024-08-20T22:08:43.2602353Z 5645520256fa: Pulling fs layer 2024-08-20T22:08:43.2602841Z 829ed7d91e38: Pulling fs layer 2024-08-20T22:08:43.2603474Z 385d0421d67c: Pulling fs layer 2024-08-20T22:08:43.2603978Z 7cbfce999d76: Pulling fs layer 2024-08-20T22:08:43.2604463Z 6658e558ebe4: Pulling fs layer 2024-08-20T22:08:43.2605001Z 33444a5a7e01: Pulling fs layer 2024-08-20T22:08:43.2605494Z 7c47122f4e09: Pulling fs layer 2024-08-20T22:08:43.2605979Z 1811ff4d3556: Pulling fs layer 2024-08-20T22:08:43.2606454Z d527cbbb87e3: Waiting 2024-08-20T22:08:43.2606885Z b57676e46aee: Waiting 2024-08-20T22:08:43.2607312Z ac23c5fcc1ef: Waiting 2024-08-20T22:08:43.2607735Z a8c1e85b5e14: Waiting 2024-08-20T22:08:43.2608135Z 2df849f72ba4: Waiting 2024-08-20T22:08:43.2608471Z ca426ffc9d2b: Waiting 2024-08-20T22:08:43.2608880Z 34c47d63c5de: Waiting 2024-08-20T22:08:43.2609317Z 33b639226eee: Waiting 2024-08-20T22:08:43.2609753Z 7754ee068b33: Pulling fs layer 2024-08-20T22:08:43.2610227Z d3547bd43146: Waiting 2024-08-20T22:08:43.2610655Z 0479fbac3361: Pulling fs layer 2024-08-20T22:08:43.2611167Z f1d6874dee9c: Pulling fs layer 2024-08-20T22:08:43.2611676Z afc429e87b59: Pulling fs layer 2024-08-20T22:08:43.2612157Z 385d0421d67c: Waiting 2024-08-20T22:08:43.2612588Z 019620471c25: Pulling fs layer 2024-08-20T22:08:43.2613050Z 0c1227890755: Waiting 2024-08-20T22:08:43.2613467Z d8d1234baab3: Waiting 2024-08-20T22:08:43.2613898Z 27aeb1b2cd5e: Pulling fs layer 2024-08-20T22:08:43.2614403Z 829c0e19f4b7: Pulling fs layer 2024-08-20T22:08:43.2614902Z 560cd6122a38: Pulling fs layer 2024-08-20T22:08:43.2615754Z 5a4eb19d7219: Pulling fs layer 2024-08-20T22:08:43.2616300Z da6a838a66c7: Pulling fs layer 2024-08-20T22:08:43.2616845Z cbb79a705353: Pulling fs layer 2024-08-20T22:08:43.2617418Z 4d4fa6802ab3: Pulling fs layer 2024-08-20T22:08:43.2617944Z 0f085d4fa279: Pulling fs layer 2024-08-20T22:08:43.2618452Z b119c4fc6c58: Waiting 2024-08-20T22:08:43.2618957Z 4a0412373305: Pulling fs layer 2024-08-20T22:08:43.2619475Z 302c8c177944: Pulling fs layer 2024-08-20T22:08:43.2619985Z 02970347dff6: Pulling fs layer 2024-08-20T22:08:43.2620493Z 1fef4bd19792: Waiting 2024-08-20T22:08:43.2621153Z 6efb96fc87ea: Waiting 2024-08-20T22:08:43.2621630Z 6658e558ebe4: Waiting 2024-08-20T22:08:43.2622065Z ef1e3ef1e85f: Waiting 2024-08-20T22:08:43.2622859Z 7cbfce999d76: Waiting 2024-08-20T22:08:43.2623194Z 5645520256fa: Waiting 2024-08-20T22:08:43.2623511Z 829ed7d91e38: Waiting 2024-08-20T22:08:43.2623904Z 33444a5a7e01: Waiting 2024-08-20T22:08:43.2624326Z ec1e7978c1fe: Waiting 2024-08-20T22:08:43.2624747Z 7ed32bc8e469: Waiting 2024-08-20T22:08:43.2625134Z 75b2abb16c68: Waiting 2024-08-20T22:08:43.2625445Z 0bf61114d37b: Waiting 2024-08-20T22:08:43.2625755Z 7c47122f4e09: Waiting 2024-08-20T22:08:43.2626305Z 916c5e858016: Pulling fs layer 2024-08-20T22:08:43.2626784Z 560cd6122a38: Waiting 2024-08-20T22:08:43.2627092Z 1811ff4d3556: Waiting 2024-08-20T22:08:43.2627403Z da6a838a66c7: Waiting 2024-08-20T22:08:43.2627921Z f1d6874dee9c: Waiting 2024-08-20T22:08:43.2628243Z 4a0412373305: Waiting 2024-08-20T22:08:43.2628562Z afc429e87b59: Waiting 2024-08-20T22:08:43.2628882Z 7754ee068b33: Waiting 2024-08-20T22:08:43.2629198Z 0479fbac3361: Waiting 2024-08-20T22:08:43.2629632Z 302c8c177944: Waiting 2024-08-20T22:08:43.2629941Z 0f085d4fa279: Waiting 2024-08-20T22:08:43.2630254Z 4d4fa6802ab3: Waiting 2024-08-20T22:08:43.2630558Z 02970347dff6: Waiting 2024-08-20T22:08:43.2630984Z 916c5e858016: Waiting 2024-08-20T22:08:43.2631296Z 27aeb1b2cd5e: Waiting 2024-08-20T22:08:43.2631610Z 5a4eb19d7219: Waiting 2024-08-20T22:08:43.2631916Z a41a8d1c11c8: Waiting 2024-08-20T22:08:43.2632432Z 019620471c25: Waiting 2024-08-20T22:08:43.2632762Z c42b3602a1fb: Pulling fs layer 2024-08-20T22:08:43.2633142Z 11fad972ef6c: Pulling fs layer 2024-08-20T22:08:43.2633505Z cbb79a705353: Waiting 2024-08-20T22:08:43.2633832Z f957c1624481: Pulling fs layer 2024-08-20T22:08:43.2634279Z f957c1624481: Waiting 2024-08-20T22:08:43.2634600Z 768c31dca016: Pulling fs layer 2024-08-20T22:08:43.2635005Z 768c31dca016: Waiting 2024-08-20T22:08:43.2635523Z 8c32d0ecebbe: Pulling fs layer 2024-08-20T22:08:43.2635905Z 4f4fb700ef54: Pulling fs layer 2024-08-20T22:08:43.2636275Z 78ee77eddef2: Pulling fs layer 2024-08-20T22:08:43.2636642Z 8c32d0ecebbe: Waiting 2024-08-20T22:08:43.2636964Z 78ee77eddef2: Waiting 2024-08-20T22:08:43.2637294Z 37513e9b43d7: Pulling fs layer 2024-08-20T22:08:43.2637646Z 37513e9b43d7: Waiting 2024-08-20T22:08:43.2637974Z a7a5392e2bf6: Pulling fs layer 2024-08-20T22:08:43.2638349Z b10de02255c0: Pulling fs layer 2024-08-20T22:08:43.2638701Z a7a5392e2bf6: Waiting 2024-08-20T22:08:43.2639017Z b10de02255c0: Waiting 2024-08-20T22:08:43.2639355Z e56eeff2db93: Pulling fs layer 2024-08-20T22:08:43.2639721Z e56eeff2db93: Waiting 2024-08-20T22:08:43.2640053Z 70ac3e38d0fd: Pulling fs layer 2024-08-20T22:08:43.2640431Z eb09cbf3e416: Pulling fs layer 2024-08-20T22:08:43.2640806Z ee1273153675: Pulling fs layer 2024-08-20T22:08:43.2641342Z e22979f9e0ee: Pulling fs layer 2024-08-20T22:08:43.2641724Z 6c1f0aabce7c: Pulling fs layer 2024-08-20T22:08:43.2642106Z 7ebe49c3a23e: Pulling fs layer 2024-08-20T22:08:43.2642617Z 70ac3e38d0fd: Waiting 2024-08-20T22:08:43.2642932Z eb09cbf3e416: Waiting 2024-08-20T22:08:43.2643353Z ee1273153675: Waiting 2024-08-20T22:08:43.2643665Z e22979f9e0ee: Waiting 2024-08-20T22:08:43.2643983Z 6c1f0aabce7c: Waiting 2024-08-20T22:08:43.2644297Z 7ebe49c3a23e: Waiting 2024-08-20T22:08:43.2644631Z 5d115b29b5d5: Pulling fs layer 2024-08-20T22:08:43.2645011Z 7f2d7a04fbe6: Pulling fs layer 2024-08-20T22:08:43.2645381Z d79fd7d52698: Pulling fs layer 2024-08-20T22:08:43.2645823Z 9752d6fbbcab: Pulling fs layer 2024-08-20T22:08:43.2646199Z 83808b2eff4d: Pulling fs layer 2024-08-20T22:08:43.2646557Z 83808b2eff4d: Waiting 2024-08-20T22:08:43.2646872Z 9752d6fbbcab: Waiting 2024-08-20T22:08:43.3972547Z 224fe954d725: Verifying Checksum 2024-08-20T22:08:43.3972983Z 224fe954d725: Download complete 2024-08-20T22:08:43.4897407Z d527cbbb87e3: Verifying Checksum 2024-08-20T22:08:43.4897846Z d527cbbb87e3: Download complete 2024-08-20T22:08:43.5568444Z b57676e46aee: Verifying Checksum 2024-08-20T22:08:43.6089359Z 7a2c55901189: Download complete 2024-08-20T22:08:43.7000281Z a41a8d1c11c8: Verifying Checksum 2024-08-20T22:08:43.7001239Z a41a8d1c11c8: Download complete 2024-08-20T22:08:43.7758485Z 0c1227890755: Download complete 2024-08-20T22:08:43.8479328Z d8d1234baab3: Verifying Checksum 2024-08-20T22:08:43.8479783Z d8d1234baab3: Download complete 2024-08-20T22:08:43.8674165Z 75722010b82e: Verifying Checksum 2024-08-20T22:08:43.8674658Z 75722010b82e: Download complete 2024-08-20T22:08:43.9708230Z ec1e7978c1fe: Download complete 2024-08-20T22:08:44.0659174Z ac23c5fcc1ef: Download complete 2024-08-20T22:08:44.7965276Z 7a2c55901189: Pull complete 2024-08-20T22:08:45.1053240Z 224fe954d725: Pull complete 2024-08-20T22:08:46.0149174Z 75722010b82e: Pull complete 2024-08-20T22:08:46.0325712Z d527cbbb87e3: Pull complete 2024-08-20T22:08:46.0513727Z b57676e46aee: Pull complete 2024-08-20T22:08:46.6204919Z ca426ffc9d2b: Verifying Checksum 2024-08-20T22:08:46.6205462Z ca426ffc9d2b: Download complete 2024-08-20T22:08:46.6938253Z 4373aa743caf: Verifying Checksum 2024-08-20T22:08:46.6938804Z 4373aa743caf: Download complete 2024-08-20T22:08:46.8020530Z 75b2abb16c68: Download complete 2024-08-20T22:08:46.8771897Z ef1e3ef1e85f: Verifying Checksum 2024-08-20T22:08:46.8772459Z ef1e3ef1e85f: Download complete 2024-08-20T22:08:47.8501806Z 0bf61114d37b: Verifying Checksum 2024-08-20T22:08:47.8502421Z 0bf61114d37b: Download complete 2024-08-20T22:08:47.9290893Z 2df849f72ba4: Verifying Checksum 2024-08-20T22:08:47.9291468Z 2df849f72ba4: Download complete 2024-08-20T22:08:47.9973718Z 34c47d63c5de: Download complete 2024-08-20T22:08:48.0795047Z 1fef4bd19792: Verifying Checksum 2024-08-20T22:08:48.0795668Z 1fef4bd19792: Download complete 2024-08-20T22:08:56.8993684Z a8c1e85b5e14: Verifying Checksum 2024-08-20T22:08:56.8994335Z a8c1e85b5e14: Download complete 2024-08-20T22:08:56.9848594Z b119c4fc6c58: Download complete 2024-08-20T22:08:57.0593664Z 33b639226eee: Verifying Checksum 2024-08-20T22:08:57.0594306Z 33b639226eee: Download complete 2024-08-20T22:08:57.1244798Z d3547bd43146: Verifying Checksum 2024-08-20T22:08:57.1245277Z d3547bd43146: Download complete 2024-08-20T22:08:57.2007580Z 5645520256fa: Download complete 2024-08-20T22:08:57.2693063Z 829ed7d91e38: Verifying Checksum 2024-08-20T22:08:57.2693625Z 829ed7d91e38: Download complete 2024-08-20T22:08:58.5354256Z 385d0421d67c: Verifying Checksum 2024-08-20T22:08:58.5354845Z 385d0421d67c: Download complete 2024-08-20T22:08:58.6197138Z 7cbfce999d76: Download complete 2024-08-20T22:08:58.6929277Z 6658e558ebe4: Verifying Checksum 2024-08-20T22:08:58.6929835Z 6658e558ebe4: Download complete 2024-08-20T22:08:58.7686728Z 33444a5a7e01: Download complete 2024-08-20T22:08:58.8498264Z 7c47122f4e09: Download complete 2024-08-20T22:08:58.9261071Z 1811ff4d3556: Download complete 2024-08-20T22:09:04.9300676Z 7754ee068b33: Verifying Checksum 2024-08-20T22:09:04.9301268Z 7754ee068b33: Download complete 2024-08-20T22:09:05.0036898Z 0479fbac3361: Verifying Checksum 2024-08-20T22:09:05.0037466Z 0479fbac3361: Download complete 2024-08-20T22:09:05.0743873Z f1d6874dee9c: Verifying Checksum 2024-08-20T22:09:05.0744468Z f1d6874dee9c: Download complete 2024-08-20T22:09:05.4886126Z afc429e87b59: Verifying Checksum 2024-08-20T22:09:05.4886599Z afc429e87b59: Download complete 2024-08-20T22:09:05.5702871Z 019620471c25: Verifying Checksum 2024-08-20T22:09:05.5703513Z 019620471c25: Download complete 2024-08-20T22:09:05.6680024Z 27aeb1b2cd5e: Verifying Checksum 2024-08-20T22:09:05.6680590Z 27aeb1b2cd5e: Download complete 2024-08-20T22:09:05.9438569Z 829c0e19f4b7: Verifying Checksum 2024-08-20T22:09:05.9439333Z 829c0e19f4b7: Download complete 2024-08-20T22:09:06.0342273Z 560cd6122a38: Verifying Checksum 2024-08-20T22:09:06.0342754Z 560cd6122a38: Download complete 2024-08-20T22:09:06.1158441Z 5a4eb19d7219: Verifying Checksum 2024-08-20T22:09:06.1158919Z 5a4eb19d7219: Download complete 2024-08-20T22:09:06.1931810Z da6a838a66c7: Verifying Checksum 2024-08-20T22:09:06.1932342Z da6a838a66c7: Download complete 2024-08-20T22:09:09.1770564Z 7ed32bc8e469: Verifying Checksum 2024-08-20T22:09:09.1771057Z 7ed32bc8e469: Download complete 2024-08-20T22:09:09.2422565Z 4d4fa6802ab3: Verifying Checksum 2024-08-20T22:09:09.2423103Z 4d4fa6802ab3: Download complete 2024-08-20T22:09:09.3182762Z 0f085d4fa279: Verifying Checksum 2024-08-20T22:09:09.3183269Z 0f085d4fa279: Download complete 2024-08-20T22:09:09.7994909Z 4a0412373305: Verifying Checksum 2024-08-20T22:09:09.7995499Z 4a0412373305: Download complete 2024-08-20T22:09:09.8822652Z 302c8c177944: Download complete 2024-08-20T22:09:09.9551881Z 02970347dff6: Verifying Checksum 2024-08-20T22:09:09.9552473Z 02970347dff6: Download complete 2024-08-20T22:09:10.0355171Z 916c5e858016: Verifying Checksum 2024-08-20T22:09:10.0355782Z 916c5e858016: Download complete 2024-08-20T22:09:10.1047627Z c42b3602a1fb: Download complete 2024-08-20T22:09:10.2634388Z a8c1e85b5e14: Pull complete 2024-08-20T22:09:10.4873345Z a41a8d1c11c8: Pull complete 2024-08-20T22:09:10.7051164Z 0c1227890755: Pull complete 2024-08-20T22:09:10.9397954Z d8d1234baab3: Pull complete 2024-08-20T22:09:16.7763138Z 6efb96fc87ea: Verifying Checksum 2024-08-20T22:09:16.7763965Z 6efb96fc87ea: Download complete 2024-08-20T22:09:16.8538199Z f957c1624481: Download complete 2024-08-20T22:09:16.9198331Z 768c31dca016: Download complete 2024-08-20T22:09:17.0160116Z 8c32d0ecebbe: Verifying Checksum 2024-08-20T22:09:17.0160574Z 8c32d0ecebbe: Download complete 2024-08-20T22:09:17.0236524Z 4f4fb700ef54: Verifying Checksum 2024-08-20T22:09:17.0237099Z 4f4fb700ef54: Download complete 2024-08-20T22:09:17.1015107Z 78ee77eddef2: Download complete 2024-08-20T22:09:17.1700122Z 37513e9b43d7: Verifying Checksum 2024-08-20T22:09:17.1700805Z 37513e9b43d7: Download complete 2024-08-20T22:09:17.2893222Z a7a5392e2bf6: Verifying Checksum 2024-08-20T22:09:17.2893854Z a7a5392e2bf6: Download complete 2024-08-20T22:09:17.3612720Z b10de02255c0: Verifying Checksum 2024-08-20T22:09:17.3613191Z b10de02255c0: Download complete 2024-08-20T22:09:17.4304425Z e56eeff2db93: Download complete 2024-08-20T22:09:17.5010441Z 70ac3e38d0fd: Download complete 2024-08-20T22:09:17.5729106Z eb09cbf3e416: Verifying Checksum 2024-08-20T22:09:17.5729753Z eb09cbf3e416: Download complete 2024-08-20T22:09:17.7040510Z ee1273153675: Verifying Checksum 2024-08-20T22:09:17.7041050Z ee1273153675: Download complete 2024-08-20T22:09:17.7756969Z e22979f9e0ee: Verifying Checksum 2024-08-20T22:09:17.7757565Z e22979f9e0ee: Download complete 2024-08-20T22:09:18.3613112Z 6c1f0aabce7c: Verifying Checksum 2024-08-20T22:09:18.3613717Z 6c1f0aabce7c: Download complete 2024-08-20T22:09:18.4495215Z 7ebe49c3a23e: Verifying Checksum 2024-08-20T22:09:18.4495790Z 7ebe49c3a23e: Download complete 2024-08-20T22:09:46.2078443Z 5d115b29b5d5: Verifying Checksum 2024-08-20T22:09:46.2082187Z 5d115b29b5d5: Download complete 2024-08-20T22:09:46.2868172Z 7f2d7a04fbe6: Verifying Checksum 2024-08-20T22:09:46.2868685Z 7f2d7a04fbe6: Download complete 2024-08-20T22:09:46.3568863Z d79fd7d52698: Verifying Checksum 2024-08-20T22:09:46.3569435Z d79fd7d52698: Download complete 2024-08-20T22:09:47.8100299Z 9752d6fbbcab: Verifying Checksum 2024-08-20T22:09:47.8100777Z 9752d6fbbcab: Download complete 2024-08-20T22:09:47.8810984Z 83808b2eff4d: Verifying Checksum 2024-08-20T22:09:47.8811443Z 83808b2eff4d: Download complete 2024-08-20T22:10:25.9593912Z 11fad972ef6c: Verifying Checksum 2024-08-20T22:10:25.9596109Z 11fad972ef6c: Download complete 2024-08-20T22:10:27.0908596Z cbb79a705353: Verifying Checksum 2024-08-20T22:10:27.0909055Z cbb79a705353: Download complete 2024-08-20T22:10:42.3027072Z 7ed32bc8e469: Pull complete 2024-08-20T22:10:42.5421203Z ec1e7978c1fe: Pull complete 2024-08-20T22:10:42.7610190Z ac23c5fcc1ef: Pull complete 2024-08-20T22:10:51.4609206Z ca426ffc9d2b: Pull complete 2024-08-20T22:10:51.6876974Z 4373aa743caf: Pull complete 2024-08-20T22:10:51.9176113Z 75b2abb16c68: Pull complete 2024-08-20T22:10:52.1287285Z ef1e3ef1e85f: Pull complete 2024-08-20T22:10:54.6505326Z 0bf61114d37b: Pull complete 2024-08-20T22:10:54.7547597Z 2df849f72ba4: Pull complete 2024-08-20T22:10:54.8346731Z 34c47d63c5de: Pull complete 2024-08-20T22:10:54.9534300Z 1fef4bd19792: Pull complete 2024-08-20T22:11:53.2755069Z 6efb96fc87ea: Pull complete 2024-08-20T22:11:53.4862970Z b119c4fc6c58: Pull complete 2024-08-20T22:11:53.7034847Z 33b639226eee: Pull complete 2024-08-20T22:11:53.9241820Z d3547bd43146: Pull complete 2024-08-20T22:11:54.0952465Z 5645520256fa: Pull complete 2024-08-20T22:11:54.1936135Z 829ed7d91e38: Pull complete 2024-08-20T22:11:57.7755849Z 385d0421d67c: Pull complete 2024-08-20T22:11:57.9968519Z 7cbfce999d76: Pull complete 2024-08-20T22:11:58.2066203Z 6658e558ebe4: Pull complete 2024-08-20T22:11:58.4145798Z 33444a5a7e01: Pull complete 2024-08-20T22:11:58.6280610Z 7c47122f4e09: Pull complete 2024-08-20T22:11:58.8389308Z 1811ff4d3556: Pull complete 2024-08-20T22:12:08.9979168Z 7754ee068b33: Pull complete 2024-08-20T22:12:09.1441523Z 0479fbac3361: Pull complete 2024-08-20T22:12:09.2923220Z f1d6874dee9c: Pull complete 2024-08-20T22:12:10.2807622Z afc429e87b59: Pull complete 2024-08-20T22:12:10.4890380Z 019620471c25: Pull complete 2024-08-20T22:12:10.6948787Z 27aeb1b2cd5e: Pull complete 2024-08-20T22:12:11.1972360Z 829c0e19f4b7: Pull complete 2024-08-20T22:12:11.3661160Z 560cd6122a38: Pull complete 2024-08-20T22:12:11.7374045Z 5a4eb19d7219: Pull complete 2024-08-20T22:12:11.9399112Z da6a838a66c7: Pull complete 2024-08-20T22:12:50.6274956Z cbb79a705353: Pull complete 2024-08-20T22:12:50.8397205Z 4d4fa6802ab3: Pull complete 2024-08-20T22:12:50.9899395Z 0f085d4fa279: Pull complete 2024-08-20T22:12:53.3754241Z 4a0412373305: Pull complete 2024-08-20T22:12:53.3980091Z 302c8c177944: Pull complete 2024-08-20T22:12:53.4174525Z 02970347dff6: Pull complete 2024-08-20T22:12:53.4569529Z 916c5e858016: Pull complete 2024-08-20T22:12:53.4781355Z c42b3602a1fb: Pull complete 2024-08-20T22:14:15.2814599Z 11fad972ef6c: Pull complete 2024-08-20T22:14:15.4755995Z f957c1624481: Pull complete 2024-08-20T22:14:15.5723845Z 768c31dca016: Pull complete 2024-08-20T22:14:15.7642517Z 8c32d0ecebbe: Pull complete 2024-08-20T22:14:15.8234029Z 4f4fb700ef54: Pull complete 2024-08-20T22:14:15.9569789Z 78ee77eddef2: Pull complete 2024-08-20T22:14:16.1395356Z 37513e9b43d7: Pull complete 2024-08-20T22:14:16.5441340Z a7a5392e2bf6: Pull complete 2024-08-20T22:14:16.6206298Z b10de02255c0: Pull complete 2024-08-20T22:14:16.7552780Z e56eeff2db93: Pull complete 2024-08-20T22:14:16.9345965Z 70ac3e38d0fd: Pull complete 2024-08-20T22:14:17.1230176Z eb09cbf3e416: Pull complete 2024-08-20T22:14:18.5494728Z ee1273153675: Pull complete 2024-08-20T22:14:18.7282257Z e22979f9e0ee: Pull complete 2024-08-20T22:14:21.4441081Z 6c1f0aabce7c: Pull complete 2024-08-20T22:14:21.5460221Z 7ebe49c3a23e: Pull complete 2024-08-20T22:15:08.6839313Z 5d115b29b5d5: Pull complete 2024-08-20T22:15:08.7916124Z 7f2d7a04fbe6: Pull complete 2024-08-20T22:15:08.9932320Z d79fd7d52698: Pull complete 2024-08-20T22:15:10.9823476Z 9752d6fbbcab: Pull complete 2024-08-20T22:15:11.0840975Z 83808b2eff4d: Pull complete 2024-08-20T22:15:11.9839928Z Digest: sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T22:15:12.0135532Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:15:12.0307486Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:15:12.0401784Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:15:12.0402722Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:15:12.0412915Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:12.0413424Z env: 2024-08-20T22:15:12.0413693Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:12.0414045Z ##[endgroup] 2024-08-20T22:15:12.0595017Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-08-20T22:15:12.0595576Z with: 2024-08-20T22:15:12.0595863Z driver-version: 550.54.15 2024-08-20T22:15:12.0596225Z env: 2024-08-20T22:15:12.0596504Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:12.0596865Z ##[endgroup] 2024-08-20T22:15:12.0693952Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:15:12.0694511Z with: 2024-08-20T22:15:12.0694788Z timeout_minutes: 10 2024-08-20T22:15:12.0695114Z max_attempts: 3 2024-08-20T22:15:12.0728067Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-08-20T22:15:12.0761582Z retry_wait_seconds: 10 2024-08-20T22:15:12.0761948Z polling_interval_seconds: 1 2024-08-20T22:15:12.0762316Z warning_on_retry: true 2024-08-20T22:15:12.0762666Z continue_on_error: false 2024-08-20T22:15:12.0763005Z env: 2024-08-20T22:15:12.0763324Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:12.0763668Z DRIVER_VERSION: 550.54.15 2024-08-20T22:15:12.0764015Z ##[endgroup] 2024-08-20T22:15:12.5204095Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-08-20T22:15:12.5204958Z + pre_install_nvidia_driver_amzn2 2024-08-20T22:15:12.5205591Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-08-20T22:15:13.9874778Z No match for argument: nvidia-driver-latest-dkms 2024-08-20T22:15:13.9875336Z No packages marked for removal. 2024-08-20T22:15:14.0143246Z Dependencies resolved. 2024-08-20T22:15:14.0153123Z Nothing to do. 2024-08-20T22:15:14.0153422Z Complete! 2024-08-20T22:15:14.1059146Z + install_nvidia_driver_common 2024-08-20T22:15:14.1062733Z + echo 'Before installing NVIDIA driver' 2024-08-20T22:15:14.1063172Z + lspci 2024-08-20T22:15:14.1063460Z Before installing NVIDIA driver 2024-08-20T22:15:14.1177965Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:15:14.1178827Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:15:14.1179780Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:15:14.1180751Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:15:14.1181574Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:15:14.1182383Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:15:14.1183134Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:15:14.1183954Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:15:14.1184544Z + lsmod 2024-08-20T22:15:14.1382462Z Module Size Used by 2024-08-20T22:15:14.1383047Z xt_nat 16384 0 2024-08-20T22:15:14.1383430Z veth 36864 0 2024-08-20T22:15:14.1383804Z nvidia_modeset 1351680 0 2024-08-20T22:15:14.1384231Z video 65536 1 nvidia_modeset 2024-08-20T22:15:14.1384679Z wmi 36864 1 video 2024-08-20T22:15:14.1385076Z nvidia_uvm 4706304 0 2024-08-20T22:15:14.1385524Z nvidia 54071296 7 nvidia_uvm,nvidia_modeset 2024-08-20T22:15:14.1386013Z drm 602112 1 nvidia 2024-08-20T22:15:14.1389259Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T22:15:14.1390014Z backlight 24576 3 video,drm,nvidia_modeset 2024-08-20T22:15:14.1390665Z i2c_core 106496 2 nvidia,drm 2024-08-20T22:15:14.1391169Z xt_conntrack 16384 1 2024-08-20T22:15:14.1391557Z nft_chain_nat 16384 3 2024-08-20T22:15:14.1391941Z xt_MASQUERADE 20480 1 2024-08-20T22:15:14.1392561Z nf_nat 57344 3 xt_nat,nft_chain_nat,xt_MASQUERADE 2024-08-20T22:15:14.1393238Z nf_conntrack_netlink 57344 0 2024-08-20T22:15:14.1394071Z nf_conntrack 184320 5 xt_conntrack,nf_nat,xt_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:15:14.1395227Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:15:14.1395752Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:15:14.1396232Z xfrm_user 57344 1 2024-08-20T22:15:14.1396629Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:15:14.1397053Z xt_addrtype 16384 2 2024-08-20T22:15:14.1397431Z nft_compat 20480 4 2024-08-20T22:15:14.1397886Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:15:14.1398506Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:15:14.1399059Z br_netfilter 36864 0 2024-08-20T22:15:14.1399474Z bridge 307200 1 br_netfilter 2024-08-20T22:15:14.1399912Z stp 16384 1 bridge 2024-08-20T22:15:14.1400342Z llc 16384 2 bridge,stp 2024-08-20T22:15:14.1400766Z overlay 167936 0 2024-08-20T22:15:14.1401129Z tls 114688 0 2024-08-20T22:15:14.1401504Z nls_ascii 16384 1 2024-08-20T22:15:14.1401874Z nls_cp437 20480 1 2024-08-20T22:15:14.1402380Z vfat 24576 1 2024-08-20T22:15:14.1402864Z fat 86016 1 vfat 2024-08-20T22:15:14.1403547Z ena 167936 0 2024-08-20T22:15:14.1403994Z ghash_clmulni_intel 16384 0 2024-08-20T22:15:14.1404384Z ptp 36864 1 ena 2024-08-20T22:15:14.1404783Z pps_core 24576 1 ptp 2024-08-20T22:15:14.1405176Z aesni_intel 393216 0 2024-08-20T22:15:14.1405592Z sunrpc 692224 1 2024-08-20T22:15:14.1406078Z i8042 45056 0 2024-08-20T22:15:14.1406496Z serio 28672 3 i8042 2024-08-20T22:15:14.1407010Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:15:14.1407719Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:15:14.1408339Z button 24576 0 2024-08-20T22:15:14.1408790Z sch_fq_codel 20480 17 2024-08-20T22:15:14.1409283Z dm_mod 188416 0 2024-08-20T22:15:14.1409698Z fuse 163840 1 2024-08-20T22:15:14.1410054Z configfs 57344 1 2024-08-20T22:15:14.1410419Z loop 36864 0 2024-08-20T22:15:14.1410794Z dax 45056 1 dm_mod 2024-08-20T22:15:14.1411188Z dmi_sysfs 20480 0 2024-08-20T22:15:14.1411549Z crc32_pclmul 16384 0 2024-08-20T22:15:14.1411916Z crc32c_intel 24576 0 2024-08-20T22:15:14.1412381Z efivarfs 24576 1 2024-08-20T22:15:14.1412885Z + modinfo nvidia 2024-08-20T22:15:14.1413722Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:15:14.1414425Z alias: char-major-195-* 2024-08-20T22:15:14.1414938Z version: 550.54.15 2024-08-20T22:15:14.1415290Z supported: external 2024-08-20T22:15:14.1415621Z license: NVIDIA 2024-08-20T22:15:14.1415996Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:15:14.1416621Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:15:14.1417216Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:15:14.1417723Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:15:14.1418223Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:15:14.1418706Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:15:14.1419204Z depends: i2c-core,drm 2024-08-20T22:15:14.1419561Z retpoline: Y 2024-08-20T22:15:14.1419862Z name: nvidia 2024-08-20T22:15:14.1420470Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:15:14.1421166Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:15:14.1421822Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:15:14.1422416Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:15:14.1422855Z parm: NVreg_RmLogonRC:int 2024-08-20T22:15:14.1423283Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:15:14.1423858Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:15:14.1424292Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:15:14.1424729Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:15:14.1425250Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:15:14.1425817Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:15:14.1426295Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:15:14.1426722Z parm: NVreg_EnableMSI:int 2024-08-20T22:15:14.1427133Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:15:14.1427587Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:15:14.1428116Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:15:14.1428698Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:15:14.1429262Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:15:14.1429871Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:15:14.1430480Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:15:14.1431088Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:15:14.1431681Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:15:14.1432171Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:15:14.1432706Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:15:14.1433241Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:15:14.1433734Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:15:14.1434200Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:15:14.1434682Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:15:14.1435144Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:15:14.1435593Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:15:14.1436093Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:15:14.1436615Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:15:14.1437092Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:15:14.1437579Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:15:14.1438078Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:15:14.1438563Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:15:14.1439058Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:15:14.1439535Z parm: NVreg_RmMsg:charp 2024-08-20T22:15:14.1439945Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:15:14.1440405Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:15:14.1440877Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:15:14.1441335Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:15:14.1441815Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:15:14.1442384Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:15:14.1442892Z parm: rm_firmware_active:charp 2024-08-20T22:15:14.1443416Z + HAS_NVIDIA_DRIVER=0 2024-08-20T22:15:14.1443800Z ++ command -v nvidia-smi 2024-08-20T22:15:14.1444198Z + '[' -x /usr/bin/nvidia-smi ']' 2024-08-20T22:15:14.1444564Z + set +e 2024-08-20T22:15:14.1445081Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-08-20T22:15:14.1684865Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-08-20T22:15:14.1685455Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:15:14.1685991Z + '[' 0 -ne 0 ']' 2024-08-20T22:15:14.1686464Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-08-20T22:15:14.1686846Z + HAS_NVIDIA_DRIVER=1 2024-08-20T22:15:14.1687571Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-08-20T22:15:14.1688301Z + set -e 2024-08-20T22:15:14.1688920Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-08-20T22:15:14.1689835Z + '[' 1 -eq 0 ']' 2024-08-20T22:15:14.1690166Z + post_install_nvidia_driver_common 2024-08-20T22:15:14.1692130Z + sudo modprobe nvidia 2024-08-20T22:15:14.3462813Z + echo 'After installing NVIDIA driver' 2024-08-20T22:15:14.3464382Z + lspci 2024-08-20T22:15:14.3465142Z After installing NVIDIA driver 2024-08-20T22:15:14.3568381Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:15:14.3569205Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:15:14.3570153Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:15:14.3571108Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:15:14.3572031Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:15:14.3572838Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:15:14.3573584Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:15:14.3574413Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:15:14.3575019Z + lsmod 2024-08-20T22:15:14.3603286Z Module Size Used by 2024-08-20T22:15:14.3603850Z xt_nat 16384 0 2024-08-20T22:15:14.3604336Z veth 36864 0 2024-08-20T22:15:14.3604845Z nvidia_modeset 1351680 0 2024-08-20T22:15:14.3605400Z video 65536 1 nvidia_modeset 2024-08-20T22:15:14.3605994Z wmi 36864 1 video 2024-08-20T22:15:14.3606492Z nvidia_uvm 4706304 0 2024-08-20T22:15:14.3606937Z nvidia 54071296 7 nvidia_uvm,nvidia_modeset 2024-08-20T22:15:14.3607418Z drm 602112 1 nvidia 2024-08-20T22:15:14.3607858Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T22:15:14.3608385Z backlight 24576 3 video,drm,nvidia_modeset 2024-08-20T22:15:14.3608888Z i2c_core 106496 2 nvidia,drm 2024-08-20T22:15:14.3609314Z xt_conntrack 16384 1 2024-08-20T22:15:14.3609700Z nft_chain_nat 16384 3 2024-08-20T22:15:14.3610075Z xt_MASQUERADE 20480 1 2024-08-20T22:15:14.3610555Z nf_nat 57344 3 xt_nat,nft_chain_nat,xt_MASQUERADE 2024-08-20T22:15:14.3611073Z nf_conntrack_netlink 57344 0 2024-08-20T22:15:14.3611692Z nf_conntrack 184320 5 xt_conntrack,nf_nat,xt_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:15:14.3612357Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:15:14.3612823Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:15:14.3613251Z xfrm_user 57344 1 2024-08-20T22:15:14.3613644Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:15:14.3614056Z xt_addrtype 16384 2 2024-08-20T22:15:14.3614436Z nft_compat 20480 4 2024-08-20T22:15:14.3614894Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:15:14.3615511Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:15:14.3616265Z br_netfilter 36864 0 2024-08-20T22:15:14.3616680Z bridge 307200 1 br_netfilter 2024-08-20T22:15:14.3617117Z stp 16384 1 bridge 2024-08-20T22:15:14.3617543Z llc 16384 2 bridge,stp 2024-08-20T22:15:14.3617958Z overlay 167936 0 2024-08-20T22:15:14.3618325Z tls 114688 0 2024-08-20T22:15:14.3618691Z nls_ascii 16384 1 2024-08-20T22:15:14.3619054Z nls_cp437 20480 1 2024-08-20T22:15:14.3619423Z vfat 24576 1 2024-08-20T22:15:14.3619789Z fat 86016 1 vfat 2024-08-20T22:15:14.3620176Z ena 167936 0 2024-08-20T22:15:14.3620598Z ghash_clmulni_intel 16384 0 2024-08-20T22:15:14.3621124Z ptp 36864 1 ena 2024-08-20T22:15:14.3621668Z pps_core 24576 1 ptp 2024-08-20T22:15:14.3622208Z aesni_intel 393216 0 2024-08-20T22:15:14.3622696Z sunrpc 692224 1 2024-08-20T22:15:14.3623174Z i8042 45056 0 2024-08-20T22:15:14.3623548Z serio 28672 3 i8042 2024-08-20T22:15:14.3623970Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:15:14.3624490Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:15:14.3625148Z button 24576 0 2024-08-20T22:15:14.3625523Z sch_fq_codel 20480 17 2024-08-20T22:15:14.3625893Z dm_mod 188416 0 2024-08-20T22:15:14.3626258Z fuse 163840 1 2024-08-20T22:15:14.3626743Z configfs 57344 1 2024-08-20T22:15:14.3627179Z loop 36864 0 2024-08-20T22:15:14.3627546Z dax 45056 1 dm_mod 2024-08-20T22:15:14.3627947Z dmi_sysfs 20480 0 2024-08-20T22:15:14.3628323Z crc32_pclmul 16384 0 2024-08-20T22:15:14.3628694Z crc32c_intel 24576 0 2024-08-20T22:15:14.3629053Z efivarfs 24576 1 2024-08-20T22:15:14.3629415Z + modinfo nvidia 2024-08-20T22:15:14.3630133Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:15:14.3630844Z alias: char-major-195-* 2024-08-20T22:15:14.3631223Z version: 550.54.15 2024-08-20T22:15:14.3631583Z supported: external 2024-08-20T22:15:14.3631922Z license: NVIDIA 2024-08-20T22:15:14.3632299Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:15:14.3632776Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:15:14.3633248Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:15:14.3633714Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:15:14.3634216Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:15:14.3634698Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:15:14.3635194Z depends: i2c-core,drm 2024-08-20T22:15:14.3635553Z retpoline: Y 2024-08-20T22:15:14.3635855Z name: nvidia 2024-08-20T22:15:14.3636474Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:15:14.3637177Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:15:14.3637829Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:15:14.3638423Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:15:14.3638868Z parm: NVreg_RmLogonRC:int 2024-08-20T22:15:14.3639304Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:15:14.3639763Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:15:14.3640197Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:15:14.3640645Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:15:14.3641176Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:15:14.3641741Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:15:14.3642216Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:15:14.3642648Z parm: NVreg_EnableMSI:int 2024-08-20T22:15:14.3643166Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:15:14.3643758Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:15:14.3644296Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:15:14.3644888Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:15:14.3645460Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:15:14.3646070Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:15:14.3646673Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:15:14.3647287Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:15:14.3647889Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:15:14.3648378Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:15:14.3648920Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:15:14.3649479Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:15:14.3649979Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:15:14.3650438Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:15:14.3650937Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:15:14.3651412Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:15:14.3651866Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:15:14.3652458Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:15:14.3652992Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:15:14.3653473Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:15:14.3653963Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:15:14.3654458Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:15:14.3654948Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:15:14.3655450Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:15:14.3655942Z parm: NVreg_RmMsg:charp 2024-08-20T22:15:14.3656350Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:15:14.3656824Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:15:14.3657300Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:15:14.3657756Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:15:14.3658241Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:15:14.3658719Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:15:14.3659183Z parm: rm_firmware_active:charp 2024-08-20T22:15:14.3659580Z + set +e 2024-08-20T22:15:14.3660239Z + nvidia-smi 2024-08-20T22:15:14.3827020Z Tue Aug 20 22:15:14 2024 2024-08-20T22:15:14.3828501Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:15:14.3830149Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:15:14.3831752Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:15:14.3832910Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:15:14.3833839Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:15:14.3834541Z | | | MIG M. | 2024-08-20T22:15:14.3835086Z |=========================================+========================+======================| 2024-08-20T22:15:14.4077004Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:15:14.4077821Z | 0% 26C P8 24W / 300W | 0MiB / 23028MiB | 0% Default | 2024-08-20T22:15:14.4078460Z | | | N/A | 2024-08-20T22:15:14.4079148Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:15:14.4082214Z 2024-08-20T22:15:14.4083034Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:15:14.4083960Z | Processes: | 2024-08-20T22:15:14.4084689Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:15:14.4085392Z | ID ID Usage | 2024-08-20T22:15:14.4085962Z |=========================================================================================| 2024-08-20T22:15:14.4087431Z | No running processes found | 2024-08-20T22:15:14.4088274Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:15:14.6964071Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-08-20T22:15:14.7152689Z NVIDIA A10G 2024-08-20T22:15:14.7216821Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:15:14.7217514Z + '[' 0 -eq 0 ']' 2024-08-20T22:15:14.7217933Z + echo 'INFO: Ignoring allowed status 0' 2024-08-20T22:15:14.7218364Z + set -e 2024-08-20T22:15:14.7218691Z INFO: Ignoring allowed status 0 2024-08-20T22:15:14.7226688Z == Installing nvidia container toolkit for amzn2023 == 2024-08-20T22:15:14.7230285Z + sudo yum install -y yum-utils 2024-08-20T22:15:15.1553216Z Last metadata expiration check: 0:34:07 ago on Tue Aug 20 21:41:08 2024. 2024-08-20T22:15:15.1775940Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-08-20T22:15:15.2076255Z Dependencies resolved. 2024-08-20T22:15:15.2199431Z Nothing to do. 2024-08-20T22:15:15.2199901Z Complete! 2024-08-20T22:15:15.2792445Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-08-20T22:15:15.2793881Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:15:15.2795336Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:15:15.5664498Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:15:15.6351904Z + sudo yum install -y nvidia-docker2 2024-08-20T22:15:16.1014517Z nvidia-container-toolkit 11 kB/s | 833 B 00:00 2024-08-20T22:15:16.1230875Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-08-20T22:15:16.1526355Z Dependencies resolved. 2024-08-20T22:15:16.1648617Z Nothing to do. 2024-08-20T22:15:16.1649034Z Complete! 2024-08-20T22:15:16.2418218Z + sudo systemctl restart docker 2024-08-20T22:15:50.4924922Z nvidia-persistenced failed to initialize. Check syslog for more details. 2024-08-20T22:15:50.5152954Z Tue Aug 20 22:15:50 2024 2024-08-20T22:15:50.5154695Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:15:50.5156091Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:15:50.5156964Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:15:50.5157805Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:15:50.5159053Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:15:50.5160137Z | | | MIG M. | 2024-08-20T22:15:50.5160701Z |=========================================+========================+======================| 2024-08-20T22:15:50.5361535Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:15:50.5363520Z | 0% 26C P8 24W / 300W | 0MiB / 23028MiB | 0% Default | 2024-08-20T22:15:50.5364753Z | | | N/A | 2024-08-20T22:15:50.5365794Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:15:50.5366999Z 2024-08-20T22:15:50.5367953Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:15:50.5368774Z | Processes: | 2024-08-20T22:15:50.5369507Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:15:50.5370195Z | ID ID Usage | 2024-08-20T22:15:50.5370753Z |=========================================================================================| 2024-08-20T22:15:50.5371402Z | No running processes found | 2024-08-20T22:15:50.5372189Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:15:51.1884608Z Command completed after 1 attempt(s). 2024-08-20T22:15:51.1964822Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:15:51.1965600Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:15:51.1966295Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-20T22:15:51.1967138Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:15:51.1981059Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:51.1981557Z env: 2024-08-20T22:15:51.1981832Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:51.1982289Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:51.1982764Z ##[endgroup] 2024-08-20T22:15:51.6644814Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:15:52.0877502Z Collecting psutil==5.9.1 2024-08-20T22:15:52.1232089Z Downloading psutil-5.9.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2024-08-20T22:15:52.1736658Z Collecting nvidia-ml-py==11.525.84 2024-08-20T22:15:52.1883594Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2024-08-20T22:15:52.2737287Z Installing collected packages: psutil, nvidia-ml-py 2024-08-20T22:15:52.4321070Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1 2024-08-20T22:15:52.6296498Z Prepare all required actions 2024-08-20T22:15:52.6297005Z Getting action download info 2024-08-20T22:15:52.7422586Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-20T22:15:52.9972488Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-20T22:15:53.1236978Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-20T22:15:53.1237473Z with: 2024-08-20T22:15:53.1237807Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:15:53.1238267Z s3-bucket: gha-artifacts 2024-08-20T22:15:53.1238606Z env: 2024-08-20T22:15:53.1238886Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:53.1239334Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:53.1239825Z ##[endgroup] 2024-08-20T22:15:53.1274673Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:15:53.1275124Z with: 2024-08-20T22:15:53.1275520Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:15:53.1275982Z s3-bucket: gha-artifacts 2024-08-20T22:15:53.1276354Z region: us-east-1 2024-08-20T22:15:53.1276657Z env: 2024-08-20T22:15:53.1276941Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:53.1277392Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:53.1277880Z ##[endgroup] 2024-08-20T22:15:53.6016879Z (node:182560) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:15:53.6017629Z 2024-08-20T22:15:53.6017905Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:15:53.6018636Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:15:53.6019509Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:15:53.6711512Z Found 1 objects with prefix pytorch/pytorch/10479310721/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-08-20T22:15:53.6712618Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:16:01.1987848Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:16:01.1994408Z Artifact download has finished successfully 2024-08-20T22:16:01.2346169Z ##[group]Run unzip -o artifacts.zip 2024-08-20T22:16:01.2346613Z unzip -o artifacts.zip 2024-08-20T22:16:01.2355587Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:16:01.2356098Z env: 2024-08-20T22:16:01.2356374Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:16:01.2356837Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:16:01.2357327Z ##[endgroup] 2024-08-20T22:16:01.2462662Z Archive: artifacts.zip 2024-08-20T22:16:01.2463747Z creating: dist/ 2024-08-20T22:16:03.3665638Z inflating: dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:16:03.3666533Z creating: build/custom_test_artifacts/ 2024-08-20T22:16:03.3667347Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-20T22:16:03.3668376Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-20T22:16:03.3669228Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:16:03.3675054Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:16:03.3676149Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-20T22:16:03.3677205Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:16:03.3678245Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:16:03.3679247Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:16:03.3680507Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:16:03.3682039Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:16:03.3683231Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:16:03.3684248Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:16:03.3685537Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:16:03.3687054Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:16:03.3688827Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:16:03.3690107Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:16:03.3691517Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:16:03.3693019Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:16:03.3694227Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:16:03.3695269Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:16:03.3734921Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:16:03.3775195Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:16:03.3776896Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:16:03.3821677Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:16:03.3824258Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:16:03.3827568Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:16:03.3831617Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:16:03.3833776Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:16:03.3835274Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:16:03.3836762Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:16:03.3838186Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:16:03.3839767Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:16:03.3841084Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:16:03.3842354Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:16:03.3843684Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:16:03.3844943Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:16:03.3846164Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:16:03.3847392Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:16:03.3897011Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:16:03.3957180Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:16:03.3958631Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:16:03.3959966Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:16:03.3961189Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:16:03.3962207Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-20T22:16:03.3963293Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-20T22:16:03.3964477Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-20T22:16:03.3965703Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-20T22:16:03.3966959Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-20T22:16:03.3968092Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-20T22:16:03.3969470Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-20T22:16:03.3970580Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-20T22:16:03.3971674Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-20T22:16:03.3972789Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-20T22:16:03.3973884Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-20T22:16:03.3990055Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-20T22:16:03.4133266Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-20T22:16:03.4134314Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-20T22:16:03.4135438Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-20T22:16:03.4136980Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-20T22:16:03.4138309Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-20T22:16:03.4139540Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-20T22:16:03.4140684Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-20T22:16:03.4141855Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-20T22:16:03.4143269Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-20T22:16:03.4144431Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-20T22:16:03.4145571Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-20T22:16:03.4166398Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-20T22:16:03.4247524Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-20T22:16:03.4248729Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:16:03.4249819Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:16:03.4251066Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-20T22:16:03.4252016Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-20T22:16:03.4253053Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-20T22:16:03.4253972Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-08-20T22:16:03.4257093Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-20T22:16:03.4258043Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-20T22:16:03.4258940Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-20T22:16:03.4379477Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-20T22:16:03.4442198Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-20T22:16:03.4442966Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-20T22:16:03.4443785Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-20T22:16:03.4444624Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:16:03.4451641Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:16:03.4452591Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-20T22:16:03.4453546Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:16:03.4454555Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:16:03.4455535Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:16:03.4456773Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:16:03.4458600Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:16:03.4459632Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:16:03.4460895Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:16:03.4462710Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:16:03.4464466Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:16:03.4466539Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:16:03.4467703Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:16:03.4469239Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:16:03.4470569Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:16:03.4471857Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:16:03.4472891Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:16:03.4512744Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:16:03.4552192Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:16:03.4553706Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:16:03.4599607Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:16:03.4601060Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:16:03.4602742Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:16:03.4604337Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:16:03.4605782Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:16:03.4607193Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:16:03.4608618Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:16:03.4610044Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:16:03.4611445Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:16:03.4612760Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:16:03.4614017Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:16:03.4615268Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:16:03.4616528Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:16:03.4617747Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:16:03.4618992Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:16:03.4677608Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:16:03.4737821Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:16:03.4739032Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:16:03.4740043Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:16:03.4740925Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:16:03.4741832Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-20T22:16:03.4742805Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-20T22:16:03.4743901Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-20T22:16:03.4745114Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-20T22:16:03.4746289Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-20T22:16:03.4747556Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-20T22:16:03.4748689Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-20T22:16:03.4749825Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-20T22:16:03.4750968Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-20T22:16:03.4752106Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-20T22:16:03.4753221Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-20T22:16:03.4772229Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-20T22:16:03.4836479Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-20T22:16:03.4837679Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:16:03.4838777Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:16:03.4839756Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-20T22:16:03.4840679Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-20T22:16:03.4841712Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-20T22:16:03.4842654Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-08-20T22:16:03.4845529Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-20T22:16:03.4846548Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-20T22:16:03.4847584Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-20T22:16:03.4898086Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-20T22:16:03.4898867Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-20T22:16:03.4899655Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-20T22:16:03.4900567Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:16:03.4907180Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:16:03.4908195Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-20T22:16:03.4909217Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:16:03.4910300Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:16:03.4911362Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:16:03.4912595Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:16:03.4914137Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:16:03.4915235Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:16:03.4916312Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:16:03.4918098Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:16:03.4919806Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:16:03.4921693Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:16:03.4923209Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:16:03.4924844Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:16:03.4926465Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:16:03.4927707Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:16:03.4928917Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:16:03.4969673Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:16:03.5009014Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:16:03.5010729Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:16:03.5055944Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:16:03.5057440Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:16:03.5058964Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:16:03.5060866Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:16:03.5062371Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:16:03.5063851Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:16:03.5065337Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:16:03.5066803Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:16:03.5068244Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:16:03.5069608Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:16:03.5070929Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:16:03.5072231Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:16:03.5073533Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:16:03.5074810Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:16:03.5076114Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:16:03.5133674Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:16:03.5194016Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:16:03.5195284Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:16:03.5196371Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:16:03.5197323Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:16:03.5198516Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-20T22:16:03.5199555Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-20T22:16:03.5200715Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-20T22:16:03.5202003Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-20T22:16:03.5203362Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-20T22:16:03.5204539Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-20T22:16:03.5205746Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-20T22:16:03.5207087Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-20T22:16:03.5208326Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-20T22:16:03.5209539Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-20T22:16:03.5210747Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-20T22:16:03.5213068Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-20T22:16:03.5339225Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-20T22:16:03.5340428Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-20T22:16:03.5341642Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-20T22:16:03.5342991Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-20T22:16:03.5344281Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-20T22:16:03.5345502Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-20T22:16:03.5346770Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-20T22:16:03.5348041Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-20T22:16:03.5349306Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-20T22:16:03.5350568Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-20T22:16:03.5351820Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-20T22:16:03.5372689Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-20T22:16:03.5428118Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-20T22:16:03.5429441Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:16:03.5430618Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:16:03.5431680Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-20T22:16:03.5432716Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-20T22:16:03.5433852Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-20T22:16:03.5435015Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-08-20T22:16:03.5437960Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-20T22:16:03.5438932Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-20T22:16:03.5439905Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-20T22:16:03.5543762Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-20T22:16:03.5587080Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-20T22:16:03.5587729Z creating: build/lib/ 2024-08-20T22:16:03.5676321Z inflating: build/lib/libprotobuf-lite.a 2024-08-20T22:16:03.6132314Z inflating: build/lib/libprotobuf.a 2024-08-20T22:16:03.6141601Z inflating: build/lib/libpthreadpool.a 2024-08-20T22:16:03.6149934Z inflating: build/lib/libcpuinfo.a 2024-08-20T22:16:03.6158136Z inflating: build/lib/libcpuinfo_internals.a 2024-08-20T22:16:03.6158780Z inflating: build/lib/libclog.a 2024-08-20T22:16:03.6177732Z inflating: build/lib/libnnpack.a 2024-08-20T22:16:03.6180276Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-20T22:16:03.6245590Z inflating: build/lib/libgtest.a 2024-08-20T22:16:03.6320286Z inflating: build/lib/libbenchmark.a 2024-08-20T22:16:03.6384507Z inflating: build/lib/libasmjit.a 2024-08-20T22:16:03.6392552Z inflating: build/lib/libittnotify.a 2024-08-20T22:16:03.6420224Z inflating: build/lib/libtensorpipe_uv.a 2024-08-20T22:16:03.6547975Z inflating: build/lib/libgloo.a 2024-08-20T22:16:03.6568439Z inflating: build/lib/libfmt.a 2024-08-20T22:16:03.6663716Z inflating: build/lib/libc10.so 2024-08-20T22:16:03.6664982Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-20T22:16:03.6666981Z inflating: build/lib/libtorch_global_deps.so 2024-08-20T22:16:03.7169887Z inflating: build/lib/libprotoc.a 2024-08-20T22:16:03.7188798Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-20T22:16:03.7206819Z inflating: build/lib/libgmock.a 2024-08-20T22:16:03.7207550Z inflating: build/lib/libgtest_main.a 2024-08-20T22:16:03.7208490Z inflating: build/lib/libbenchmark_main.a 2024-08-20T22:16:03.7598463Z inflating: build/lib/libgloo_cuda.a 2024-08-20T22:16:04.7761429Z inflating: build/lib/libdnnl.a 2024-08-20T22:16:04.8334053Z inflating: build/lib/libtensorpipe.a 2024-08-20T22:16:04.8391471Z inflating: build/lib/libc10_cuda.so 2024-08-20T22:16:04.8392226Z inflating: build/lib/libgmock_main.a 2024-08-20T22:16:04.9650558Z inflating: build/lib/libfbgemm.a 2024-08-20T22:16:04.9906058Z inflating: build/lib/libtensorpipe_cuda.a 2024-08-20T22:16:05.0411957Z inflating: build/lib/libkineto.a 2024-08-20T22:16:05.0595405Z inflating: build/lib/libXNNPACK.a 2024-08-20T22:16:05.0637982Z inflating: build/lib/libonnx_proto.a 2024-08-20T22:16:05.1330686Z inflating: build/lib/libonnx.a 2024-08-20T22:16:07.5777879Z inflating: build/lib/libtorch_cpu.so 2024-08-20T22:16:07.5782671Z inflating: build/lib/libunbox_lib.a 2024-08-20T22:16:07.5786931Z inflating: build/lib/libshm.so 2024-08-20T22:16:09.6427451Z inflating: build/lib/libtorch_cuda.so 2024-08-20T22:16:09.6428542Z inflating: build/lib/libtorch.so 2024-08-20T22:16:09.6431484Z inflating: build/lib/libc10d_cuda_test.so 2024-08-20T22:16:10.4614265Z inflating: build/lib/libtorch_cuda_linalg.so 2024-08-20T22:16:10.6611334Z inflating: build/lib/libtorch_python.so 2024-08-20T22:16:10.6683152Z inflating: build/lib/libtorchbind_test.so 2024-08-20T22:16:10.6704100Z inflating: build/lib/libjitbackend_test.so 2024-08-20T22:16:10.6729744Z inflating: build/lib/libbackend_with_compiler.so 2024-08-20T22:16:10.6756705Z inflating: build/lib/libaoti_custom_ops.so 2024-08-20T22:16:10.6790914Z inflating: build/lib/libnnapi_backend.so 2024-08-20T22:16:10.6791910Z creating: build/bin/ 2024-08-20T22:16:10.6841734Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-20T22:16:10.6893698Z inflating: build/bin/c10_DeviceGuard_test 2024-08-20T22:16:10.6944838Z inflating: build/bin/c10_Device_test 2024-08-20T22:16:10.7004937Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-20T22:16:10.7057820Z inflating: build/bin/c10_Scalar_test 2024-08-20T22:16:10.7107243Z inflating: build/bin/c10_StreamGuard_test 2024-08-20T22:16:10.7159033Z inflating: build/bin/c10_SymInt_test 2024-08-20T22:16:10.7215017Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-20T22:16:10.7271239Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-20T22:16:10.7328019Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-20T22:16:10.7399294Z inflating: build/bin/c10_cow_test 2024-08-20T22:16:10.7452626Z inflating: build/bin/c10_Bitset_test 2024-08-20T22:16:10.7502163Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-20T22:16:10.7552454Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-20T22:16:10.7603436Z inflating: build/bin/c10_Half_test 2024-08-20T22:16:10.7660063Z inflating: build/bin/c10_LeftRight_test 2024-08-20T22:16:10.7715720Z inflating: build/bin/c10_Metaprogramming_test 2024-08-20T22:16:10.7766156Z inflating: build/bin/c10_Synchronized_test 2024-08-20T22:16:10.7822063Z inflating: build/bin/c10_ThreadLocal_test 2024-08-20T22:16:10.7873966Z inflating: build/bin/c10_TypeIndex_test 2024-08-20T22:16:10.7925581Z inflating: build/bin/c10_TypeList_test 2024-08-20T22:16:10.7974957Z inflating: build/bin/c10_TypeTraits_test 2024-08-20T22:16:10.8027188Z inflating: build/bin/c10_accumulate_test 2024-08-20T22:16:10.8083605Z inflating: build/bin/c10_bfloat16_test 2024-08-20T22:16:10.8133837Z inflating: build/bin/c10_bit_cast_test 2024-08-20T22:16:10.8190677Z inflating: build/bin/c10_complex_math_test 2024-08-20T22:16:10.8246791Z inflating: build/bin/c10_complex_test 2024-08-20T22:16:10.8299917Z inflating: build/bin/c10_exception_test 2024-08-20T22:16:10.8350601Z inflating: build/bin/c10_flags_test 2024-08-20T22:16:10.8401203Z inflating: build/bin/c10_generic_math_test 2024-08-20T22:16:10.8566510Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-20T22:16:10.8617608Z inflating: build/bin/c10_irange_test 2024-08-20T22:16:10.8671443Z inflating: build/bin/c10_lazy_test 2024-08-20T22:16:10.8728888Z inflating: build/bin/c10_logging_test 2024-08-20T22:16:10.8804246Z inflating: build/bin/c10_optional_test 2024-08-20T22:16:10.8866936Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-20T22:16:10.8921064Z inflating: build/bin/c10_registry_test 2024-08-20T22:16:10.9073179Z inflating: build/bin/c10_small_vector_test 2024-08-20T22:16:10.9124776Z inflating: build/bin/c10_ssize_test 2024-08-20T22:16:10.9177227Z inflating: build/bin/c10_string_util_test 2024-08-20T22:16:10.9228429Z inflating: build/bin/c10_tempfile_test 2024-08-20T22:16:10.9287175Z inflating: build/bin/c10_string_view_test 2024-08-20T22:16:10.9335797Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-20T22:16:10.9392336Z inflating: build/bin/c10_typeid_test 2024-08-20T22:16:10.9840342Z inflating: build/bin/protoc-3.13.0.0 2024-08-20T22:16:11.0287026Z inflating: build/bin/protoc 2024-08-20T22:16:11.0340057Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-08-20T22:16:11.0393423Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-08-20T22:16:11.0446515Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-08-20T22:16:11.0498920Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-08-20T22:16:11.0553190Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-20T22:16:11.0605438Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-20T22:16:11.0654794Z inflating: build/bin/c10_cuda_CUDATest 2024-08-20T22:16:11.0708035Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-08-20T22:16:11.1038799Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-20T22:16:11.1384810Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-20T22:16:11.1745448Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-20T22:16:11.1799612Z inflating: build/bin/BackoffTest 2024-08-20T22:16:11.1852284Z inflating: build/bin/FileStoreTest 2024-08-20T22:16:11.1908961Z inflating: build/bin/TCPStoreTest 2024-08-20T22:16:11.1962751Z inflating: build/bin/HashStoreTest 2024-08-20T22:16:11.1976152Z inflating: build/bin/ProcessGroupMPITest 2024-08-20T22:16:11.2030497Z inflating: build/bin/test_edge_op_registration 2024-08-20T22:16:11.2034851Z inflating: build/bin/torch_shm_manager 2024-08-20T22:16:11.2037878Z inflating: build/bin/example_allreduce 2024-08-20T22:16:11.2093919Z inflating: build/bin/test_dist_autograd 2024-08-20T22:16:11.2163502Z inflating: build/bin/test_cpp_rpc 2024-08-20T22:16:11.2166203Z inflating: build/bin/parallel_benchmark 2024-08-20T22:16:11.2233951Z inflating: build/bin/test_mobile_nnc 2024-08-20T22:16:11.2242920Z inflating: build/bin/aot_model_compiler_test 2024-08-20T22:16:11.2590808Z inflating: build/bin/test_lazy 2024-08-20T22:16:11.3774935Z inflating: build/bin/test_api 2024-08-20T22:16:11.3848712Z inflating: build/bin/Dict_test 2024-08-20T22:16:11.3901414Z inflating: build/bin/Dimname_test 2024-08-20T22:16:11.3966360Z inflating: build/bin/MaybeOwned_test 2024-08-20T22:16:11.4024035Z inflating: build/bin/NamedTensor_test 2024-08-20T22:16:11.4083989Z inflating: build/bin/apply_utils_test 2024-08-20T22:16:11.4143341Z inflating: build/bin/atest 2024-08-20T22:16:11.4207298Z inflating: build/bin/basic 2024-08-20T22:16:11.4262127Z inflating: build/bin/broadcast_test 2024-08-20T22:16:11.4313301Z inflating: build/bin/cpu_allocator_test 2024-08-20T22:16:11.4371947Z inflating: build/bin/cpu_generator_test 2024-08-20T22:16:11.4425465Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-20T22:16:11.4519439Z inflating: build/bin/cpu_rng_test 2024-08-20T22:16:11.4569928Z inflating: build/bin/dispatch_key_set_test 2024-08-20T22:16:11.4620555Z inflating: build/bin/dlconvertor_test 2024-08-20T22:16:11.4679147Z inflating: build/bin/extension_backend_test 2024-08-20T22:16:11.4734779Z inflating: build/bin/half_test 2024-08-20T22:16:11.4832279Z inflating: build/bin/ivalue_test 2024-08-20T22:16:11.4881647Z inflating: build/bin/lazy_tensor_test 2024-08-20T22:16:11.4936507Z inflating: build/bin/math_kernel_test 2024-08-20T22:16:11.4991571Z inflating: build/bin/memory_format_test 2024-08-20T22:16:11.5045262Z inflating: build/bin/memory_overlapping_test 2024-08-20T22:16:11.5098703Z inflating: build/bin/mobile_memory_cleanup 2024-08-20T22:16:11.5154298Z inflating: build/bin/native_test 2024-08-20T22:16:11.5206307Z inflating: build/bin/operator_name_test 2024-08-20T22:16:11.5257330Z inflating: build/bin/operators_test 2024-08-20T22:16:11.5309384Z inflating: build/bin/packedtensoraccessor_test 2024-08-20T22:16:11.5376939Z inflating: build/bin/pow_test 2024-08-20T22:16:11.5435121Z inflating: build/bin/quantized_test 2024-08-20T22:16:11.5484961Z inflating: build/bin/reduce_ops_test 2024-08-20T22:16:11.5536373Z inflating: build/bin/reportMemoryUsage_test 2024-08-20T22:16:11.5593552Z inflating: build/bin/scalar_tensor_test 2024-08-20T22:16:11.5652277Z inflating: build/bin/scalar_test 2024-08-20T22:16:11.5704476Z inflating: build/bin/StorageUtils_test 2024-08-20T22:16:11.5757549Z inflating: build/bin/stride_properties_test 2024-08-20T22:16:11.5836522Z inflating: build/bin/tensor_iterator_test 2024-08-20T22:16:11.5891298Z inflating: build/bin/test_parallel 2024-08-20T22:16:11.5894395Z inflating: build/bin/thread_init_test 2024-08-20T22:16:11.5951347Z inflating: build/bin/type_ptr_test 2024-08-20T22:16:11.6011261Z inflating: build/bin/type_test 2024-08-20T22:16:11.6064925Z inflating: build/bin/undefined_tensor_test 2024-08-20T22:16:11.6066278Z inflating: build/bin/verify_api_visibility 2024-08-20T22:16:11.6135477Z inflating: build/bin/legacy_vmap_test 2024-08-20T22:16:11.6187760Z inflating: build/bin/weakref_test 2024-08-20T22:16:11.6239469Z inflating: build/bin/wrapdim_test 2024-08-20T22:16:11.6291746Z inflating: build/bin/xla_tensor_test 2024-08-20T22:16:11.6351961Z inflating: build/bin/IListRef_test 2024-08-20T22:16:11.6459158Z inflating: build/bin/List_test 2024-08-20T22:16:11.6526151Z inflating: build/bin/KernelFunction_test 2024-08-20T22:16:11.6648652Z inflating: build/bin/kernel_function_legacy_test 2024-08-20T22:16:11.6746870Z inflating: build/bin/kernel_function_test 2024-08-20T22:16:11.6875592Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-20T22:16:11.6979812Z inflating: build/bin/kernel_lambda_test 2024-08-20T22:16:11.7042085Z inflating: build/bin/kernel_stackbased_test 2024-08-20T22:16:11.7138399Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-20T22:16:11.7189838Z inflating: build/bin/CppSignature_test 2024-08-20T22:16:11.7239732Z inflating: build/bin/op_allowlist_test 2024-08-20T22:16:11.7295557Z inflating: build/bin/backend_fallback_test 2024-08-20T22:16:11.7599408Z inflating: build/bin/op_registration_test 2024-08-20T22:16:11.7663719Z inflating: build/bin/inline_container_test 2024-08-20T22:16:11.7716968Z inflating: build/bin/cuda_apply_test 2024-08-20T22:16:11.7769441Z inflating: build/bin/cuda_allocator_test 2024-08-20T22:16:11.7828743Z inflating: build/bin/cuda_atomic_ops_test 2024-08-20T22:16:11.7884033Z inflating: build/bin/cuda_caching_host_allocator_test 2024-08-20T22:16:11.7955385Z inflating: build/bin/cuda_complex_math_test 2024-08-20T22:16:11.8014418Z inflating: build/bin/cuda_complex_test 2024-08-20T22:16:11.8064421Z inflating: build/bin/cuda_device_test 2024-08-20T22:16:11.8122117Z inflating: build/bin/cuda_cub_test 2024-08-20T22:16:11.8173683Z inflating: build/bin/cuda_dlconvertor_test 2024-08-20T22:16:11.8238504Z inflating: build/bin/cuda_distributions_test 2024-08-20T22:16:11.8295647Z inflating: build/bin/cuda_generator_test 2024-08-20T22:16:11.8345298Z inflating: build/bin/cuda_half_test 2024-08-20T22:16:11.8397737Z inflating: build/bin/cuda_integer_divider_test 2024-08-20T22:16:11.8446569Z inflating: build/bin/cuda_optional_test 2024-08-20T22:16:11.8498889Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-08-20T22:16:11.8552380Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-08-20T22:16:11.8602823Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-08-20T22:16:11.8663050Z inflating: build/bin/cuda_stream_test 2024-08-20T22:16:11.8712647Z inflating: build/bin/cuda_cudnn_test 2024-08-20T22:16:11.8765083Z inflating: build/bin/cuda_vectorized_test 2024-08-20T22:16:11.8779631Z inflating: build/bin/tutorial_tensorexpr 2024-08-20T22:16:11.8846181Z inflating: build/bin/ProcessGroupGlooTest 2024-08-20T22:16:11.8903560Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-08-20T22:16:11.8967734Z inflating: build/bin/ProcessGroupNCCLTest 2024-08-20T22:16:11.9030779Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-08-20T22:16:11.9860270Z inflating: build/bin/test_tensorexpr 2024-08-20T22:16:12.0442759Z inflating: build/bin/test_jit 2024-08-20T22:16:12.0443468Z creating: .additional_ci_files/ 2024-08-20T22:16:12.0504228Z inflating: .additional_ci_files/test-times.json 2024-08-20T22:16:12.0744418Z inflating: .additional_ci_files/test-class-times.json 2024-08-20T22:16:12.0785007Z ##[group]Run rm artifacts.zip 2024-08-20T22:16:12.0785399Z rm artifacts.zip 2024-08-20T22:16:12.0795491Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:16:12.0796180Z env: 2024-08-20T22:16:12.0796454Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:16:12.0796908Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:16:12.0797394Z ##[endgroup] 2024-08-20T22:16:12.1942913Z ##[group]Run df -H 2024-08-20T22:16:12.1943239Z df -H 2024-08-20T22:16:12.1952032Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:16:12.1952537Z env: 2024-08-20T22:16:12.1952809Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:16:12.1953264Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:16:12.1953749Z ##[endgroup] 2024-08-20T22:16:12.2140563Z Filesystem Size Used Avail Use% Mounted on 2024-08-20T22:16:12.2141199Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-08-20T22:16:12.2141683Z tmpfs 34G 0 34G 0% /dev/shm 2024-08-20T22:16:12.2142163Z tmpfs 14G 553k 14G 1% /run 2024-08-20T22:16:12.2142613Z /dev/nvme0n1p1 161G 77G 85G 48% / 2024-08-20T22:16:12.2143288Z tmpfs 34G 955k 34G 1% /tmp 2024-08-20T22:16:12.2143783Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-08-20T22:16:12.2144296Z tmpfs 6.7G 0 6.7G 0% /run/user/0 2024-08-20T22:16:12.2205257Z Prepare all required actions 2024-08-20T22:16:12.2205858Z Getting action download info 2024-08-20T22:16:12.3323566Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-20T22:16:12.3324043Z with: 2024-08-20T22:16:12.3324294Z env: 2024-08-20T22:16:12.3324573Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:16:12.3325034Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:16:12.3325521Z ##[endgroup] 2024-08-20T22:16:12.3410111Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:16:12.3410564Z with: 2024-08-20T22:16:12.3410832Z name: td_results 2024-08-20T22:16:12.3411160Z s3-bucket: gha-artifacts 2024-08-20T22:16:12.3411517Z region: us-east-1 2024-08-20T22:16:12.3411814Z env: 2024-08-20T22:16:12.3412108Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:16:12.3412565Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:16:12.3413053Z ##[endgroup] 2024-08-20T22:16:12.8294940Z (node:182583) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:16:12.8296354Z 2024-08-20T22:16:12.8296883Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:16:12.8298192Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:16:12.8299168Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:16:12.8833031Z Found 0 objects with prefix pytorch/pytorch/10479310721/td_results/ 2024-08-20T22:16:12.8839028Z Artifact download has finished successfully 2024-08-20T22:16:12.9156349Z ##[group]Run mkdir -p .additional_ci_files 2024-08-20T22:16:12.9156840Z mkdir -p .additional_ci_files 2024-08-20T22:16:12.9157416Z mv td_results.json .additional_ci_files/td_results.json 2024-08-20T22:16:12.9168239Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:16:12.9168750Z env: 2024-08-20T22:16:12.9169025Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:16:12.9169498Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:16:12.9169992Z ##[endgroup] 2024-08-20T22:16:12.9248803Z mv: cannot stat 'td_results.json': No such file or directory 2024-08-20T22:16:12.9262490Z ##[error]Process completed with exit code 1. 2024-08-20T22:16:12.9296996Z ##[group]Run .github/scripts/parse_ref.py 2024-08-20T22:16:12.9297494Z .github/scripts/parse_ref.py 2024-08-20T22:16:12.9306350Z shell: /usr/bin/bash -e {0} 2024-08-20T22:16:12.9306703Z env: 2024-08-20T22:16:12.9306988Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:16:12.9307454Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:16:12.9307945Z ##[endgroup] 2024-08-20T22:16:12.9602503Z Prepare all required actions 2024-08-20T22:16:12.9641845Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-20T22:16:12.9642479Z with: 2024-08-20T22:16:12.9643034Z github-token: *** 2024-08-20T22:16:12.9643463Z env: 2024-08-20T22:16:12.9643741Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:16:12.9644207Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:16:12.9644705Z ##[endgroup] 2024-08-20T22:16:12.9662222Z ##[group]Run set -eux 2024-08-20T22:16:12.9662569Z set -eux 2024-08-20T22:16:12.9663163Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-20T22:16:12.9671730Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:16:12.9672229Z env: 2024-08-20T22:16:12.9672508Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:16:12.9672970Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:16:12.9673620Z GITHUB_TOKEN: *** 2024-08-20T22:16:12.9673935Z ##[endgroup] 2024-08-20T22:16:12.9702845Z + python3 .github/scripts/get_workflow_job_id.py 10479310721 i-029e43603fd06f026 2024-08-20T22:16:13.5866466Z setting job-id=29026298293 2024-08-20T22:16:13.5867544Z setting job-name=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:16:13.6094978Z Prepare all required actions 2024-08-20T22:16:13.6095606Z Getting action download info 2024-08-20T22:16:13.7209564Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-20T22:16:13.7210031Z with: 2024-08-20T22:16:13.7210489Z github-token: *** 2024-08-20T22:16:13.7217118Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:16:13.7224720Z job-name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:16:13.7225645Z env: 2024-08-20T22:16:13.7225929Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:16:13.7226389Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:16:13.7226883Z ##[endgroup] 2024-08-20T22:16:13.7270364Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:16:13.7270929Z with: 2024-08-20T22:16:13.7271197Z shell: bash 2024-08-20T22:16:13.7271499Z timeout_minutes: 10 2024-08-20T22:16:13.7271823Z max_attempts: 5 2024-08-20T22:16:13.7272142Z retry_wait_seconds: 30 2024-08-20T22:16:13.7273275Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:16:13.7274665Z polling_interval_seconds: 1 2024-08-20T22:16:13.7275046Z warning_on_retry: true 2024-08-20T22:16:13.7275399Z continue_on_error: false 2024-08-20T22:16:13.7275752Z env: 2024-08-20T22:16:13.7276031Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:16:13.7276482Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:16:13.7277137Z GITHUB_TOKEN: *** 2024-08-20T22:16:13.7277465Z ##[endgroup] 2024-08-20T22:16:13.9677300Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:16:14.6259327Z Collecting requests==2.27.1 2024-08-20T22:16:14.6581066Z Downloading requests-2.27.1-py2.py3-none-any.whl.metadata (5.0 kB) 2024-08-20T22:16:14.7089738Z Collecting pyyaml==6.0.1 2024-08-20T22:16:14.7115429Z Downloading PyYAML-6.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB) 2024-08-20T22:16:14.7604558Z Collecting urllib3<1.27,>=1.21.1 (from requests==2.27.1) 2024-08-20T22:16:14.7629429Z Downloading urllib3-1.26.19-py2.py3-none-any.whl.metadata (49 kB) 2024-08-20T22:16:14.7804353Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/miniconda/lib/python3.12/site-packages (from requests==2.27.1) (2024.7.4) 2024-08-20T22:16:14.8481670Z Collecting charset-normalizer~=2.0.0 (from requests==2.27.1) 2024-08-20T22:16:14.8511211Z Downloading charset_normalizer-2.0.12-py3-none-any.whl.metadata (11 kB) 2024-08-20T22:16:14.8588366Z Requirement already satisfied: idna<4,>=2.5 in /home/ec2-user/miniconda/lib/python3.12/site-packages (from requests==2.27.1) (3.7) 2024-08-20T22:16:14.8665077Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2024-08-20T22:16:14.8761231Z Downloading PyYAML-6.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (724 kB) 2024-08-20T22:16:14.8921794Z ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 725.0/725.0 kB 46.0 MB/s eta 0:00:00 2024-08-20T22:16:14.8945440Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-08-20T22:16:14.9014335Z Downloading urllib3-1.26.19-py2.py3-none-any.whl (143 kB) 2024-08-20T22:16:14.9787020Z Installing collected packages: urllib3, pyyaml, charset-normalizer, requests 2024-08-20T22:16:14.9787733Z Attempting uninstall: urllib3 2024-08-20T22:16:14.9808348Z Found existing installation: urllib3 2.2.2 2024-08-20T22:16:14.9871106Z Uninstalling urllib3-2.2.2: 2024-08-20T22:16:14.9880059Z Successfully uninstalled urllib3-2.2.2 2024-08-20T22:16:15.1084544Z Attempting uninstall: charset-normalizer 2024-08-20T22:16:15.1118781Z Found existing installation: charset-normalizer 3.3.2 2024-08-20T22:16:15.1150990Z Uninstalling charset-normalizer-3.3.2: 2024-08-20T22:16:15.1173572Z Successfully uninstalled charset-normalizer-3.3.2 2024-08-20T22:16:15.1442832Z Attempting uninstall: requests 2024-08-20T22:16:15.1470324Z Found existing installation: requests 2.32.3 2024-08-20T22:16:15.1495419Z Uninstalling requests-2.32.3: 2024-08-20T22:16:15.1501623Z Successfully uninstalled requests-2.32.3 2024-08-20T22:16:15.1899190Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-08-20T22:16:15.1900676Z conda 24.7.1 requires requests<3,>=2.28.0, but you have requests 2.27.1 which is incompatible. 2024-08-20T22:16:15.1901782Z Successfully installed charset-normalizer-2.0.12 pyyaml-6.0.1 requests-2.27.1 urllib3-1.26.19 2024-08-20T22:16:15.7824504Z Command completed after 1 attempt(s). 2024-08-20T22:16:15.7883263Z ##[group]Run set -x 2024-08-20T22:16:15.7883599Z set -x 2024-08-20T22:16:15.7883895Z  2024-08-20T22:16:15.7884448Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:16:15.7885128Z # in runner workspace 2024-08-20T22:16:15.7885663Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-20T22:16:15.7896266Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:16:15.7896936Z env: 2024-08-20T22:16:15.7897207Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:16:15.7897660Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:16:15.7898149Z ##[endgroup] 2024-08-20T22:16:15.7931126Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-20T22:16:15.8169284Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:16:15.8169817Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:16:15.8170332Z echo "Job name: ${JOB_NAME}" 2024-08-20T22:16:15.8170730Z  2024-08-20T22:16:15.8171275Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:16:15.8171945Z # in runner workspace 2024-08-20T22:16:15.8172522Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-20T22:16:15.8173162Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-20T22:16:15.8173618Z  --job-name "${JOB_NAME}" \ 2024-08-20T22:16:15.8180551Z  --test-matrix "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-08-20T22:16:15.8187612Z  --selected-test-configs "" \ 2024-08-20T22:16:15.8188069Z  --pr-number "${PR_NUMBER}" \ 2024-08-20T22:16:15.8188495Z  --tag "${TAG}" \ 2024-08-20T22:16:15.8188891Z  --event-name "${EVENT_NAME}" \ 2024-08-20T22:16:15.8189340Z  --schedule "${SCHEDULE}" \ 2024-08-20T22:16:15.8189771Z  --branch "${HEAD_BRANCH}" 2024-08-20T22:16:15.8199244Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:16:15.8199769Z env: 2024-08-20T22:16:15.8200046Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:16:15.8200503Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:16:15.8201189Z GITHUB_TOKEN: *** 2024-08-20T22:16:15.8202036Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:16:15.8202952Z PR_NUMBER: 2024-08-20T22:16:15.8203340Z TAG: ciflow/inductor/133712 2024-08-20T22:16:15.8203708Z EVENT_NAME: push 2024-08-20T22:16:15.8204012Z SCHEDULE: 2024-08-20T22:16:15.8204295Z HEAD_BRANCH: 2024-08-20T22:16:15.8204585Z ##[endgroup] 2024-08-20T22:16:15.8232517Z Workflow: inductor-periodic 2024-08-20T22:16:15.8233576Z Job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:16:16.0621849Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-20T22:16:16.2305139Z ##[group]Run echo "Filtered matrix:" 2024-08-20T22:16:16.2305756Z echo "Filtered matrix:" 2024-08-20T22:16:16.2312766Z echo "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-08-20T22:16:16.2319781Z  2024-08-20T22:16:16.2320066Z echo 2024-08-20T22:16:16.2320459Z echo "Is the current job unstable? False" 2024-08-20T22:16:16.2320918Z  2024-08-20T22:16:16.2321205Z echo 2024-08-20T22:16:16.2321584Z echo "Is keep-going label set? False" 2024-08-20T22:16:16.2322021Z  2024-08-20T22:16:16.2322294Z echo 2024-08-20T22:16:16.2322624Z echo "Renabled issues? " 2024-08-20T22:16:16.2331694Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:16:16.2332188Z env: 2024-08-20T22:16:16.2332467Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:16:16.2332937Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:16:16.2333422Z ##[endgroup] 2024-08-20T22:16:16.2364147Z Filtered matrix: 2024-08-20T22:16:16.2371721Z {include: [{config: dynamo_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-08-20T22:16:16.2378765Z 2024-08-20T22:16:16.2378929Z Is the current job unstable? False 2024-08-20T22:16:16.2379224Z 2024-08-20T22:16:16.2379585Z Is keep-going label set? False 2024-08-20T22:16:16.2379852Z 2024-08-20T22:16:16.2379977Z Renabled issues? 2024-08-20T22:16:16.2421429Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:16:16.2422149Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:16:16.2430420Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:16:16.2430908Z env: 2024-08-20T22:16:16.2431182Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:16:16.2431639Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:16:16.2432122Z JOB_TIMEOUT: 240 2024-08-20T22:16:16.2432423Z ##[endgroup] 2024-08-20T22:16:16.2519601Z ##[group]Run set -x 2024-08-20T22:16:16.2520012Z set -x 2024-08-20T22:16:16.2520305Z  2024-08-20T22:16:16.2520658Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-20T22:16:16.2521195Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-20T22:16:16.2521753Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-20T22:16:16.2522260Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-20T22:16:16.2522672Z else 2024-08-20T22:16:16.2523010Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:16:16.2523537Z fi 2024-08-20T22:16:16.2523809Z  2024-08-20T22:16:16.2524272Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-20T22:16:16.2525021Z # TODO: Stop building test binaries as part of the build phase 2024-08-20T22:16:16.2525698Z # Used for GPU_FLAG since that doesn't play nice 2024-08-20T22:16:16.2526290Z # shellcheck disable=SC2086,SC2090 2024-08-20T22:16:16.2526753Z container_name=$(docker run \ 2024-08-20T22:16:16.2527176Z  ${GPU_FLAG:-} \ 2024-08-20T22:16:16.2527542Z  -e BUILD_ENVIRONMENT \ 2024-08-20T22:16:16.2527942Z  -e PR_NUMBER \ 2024-08-20T22:16:16.2528295Z  -e GITHUB_ACTIONS \ 2024-08-20T22:16:16.2528687Z  -e GITHUB_REPOSITORY \ 2024-08-20T22:16:16.2529092Z  -e GITHUB_WORKFLOW \ 2024-08-20T22:16:16.2529474Z  -e GITHUB_JOB \ 2024-08-20T22:16:16.2529827Z  -e GITHUB_RUN_ID \ 2024-08-20T22:16:16.2530210Z  -e GITHUB_RUN_NUMBER \ 2024-08-20T22:16:16.2530612Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-20T22:16:16.2531007Z  -e JOB_ID \ 2024-08-20T22:16:16.2531341Z  -e JOB_NAME \ 2024-08-20T22:16:16.2531687Z  -e BASE_SHA \ 2024-08-20T22:16:16.2532030Z  -e BRANCH \ 2024-08-20T22:16:16.2532356Z  -e SHA1 \ 2024-08-20T22:16:16.2532698Z  -e AWS_DEFAULT_REGION \ 2024-08-20T22:16:16.2533101Z  -e IN_WHEEL_TEST \ 2024-08-20T22:16:16.2533482Z  -e SHARD_NUMBER \ 2024-08-20T22:16:16.2533847Z  -e TEST_CONFIG \ 2024-08-20T22:16:16.2534216Z  -e NUM_TEST_SHARDS \ 2024-08-20T22:16:16.2534606Z  -e REENABLED_ISSUES \ 2024-08-20T22:16:16.2535019Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-20T22:16:16.2535437Z  -e VERBOSE_TEST_LOGS \ 2024-08-20T22:16:16.2535838Z  -e TEST_SHOWLOCALS \ 2024-08-20T22:16:16.2536228Z  -e NO_TEST_TIMEOUT \ 2024-08-20T22:16:16.2536606Z  -e NO_TD \ 2024-08-20T22:16:16.2536941Z  -e TD_DISTRIBUTED \ 2024-08-20T22:16:16.2537318Z  -e PR_LABELS \ 2024-08-20T22:16:16.2537726Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-20T22:16:16.2538173Z  -e SCCACHE_BUCKET \ 2024-08-20T22:16:16.2538567Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-20T22:16:16.2538975Z  -e XLA_CUDA \ 2024-08-20T22:16:16.2539379Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-20T22:16:16.2540074Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-20T22:16:16.2540597Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-20T22:16:16.2541116Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-20T22:16:16.2541636Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-20T22:16:16.2542083Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-08-20T22:16:16.2542524Z  -e DASHBOARD_TAG \ 2024-08-20T22:16:16.2543004Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-20T22:16:16.2543560Z  --security-opt seccomp=unconfined \ 2024-08-20T22:16:16.2544022Z  --cap-add=SYS_PTRACE \ 2024-08-20T22:16:16.2544412Z  --ipc=host \ 2024-08-20T22:16:16.2544769Z  --shm-size="${SHM_SIZE}" \ 2024-08-20T22:16:16.2545164Z  --tty \ 2024-08-20T22:16:16.2545470Z  --detach \ 2024-08-20T22:16:16.2545950Z  --name="${container_name}" \ 2024-08-20T22:16:16.2546372Z  --user jenkins \ 2024-08-20T22:16:16.2546866Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-20T22:16:16.2547425Z  -w /var/lib/jenkins/workspace \ 2024-08-20T22:16:16.2547867Z  "${DOCKER_IMAGE}" 2024-08-20T22:16:16.2548217Z ) 2024-08-20T22:16:16.2548620Z # Propagate download.pytorch.org IP to container 2024-08-20T22:16:16.2549544Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-20T22:16:16.2550517Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-20T22:16:16.2551483Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-20T22:16:16.2560968Z shell: /usr/bin/bash -e {0} 2024-08-20T22:16:16.2561311Z env: 2024-08-20T22:16:16.2561584Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:16:16.2562046Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:16:16.2562652Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:16:16.2563193Z PR_NUMBER: 2024-08-20T22:16:16.2563517Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-20T22:16:16.2563946Z GITHUB_WORKFLOW: inductor-periodic 2024-08-20T22:16:16.2564347Z GITHUB_JOB: test 2024-08-20T22:16:16.2564654Z GITHUB_RUN_ID: 10479310721 2024-08-20T22:16:16.2565015Z GITHUB_RUN_NUMBER: 54565 2024-08-20T22:16:16.2565364Z GITHUB_RUN_ATTEMPT: 1 2024-08-20T22:16:16.2565693Z JOB_ID: 29026298293 2024-08-20T22:16:16.2566539Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:16:16.2567441Z BRANCH: 2024-08-20T22:16:16.2567780Z SHA1: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:16.2568301Z BASE_SHA: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:16.2568779Z TEST_CONFIG: dynamo_eager_huggingface 2024-08-20T22:16:16.2569192Z SHARD_NUMBER: 1 2024-08-20T22:16:16.2569503Z NUM_TEST_SHARDS: 1 2024-08-20T22:16:16.2569820Z REENABLED_ISSUES: 2024-08-20T22:16:16.2570152Z CONTINUE_THROUGH_ERROR: False 2024-08-20T22:16:16.2570538Z VERBOSE_TEST_LOGS: False 2024-08-20T22:16:16.2570892Z TEST_SHOWLOCALS: False 2024-08-20T22:16:16.2571230Z NO_TEST_TIMEOUT: False 2024-08-20T22:16:16.2571564Z NO_TD: False 2024-08-20T22:16:16.2571858Z TD_DISTRIBUTED: False 2024-08-20T22:16:16.2572278Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-20T22:16:16.2572787Z SCCACHE_S3_KEY_PREFIX: inductor-periodic 2024-08-20T22:16:16.2573210Z SHM_SIZE: 2g 2024-08-20T22:16:16.2574297Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:16:16.2575470Z XLA_CUDA: 2024-08-20T22:16:16.2575945Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:16:16.2576561Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-20T22:16:16.2577132Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-20T22:16:16.2577536Z DASHBOARD_TAG: 2024-08-20T22:16:16.2578045Z HUGGING_FACE_HUB_TOKEN: *** 2024-08-20T22:16:16.2578420Z SCRIBE_GRAPHQL_ACCESS_TOKEN: 2024-08-20T22:16:16.2578791Z ##[endgroup] 2024-08-20T22:16:16.2611277Z + [[ dynamo_eager_huggingface == \m\u\l\t\i\g\p\u ]] 2024-08-20T22:16:16.2612114Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-08-20T22:16:16.2612640Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:16:16.2620292Z +++ nproc --ignore=2 2024-08-20T22:16:16.2643333Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10479310721 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:16:20.3371511Z + container_name=c5602275dd27032945979a763aa9348ea9f7fd583e459e7a824e50356e8b3793 2024-08-20T22:16:20.3373720Z + grep download.pytorch.org /etc/hosts 2024-08-20T22:16:20.3376234Z + docker exec -i c5602275dd27032945979a763aa9348ea9f7fd583e459e7a824e50356e8b3793 sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-20T22:16:20.4903437Z + echo DOCKER_CONTAINER_ID=c5602275dd27032945979a763aa9348ea9f7fd583e459e7a824e50356e8b3793 2024-08-20T22:16:20.4907661Z ++ echo dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:16:20.4910448Z + docker exec -t c5602275dd27032945979a763aa9348ea9f7fd583e459e7a824e50356e8b3793 sh -c 'pip install dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-20T22:16:20.8926716Z Processing ./dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+git40ec5f6) 2024-08-20T22:16:21.2144202Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.13.1) 2024-08-20T22:16:21.2147055Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (4.12.2) 2024-08-20T22:16:21.2149628Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.8.8) 2024-08-20T22:16:21.2152832Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.1.4) 2024-08-20T22:16:21.2155545Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2024.6.1) 2024-08-20T22:16:21.2160648Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.13.1) 2024-08-20T22:16:21.2177501Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.3.0) 2024-08-20T22:16:21.2190440Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.3.0) 2024-08-20T22:16:21.2207701Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.22.4) 2024-08-20T22:16:21.2592864Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.1.5) 2024-08-20T22:16:21.8330518Z Installing collected packages: torch 2024-08-20T22:16:32.0511921Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-08-20T22:16:32.0513714Z timm 1.0.9.dev0 requires torchvision, which is not installed. 2024-08-20T22:16:32.0514468Z Successfully installed torch-2.5.0a0+git40ec5f6 2024-08-20T22:16:32.1431356Z + export TERM=vt100 2024-08-20T22:16:32.1431810Z + TERM=vt100 2024-08-20T22:16:32.1435200Z ++ dirname .ci/pytorch/test.sh 2024-08-20T22:16:32.1603703Z + source .ci/pytorch/common.sh 2024-08-20T22:16:32.1608268Z +++ dirname .ci/pytorch/common.sh 2024-08-20T22:16:32.1617366Z ++ source .ci/pytorch/common_utils.sh 2024-08-20T22:16:32.1619252Z +++ declare -f -t trap_add 2024-08-20T22:16:32.1624408Z ++ set -ex 2024-08-20T22:16:32.1624892Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:16:32.1625396Z ++ BUILD_TEST_LIBTORCH=0 2024-08-20T22:16:32.1626301Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-08-20T22:16:32.1629723Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-20T22:16:32.1842391Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-20T22:16:32.1842850Z + trap_add cleanup_workspace EXIT 2024-08-20T22:16:32.1843384Z + trap_add_cmd=cleanup_workspace 2024-08-20T22:16:32.1843774Z + shift 2024-08-20T22:16:32.1844084Z + for trap_add_name in "$@" 2024-08-20T22:16:32.1849667Z +++ trap -p EXIT 2024-08-20T22:16:32.1853185Z ++ eval 'extract_trap_cmd ' 2024-08-20T22:16:32.1853559Z +++ extract_trap_cmd 2024-08-20T22:16:32.1853966Z +++ printf '%s\n' '' 2024-08-20T22:16:32.1854530Z ++ printf '%s\n' cleanup_workspace 2024-08-20T22:16:32.1857703Z + trap -- ' 2024-08-20T22:16:32.1858123Z cleanup_workspace' EXIT 2024-08-20T22:16:32.1858609Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-20T22:16:32.9490806Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-20T22:16:32.9513533Z + echo 'Environment variables:' 2024-08-20T22:16:32.9513964Z Environment variables: 2024-08-20T22:16:32.9514306Z + env 2024-08-20T22:16:32.9727692Z INSTALLED_DB=yes 2024-08-20T22:16:32.9728264Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:16:32.9728804Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:16:32.9729378Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:16:32.9730168Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:16:32.9730761Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:16:32.9731257Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:16:32.9731778Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:16:32.9732350Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:16:32.9732884Z HOSTNAME=c5602275dd27 2024-08-20T22:16:32.9734025Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_4ea9c181-ecf7-422f-94d7-5ff215691e8e 2024-08-20T22:16:32.9734841Z GITHUB_ACTION=__self 2024-08-20T22:16:32.9735198Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:16:32.9739953Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:16:32.9744210Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:16:32.9744713Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:16:32.9745062Z GITHUB_RUN_NUMBER=54565 2024-08-20T22:16:32.9745421Z TEST_CONFIG=dynamo_eager_huggingface 2024-08-20T22:16:32.9745845Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:16:32.9746319Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:16:32.9746779Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:16:32.9747259Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:16:32.9747830Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:16:32.9748237Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:16:32.9748685Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:16:32.9749183Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:16:32.9749646Z GITHUB_REF_TYPE=tag 2024-08-20T22:16:32.9749972Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:16:32.9750367Z NCCL_VERSION=2.17.1-1 2024-08-20T22:16:32.9750780Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:32.9751224Z XLA_CUDA= 2024-08-20T22:16:32.9751729Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:16:32.9753999Z *** 2024-08-20T22:16:32.9754298Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:16:32.9754700Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:16:32.9755065Z GITHUB_ACTIONS=true 2024-08-20T22:16:32.9755394Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:16:32.9755916Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:16:32.9756460Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:16:32.9756937Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:32.9757457Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:16:32.9757941Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:32.9758886Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:16:32.9759648Z UCC_HOME=/usr 2024-08-20T22:16:32.9760356Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:16:32.9760758Z VERBOSE_TEST_LOGS=False 2024-08-20T22:16:32.9761098Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:16:32.9761555Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:16:32.9762058Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:16:32.9762528Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:16:32.9762902Z SHARD_NUMBER=1 2024-08-20T22:16:32.9763295Z GITHUB_REF_PROTECTED=false 2024-08-20T22:16:32.9763644Z HOME=/var/lib/jenkins 2024-08-20T22:16:32.9764006Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:16:32.9764443Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:16:32.9764923Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:16:32.9765478Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-08-20T22:16:32.9765893Z CUDA_VERSION=12.1.1 2024-08-20T22:16:32.9766311Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:16:32.9766751Z NUM_TEST_SHARDS=1 2024-08-20T22:16:32.9767047Z UCX_HOME=/usr 2024-08-20T22:16:32.9767573Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:16:32.9768665Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_4ea9c181-ecf7-422f-94d7-5ff215691e8e 2024-08-20T22:16:32.9770131Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:16:32.9771565Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_4ea9c181-ecf7-422f-94d7-5ff215691e8e 2024-08-20T22:16:32.9772704Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:16:32.9773531Z GITHUB_EVENT_NAME=push 2024-08-20T22:16:32.9773857Z DASHBOARD_TAG= 2024-08-20T22:16:32.9774161Z GITHUB_RUN_ID=10479310721 2024-08-20T22:16:32.9774626Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:16:32.9775162Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:16:32.9776176Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_4ea9c181-ecf7-422f-94d7-5ff215691e8e 2024-08-20T22:16:32.9777095Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:16:32.9777504Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:16:32.9777872Z PR_NUMBER= 2024-08-20T22:16:32.9778152Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:16:32.9778490Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:16:32.9778930Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:16:32.9779382Z TERM=vt100 2024-08-20T22:16:32.9779719Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:16:32.9780140Z INSTALLED_VISION=yes 2024-08-20T22:16:32.9780589Z BRANCH= 2024-08-20T22:16:32.9780881Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:16:32.9781282Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:16:32.9781708Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:16:32.9782470Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:16:32.9783222Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:16:32.9783704Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:16:32.9784168Z REENABLED_ISSUES= 2024-08-20T22:16:32.9784460Z SHLVL=1 2024-08-20T22:16:32.9784720Z MAX_JOBS=14 2024-08-20T22:16:32.9785031Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:16:32.9785373Z NVARCH=x86_64 2024-08-20T22:16:32.9785671Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:16:32.9786126Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:32.9786664Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:16:32.9787151Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:16:32.9787829Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:16:32.9788404Z GITHUB_JOB=test 2024-08-20T22:16:32.9788804Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:16:32.9789356Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:16:32.9789865Z NO_TEST_TIMEOUT=False 2024-08-20T22:16:32.9790190Z TD_DISTRIBUTED=False 2024-08-20T22:16:32.9790611Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:16:32.9791051Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:16:32.9791468Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:16:32.9791828Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:16:32.9792167Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:16:32.9792514Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:16:32.9793553Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:16:32.9794618Z GITHUB_BASE_REF= 2024-08-20T22:16:32.9794947Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:16:32.9795316Z CI=true 2024-08-20T22:16:32.9795632Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:16:32.9796044Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:16:32.9796410Z JOB_ID=29026298293 2024-08-20T22:16:32.9796718Z INSTALLED_PROTOBUF=yes 2024-08-20T22:16:32.9797044Z GITHUB_HEAD_REF= 2024-08-20T22:16:32.9797343Z GITHUB_ACTION_REF= 2024-08-20T22:16:32.9797783Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:16:32.9798247Z TEST_SHOWLOCALS=False 2024-08-20T22:16:32.9798639Z GITHUB_WORKFLOW=inductor-periodic 2024-08-20T22:16:32.9799038Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:16:32.9799958Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_4ea9c181-ecf7-422f-94d7-5ff215691e8e 2024-08-20T22:16:32.9800775Z NO_TD=False 2024-08-20T22:16:32.9801075Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:16:32.9801431Z _=/usr/bin/env 2024-08-20T22:16:32.9801913Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-20T22:16:32.9959477Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-08-20T22:16:32.9961078Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T22:16:32.9962118Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-08-20T22:16:32.9963218Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-08-20T22:16:32.9963837Z + BUILD_DIR=build 2024-08-20T22:16:32.9964169Z + BUILD_RENAMED_DIR=build_renamed 2024-08-20T22:16:32.9964565Z + BUILD_BIN_DIR=build/bin 2024-08-20T22:16:32.9964906Z + SHARD_NUMBER=1 2024-08-20T22:16:32.9965219Z + NUM_TEST_SHARDS=1 2024-08-20T22:16:32.9965537Z + export VALGRIND=ON 2024-08-20T22:16:32.9965852Z + VALGRIND=ON 2024-08-20T22:16:32.9966324Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-08-20T22:16:32.9966818Z + [[ 0 == \1 ]] 2024-08-20T22:16:32.9967109Z + [[ False == \1 ]] 2024-08-20T22:16:32.9967889Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-08-20T22:16:32.9968614Z ++ realpath build/custom_test_artifacts 2024-08-20T22:16:33.0205988Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-20T22:16:33.0206929Z + [[ -n '' ]] 2024-08-20T22:16:33.0207335Z + echo 'Environment variables' 2024-08-20T22:16:33.0207734Z Environment variables 2024-08-20T22:16:33.0208063Z + env 2024-08-20T22:16:33.0216133Z INSTALLED_DB=yes 2024-08-20T22:16:33.0216670Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:16:33.0217211Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:16:33.0217778Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:16:33.0218580Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:16:33.0219319Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:16:33.0219978Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:16:33.0220701Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:16:33.0221506Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:16:33.0222178Z HOSTNAME=c5602275dd27 2024-08-20T22:16:33.0223258Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_4ea9c181-ecf7-422f-94d7-5ff215691e8e 2024-08-20T22:16:33.0224247Z GITHUB_ACTION=__self 2024-08-20T22:16:33.0224598Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:16:33.0228417Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:16:33.0232360Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:16:33.0232881Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:16:33.0233233Z GITHUB_RUN_NUMBER=54565 2024-08-20T22:16:33.0233599Z TEST_CONFIG=dynamo_eager_huggingface 2024-08-20T22:16:33.0234022Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:16:33.0234506Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:16:33.0234986Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:16:33.0235435Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:16:33.0235843Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:16:33.0236241Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:16:33.0236692Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:16:33.0237190Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:16:33.0237650Z GITHUB_REF_TYPE=tag 2024-08-20T22:16:33.0237975Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:16:33.0238358Z NCCL_VERSION=2.17.1-1 2024-08-20T22:16:33.0238752Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:33.0239392Z XLA_CUDA= 2024-08-20T22:16:33.0239851Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:16:33.0240247Z *** 2024-08-20T22:16:33.0240529Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:16:33.0240923Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:16:33.0241290Z GITHUB_ACTIONS=true 2024-08-20T22:16:33.0241624Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:16:33.0242134Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:16:33.0242687Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:16:33.0243290Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:33.0243817Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:16:33.0244305Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:33.0245276Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:16:33.0246069Z UCC_HOME=/usr 2024-08-20T22:16:33.0246561Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:16:33.0246982Z VERBOSE_TEST_LOGS=False 2024-08-20T22:16:33.0247337Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:16:33.0247812Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:16:33.0248319Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:16:33.0248786Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:16:33.0249164Z SHARD_NUMBER=1 2024-08-20T22:16:33.0249479Z GITHUB_REF_PROTECTED=false 2024-08-20T22:16:33.0249828Z HOME=/var/lib/jenkins 2024-08-20T22:16:33.0250197Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:16:33.0250639Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:16:33.0251118Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:16:33.0251671Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-08-20T22:16:33.0252088Z CUDA_VERSION=12.1.1 2024-08-20T22:16:33.0252518Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:16:33.0252962Z NUM_TEST_SHARDS=1 2024-08-20T22:16:33.0253259Z UCX_HOME=/usr 2024-08-20T22:16:33.0253794Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:16:33.0254906Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_4ea9c181-ecf7-422f-94d7-5ff215691e8e 2024-08-20T22:16:33.0256412Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:16:33.0257848Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_4ea9c181-ecf7-422f-94d7-5ff215691e8e 2024-08-20T22:16:33.0258973Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:16:33.0259628Z GITHUB_EVENT_NAME=push 2024-08-20T22:16:33.0260289Z DASHBOARD_TAG= 2024-08-20T22:16:33.0260593Z GITHUB_RUN_ID=10479310721 2024-08-20T22:16:33.0261064Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:16:33.0261611Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:16:33.0262629Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_4ea9c181-ecf7-422f-94d7-5ff215691e8e 2024-08-20T22:16:33.0263561Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:16:33.0263969Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:16:33.0264344Z PR_NUMBER= 2024-08-20T22:16:33.0264624Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:16:33.0264939Z VALGRIND=ON 2024-08-20T22:16:33.0265230Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:16:33.0265674Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:16:33.0266131Z TERM=vt100 2024-08-20T22:16:33.0266465Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:16:33.0266882Z INSTALLED_VISION=yes 2024-08-20T22:16:33.0267194Z BRANCH= 2024-08-20T22:16:33.0267477Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:16:33.0267875Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:16:33.0268300Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:16:33.0269067Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:16:33.0269834Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:16:33.0270467Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:16:33.0270936Z REENABLED_ISSUES= 2024-08-20T22:16:33.0271233Z SHLVL=1 2024-08-20T22:16:33.0271485Z MAX_JOBS=14 2024-08-20T22:16:33.0271804Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:16:33.0272154Z NVARCH=x86_64 2024-08-20T22:16:33.0272451Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:16:33.0272904Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:33.0273444Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:16:33.0273939Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:16:33.0274608Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:16:33.0275174Z GITHUB_JOB=test 2024-08-20T22:16:33.0275578Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:16:33.0276132Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:16:33.0276644Z NO_TEST_TIMEOUT=False 2024-08-20T22:16:33.0277103Z TD_DISTRIBUTED=False 2024-08-20T22:16:33.0277498Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:16:33.0277952Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:16:33.0278375Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:16:33.0278748Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:16:33.0279105Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:16:33.0279463Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:16:33.0280517Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:16:33.0281600Z GITHUB_BASE_REF= 2024-08-20T22:16:33.0281942Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:16:33.0282318Z CI=true 2024-08-20T22:16:33.0282633Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:16:33.0283121Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:16:33.0283523Z JOB_ID=29026298293 2024-08-20T22:16:33.0283838Z INSTALLED_PROTOBUF=yes 2024-08-20T22:16:33.0284177Z GITHUB_HEAD_REF= 2024-08-20T22:16:33.0284495Z GITHUB_ACTION_REF= 2024-08-20T22:16:33.0284944Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:16:33.0285421Z TEST_SHOWLOCALS=False 2024-08-20T22:16:33.0285828Z GITHUB_WORKFLOW=inductor-periodic 2024-08-20T22:16:33.0286245Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:16:33.0287174Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_4ea9c181-ecf7-422f-94d7-5ff215691e8e 2024-08-20T22:16:33.0287993Z NO_TD=False 2024-08-20T22:16:33.0288301Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:16:33.0288670Z _=/usr/bin/env 2024-08-20T22:16:33.0289013Z + echo 'Testing pytorch' 2024-08-20T22:16:33.0289362Z Testing pytorch 2024-08-20T22:16:33.0289693Z + export LANG=C.UTF-8 2024-08-20T22:16:33.0290039Z + LANG=C.UTF-8 2024-08-20T22:16:33.1302333Z + PR_NUMBER= 2024-08-20T22:16:33.1302853Z + [[ dynamo_eager_huggingface == \d\e\f\a\u\l\t ]] 2024-08-20T22:16:33.1303521Z + [[ dynamo_eager_huggingface == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-20T22:16:33.1304154Z + [[ dynamo_eager_huggingface == \s\l\o\w ]] 2024-08-20T22:16:33.1304874Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-08-20T22:16:33.1305607Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-08-20T22:16:33.1306169Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:16:33.1306654Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:16:33.1307122Z + [[ dynamo_eager_huggingface == *crossref* ]] 2024-08-20T22:16:33.1307730Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:16:33.1308379Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-08-20T22:16:33.1309043Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:16:33.1309620Z + pip_install --user ninja==1.10.2 2024-08-20T22:16:33.1310170Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-20T22:16:33.9336915Z Collecting ninja==1.10.2 2024-08-20T22:16:33.9502520Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-20T22:16:33.9609288Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-20T22:16:34.5200129Z Installing collected packages: ninja 2024-08-20T22:16:34.5278749Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-20T22:16:34.5280042Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-20T22:16:34.5348423Z Successfully installed ninja-1.10.2 2024-08-20T22:16:34.6111688Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:16:34.6113826Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:16:34.6115814Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:16:34.6116347Z + install_tlparse 2024-08-20T22:16:34.6116735Z + pip_install --user tlparse==0.3.25 2024-08-20T22:16:34.6117295Z + pip install --progress-bar off --user tlparse==0.3.25 2024-08-20T22:16:34.9933612Z Collecting tlparse==0.3.25 2024-08-20T22:16:35.0099987Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-08-20T22:16:35.0367699Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-20T22:16:35.6337398Z Installing collected packages: tlparse 2024-08-20T22:16:35.6687813Z Successfully installed tlparse-0.3.25 2024-08-20T22:16:35.7460807Z ++ python -m site --user-base 2024-08-20T22:16:35.7705616Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:16:35.7707674Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-08-20T22:16:35.7708363Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-08-20T22:16:35.7709137Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:16:35.7710167Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-08-20T22:16:35.7711600Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-08-20T22:16:35.7712371Z + cd test 2024-08-20T22:16:35.7712911Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-20T22:16:38.1146157Z + [[ dynamo_eager_huggingface == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-20T22:16:38.1146959Z + [[ dynamo_eager_huggingface == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-20T22:16:38.1150608Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-20T22:16:38.1151280Z + [[ dynamo_eager_huggingface == *pr_time_benchmarks* ]] 2024-08-20T22:16:38.1152002Z + [[ dynamo_eager_huggingface == *dynamo_eager* ]] 2024-08-20T22:16:38.1152652Z + DYNAMO_BENCHMARK_FLAGS+=(--backend eager) 2024-08-20T22:16:38.1153160Z + [[ dynamo_eager_huggingface == *dynamic* ]] 2024-08-20T22:16:38.1153645Z + [[ dynamo_eager_huggingface == *cpu* ]] 2024-08-20T22:16:38.1154157Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-20T22:16:38.1184936Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:16:38.1186533Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-08-20T22:16:38.1187737Z + cd test 2024-08-20T22:16:38.1188877Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-20T22:16:39.5702203Z PyTorch built with: 2024-08-20T22:16:39.5702760Z - GCC 9.4 2024-08-20T22:16:39.5703208Z - C++ Version: 201703 2024-08-20T22:16:39.5704092Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:16:39.5705229Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:16:39.5705995Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:16:39.5706790Z - LAPACK is enabled (usually provided by MKL) 2024-08-20T22:16:39.5707281Z - NNPACK is enabled 2024-08-20T22:16:39.5707678Z - CPU capability usage: AVX2 2024-08-20T22:16:39.5708102Z - CUDA Runtime 12.1 2024-08-20T22:16:39.5708641Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-08-20T22:16:39.5709267Z - CuDNN 90.1 (built against CUDA 12.4) 2024-08-20T22:16:39.5709729Z - Magma 2.6.1 2024-08-20T22:16:39.5716725Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-20T22:16:39.5723215Z 2024-08-20T22:16:39.8906561Z + cd test 2024-08-20T22:16:39.8907345Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-20T22:16:41.2275759Z ATen/Parallel: 2024-08-20T22:16:41.2276235Z at::get_num_threads() : 8 2024-08-20T22:16:41.2276757Z at::get_num_interop_threads() : 16 2024-08-20T22:16:41.2277308Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:16:41.2277721Z omp_get_max_threads() : 8 2024-08-20T22:16:41.2278647Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:16:41.2279455Z mkl_get_max_threads() : 8 2024-08-20T22:16:41.2280093Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:16:41.2280738Z std::thread::hardware_concurrency() : 16 2024-08-20T22:16:41.2281173Z Environment variables: 2024-08-20T22:16:41.2281520Z OMP_NUM_THREADS : [not set] 2024-08-20T22:16:41.2281894Z MKL_NUM_THREADS : [not set] 2024-08-20T22:16:41.2282276Z ATen parallel backend: OpenMP 2024-08-20T22:16:41.2282531Z 2024-08-20T22:16:41.5030436Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:16:41.5031196Z + [[ dynamo_eager_huggingface == *backward* ]] 2024-08-20T22:16:41.5031776Z + [[ dynamo_eager_huggingface == *xla* ]] 2024-08-20T22:16:41.5032258Z + [[ dynamo_eager_huggingface == *executorch* ]] 2024-08-20T22:16:41.5032793Z + [[ dynamo_eager_huggingface == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-20T22:16:41.5033455Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:16:41.5034013Z + [[ dynamo_eager_huggingface == distributed ]] 2024-08-20T22:16:41.5034581Z + [[ dynamo_eager_huggingface == *inductor_distributed* ]] 2024-08-20T22:16:41.5035370Z + [[ dynamo_eager_huggingface == *inductor-halide* ]] 2024-08-20T22:16:41.5036015Z + [[ dynamo_eager_huggingface == *inductor-micro-benchmark* ]] 2024-08-20T22:16:41.5036588Z + [[ dynamo_eager_huggingface == *huggingface* ]] 2024-08-20T22:16:41.5037043Z + install_torchvision 2024-08-20T22:16:41.5037378Z + local orig_preload 2024-08-20T22:16:41.5037692Z + local commit 2024-08-20T22:16:41.5038011Z ++ get_pinned_commit vision 2024-08-20T22:16:41.5038406Z ++ cat .github/ci_commit_pins/vision.txt 2024-08-20T22:16:41.5060234Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:16:41.5060814Z + orig_preload= 2024-08-20T22:16:41.5061560Z + '[' -n '' ']' 2024-08-20T22:16:41.5062391Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:16:41.5063816Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:16:41.8420245Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:16:41.8424915Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-51x57e_a 2024-08-20T22:16:41.8455658Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-51x57e_a 2024-08-20T22:16:43.3804502Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-08-20T22:16:43.3834055Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:16:44.7051026Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:16:44.9979106Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:16:47.4948646Z Preparing metadata (setup.py) ... [?25l- \ done 2024-08-20T22:16:47.4981002Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4) 2024-08-20T22:16:47.4985335Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+git40ec5f6) 2024-08-20T22:16:47.4989433Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-08-20T22:16:47.5059453Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-08-20T22:16:47.5064828Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-08-20T22:16:47.5067598Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-08-20T22:16:47.5071068Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-08-20T22:16:47.5074292Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.6.1) 2024-08-20T22:16:47.5079330Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.1) 2024-08-20T22:16:47.5093380Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-08-20T22:16:47.5566594Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-08-20T22:16:47.5629823Z Building wheels for collected packages: torchvision 2024-08-20T22:18:04.8922411Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-08-20T22:18:04.8955461Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2116077 sha256=6d97ce1ea3473602960dc4ff86b3f2c9958122330edbd9930707498ed350b6ff 2024-08-20T22:18:04.8957372Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-08-20T22:18:04.8993154Z Successfully built torchvision 2024-08-20T22:18:05.3881121Z Installing collected packages: torchvision 2024-08-20T22:18:05.8060375Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-08-20T22:18:05.9570250Z + '[' -n '' ']' 2024-08-20T22:18:05.9570706Z + id=0 2024-08-20T22:18:05.9571026Z + test_dynamo_benchmark huggingface 0 2024-08-20T22:18:05.9573292Z ++ pwd 2024-08-20T22:18:05.9577943Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:18:05.9578602Z + local suite=huggingface 2024-08-20T22:18:05.9578938Z + shift 2024-08-20T22:18:05.9579208Z + local shard_id=0 2024-08-20T22:18:05.9579510Z + shift 2024-08-20T22:18:05.9580693Z + [[ dynamo_eager_huggingface == *perf_compare* ]] 2024-08-20T22:18:05.9581328Z + [[ dynamo_eager_huggingface == *perf* ]] 2024-08-20T22:18:05.9581903Z + [[ dynamo_eager_huggingface == *cpu* ]] 2024-08-20T22:18:05.9582756Z + [[ dynamo_eager_huggingface == *aot_inductor* ]] 2024-08-20T22:18:05.9590283Z + test_single_dynamo_benchmark inference huggingface 0 --inference --bfloat16 2024-08-20T22:18:05.9590920Z ++ pwd 2024-08-20T22:18:05.9591403Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:18:05.9592081Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:18:05.9633068Z + local name=inference 2024-08-20T22:18:05.9633409Z + shift 2024-08-20T22:18:05.9633692Z + local suite=huggingface 2024-08-20T22:18:05.9634113Z + shift 2024-08-20T22:18:05.9634472Z + local shard_id=0 2024-08-20T22:18:05.9634852Z + shift 2024-08-20T22:18:05.9635182Z + partition_flags=() 2024-08-20T22:18:05.9635573Z + local partition_flags 2024-08-20T22:18:05.9635965Z + [[ -n 1 ]] 2024-08-20T22:18:05.9636286Z + [[ -n 0 ]] 2024-08-20T22:18:05.9636934Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-08-20T22:18:05.9637661Z + [[ dynamo_eager_huggingface == *perf_compare* ]] 2024-08-20T22:18:05.9638177Z + [[ dynamo_eager_huggingface == *perf* ]] 2024-08-20T22:18:05.9638671Z + [[ dynamo_eager_huggingface == *aot_inductor* ]] 2024-08-20T22:18:05.9639157Z + [[ dynamo_eager_huggingface == *_avx2* ]] 2024-08-20T22:18:05.9639635Z + [[ dynamo_eager_huggingface == *_avx512* ]] 2024-08-20T22:18:05.9641322Z + python benchmarks/dynamo/huggingface.py --ci --accuracy --timing --explain --backend eager --device cuda --inference --bfloat16 --total-partitions 1 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv 2024-08-20T22:18:10.2246722Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:18:10.2248436Z warnings.warn( 2024-08-20T22:18:10.2716323Z 2024-08-20T22:18:10.2716762Z config.json: 0% 0.00/694 [00:00 will be ignored 2024-08-20T22:31:42.8453656Z pass 2024-08-20T22:31:42.9289897Z TIMING: entire_frame_compile:3.53729 backend_compile:0.00045 total_wall_time:3.53729 2024-08-20T22:31:42.9290876Z STATS: call_* op count: 533 | FakeTensorMode.__torch_dispatch__:6275 | FakeTensor.__torch_dispatch__:462 2024-08-20T22:31:42.9293043Z Dynamo produced 2 graphs covering 533 ops with 5 graph breaks (4 unique) 2024-08-20T22:31:46.6932522Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:31:46.6934262Z warnings.warn( 2024-08-20T22:31:46.8639012Z 2024-08-20T22:31:49.8163270Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:31:49.8164159Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:31:49.8165076Z cuda train AllenaiLongformerBase 2024-08-20T22:31:57.5058917Z W0820 22:31:57.505000 3741 torch/_dynamo/variables/tensor.py:776] [3/0] Graph break from `Tensor.item()`, consider setting: 2024-08-20T22:31:57.5060402Z W0820 22:31:57.505000 3741 torch/_dynamo/variables/tensor.py:776] [3/0] torch._dynamo.config.capture_scalar_outputs = True 2024-08-20T22:31:57.5062084Z W0820 22:31:57.505000 3741 torch/_dynamo/variables/tensor.py:776] [3/0] or: 2024-08-20T22:31:57.5063069Z W0820 22:31:57.505000 3741 torch/_dynamo/variables/tensor.py:776] [3/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-08-20T22:31:57.5064456Z W0820 22:31:57.505000 3741 torch/_dynamo/variables/tensor.py:776] [3/0] to include these operations in the captured graph. 2024-08-20T22:31:57.5065563Z W0820 22:31:57.505000 3741 torch/_dynamo/variables/tensor.py:776] [3/0] 2024-08-20T22:31:57.5066522Z W0820 22:31:57.505000 3741 torch/_dynamo/variables/tensor.py:776] [3/0] Graph break: from user code at: 2024-08-20T22:31:57.5068841Z W0820 22:31:57.505000 3741 torch/_dynamo/variables/tensor.py:776] [3/0] File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T22:31:57.5070461Z W0820 22:31:57.505000 3741 torch/_dynamo/variables/tensor.py:776] [3/0] pred = mod(**cloned_inputs) 2024-08-20T22:31:57.5072438Z W0820 22:31:57.505000 3741 torch/_dynamo/variables/tensor.py:776] [3/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1835, in forward 2024-08-20T22:31:57.5074025Z W0820 22:31:57.505000 3741 torch/_dynamo/variables/tensor.py:776] [3/0] outputs = self.longformer( 2024-08-20T22:31:57.5075779Z W0820 22:31:57.505000 3741 torch/_dynamo/variables/tensor.py:776] [3/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1738, in forward 2024-08-20T22:31:57.5077395Z W0820 22:31:57.505000 3741 torch/_dynamo/variables/tensor.py:776] [3/0] encoder_outputs = self.encoder( 2024-08-20T22:31:57.5079171Z W0820 22:31:57.505000 3741 torch/_dynamo/variables/tensor.py:776] [3/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1291, in forward 2024-08-20T22:31:57.5080892Z W0820 22:31:57.505000 3741 torch/_dynamo/variables/tensor.py:776] [3/0] is_global_attn = is_index_global_attn.flatten().any().item() 2024-08-20T22:31:57.5081909Z W0820 22:31:57.505000 3741 torch/_dynamo/variables/tensor.py:776] [3/0] 2024-08-20T22:31:57.5082644Z W0820 22:31:57.505000 3741 torch/_dynamo/variables/tensor.py:776] [3/0] 2024-08-20T22:32:08.6507469Z W0820 22:32:08.649000 3741 torch/_logging/_internal.py:1058] [16/0] Profiler function will be ignored 2024-08-20T22:32:17.5747431Z pass 2024-08-20T22:32:17.6443843Z TIMING: entire_frame_compile:21.08991 backend_compile:0.00291 total_wall_time:21.08991 2024-08-20T22:32:17.6445643Z STATS: call_* op count: 2366 | FakeTensorMode.__torch_dispatch__:8708 | FakeTensor.__torch_dispatch__:3728 2024-08-20T22:32:17.6451831Z Dynamo produced 7 graphs covering 2366 ops with 9 graph breaks (5 unique) 2024-08-20T22:32:21.8456423Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:32:21.8458151Z warnings.warn( 2024-08-20T22:32:22.0220606Z 2024-08-20T22:32:26.0345002Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:32:26.0346035Z loading model: 0it [00:04, ?it/s] 2024-08-20T22:32:26.0347057Z cuda train BartForCausalLM 2024-08-20T22:32:34.4603129Z pass 2024-08-20T22:32:34.5061996Z TIMING: entire_frame_compile:0.78169 backend_compile:0.0003 total_wall_time:0.78169 2024-08-20T22:32:34.5062954Z STATS: call_* op count: 39 | FakeTensorMode.__torch_dispatch__:554 | FakeTensor.__torch_dispatch__:47 2024-08-20T22:32:34.5063853Z Dynamo produced 5 graphs covering 39 ops with 6 graph breaks (5 unique) 2024-08-20T22:32:38.2755632Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:32:38.2757741Z warnings.warn( 2024-08-20T22:32:38.4612188Z 2024-08-20T22:32:45.5215194Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:32:45.5215715Z loading model: 0it [00:07, ?it/s] 2024-08-20T22:32:45.5217654Z cuda train BartForConditionalGeneration 2024-08-20T22:33:03.2469765Z pass 2024-08-20T22:33:03.5503550Z TIMING: entire_frame_compile:1.6177 backend_compile:0.00047 total_wall_time:1.6177 2024-08-20T22:33:03.5505487Z STATS: call_* op count: 93 | FakeTensorMode.__torch_dispatch__:1154 | FakeTensor.__torch_dispatch__:121 2024-08-20T22:33:03.5507290Z Dynamo produced 7 graphs covering 93 ops with 8 graph breaks (5 unique) 2024-08-20T22:33:07.3378696Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:33:07.3380577Z warnings.warn( 2024-08-20T22:33:07.5449701Z 2024-08-20T22:33:09.6356316Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:33:09.6356802Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:33:09.6357252Z cuda train BertForMaskedLM 2024-08-20T22:33:17.8931549Z W0820 22:33:17.892000 3870 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:33:26.1066136Z pass 2024-08-20T22:33:26.1958589Z TIMING: entire_frame_compile:13.63352 backend_compile:0.00245 total_wall_time:13.63352 2024-08-20T22:33:26.1959589Z STATS: call_* op count: 995 | FakeTensorMode.__torch_dispatch__:7168 | FakeTensor.__torch_dispatch__:3647 2024-08-20T22:33:26.1960731Z Dynamo produced 2 graphs covering 995 ops with 5 graph breaks (4 unique) 2024-08-20T22:33:30.2027467Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:33:30.2029283Z warnings.warn( 2024-08-20T22:33:30.3900329Z 2024-08-20T22:33:32.2525042Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:33:32.2525518Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:33:32.2526920Z cuda train BertForQuestionAnswering 2024-08-20T22:33:40.1166935Z W0820 22:33:40.115000 3913 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:33:48.1898054Z pass 2024-08-20T22:33:48.2770775Z TIMING: entire_frame_compile:13.4389 backend_compile:0.00234 total_wall_time:13.4389 2024-08-20T22:33:48.2771851Z STATS: call_* op count: 993 | FakeTensorMode.__torch_dispatch__:7207 | FakeTensor.__torch_dispatch__:3594 2024-08-20T22:33:48.2772897Z Dynamo produced 2 graphs covering 993 ops with 5 graph breaks (4 unique) 2024-08-20T22:33:52.2968216Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:33:52.2970055Z warnings.warn( 2024-08-20T22:33:52.4754737Z 2024-08-20T22:34:21.8840399Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:34:21.8843401Z loading model: 0it [00:29, ?it/s] 2024-08-20T22:34:21.8844386Z cuda train BlenderbotForCausalLM 2024-08-20T22:34:21.8848298Z Traceback (most recent call last): 2024-08-20T22:34:21.8849069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-20T22:34:21.8849867Z self.model_iter_fn(model, example_inputs) 2024-08-20T22:34:21.8852826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in forward_and_backward_pass 2024-08-20T22:34:21.8854034Z pred = mod(**cloned_inputs) 2024-08-20T22:34:21.8855432Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:34:21.8856450Z return self._call_impl(*args, **kwargs) 2024-08-20T22:34:21.8857789Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:34:21.8858661Z return forward_call(*args, **kwargs) 2024-08-20T22:34:21.8860286Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 1531, in forward 2024-08-20T22:34:21.8861583Z outputs = self.model.decoder( 2024-08-20T22:34:21.8863058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:34:21.8864498Z return self._call_impl(*args, **kwargs) 2024-08-20T22:34:21.8865757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:34:21.8866800Z return forward_call(*args, **kwargs) 2024-08-20T22:34:21.8868218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 997, in forward 2024-08-20T22:34:21.8869481Z layer_outputs = decoder_layer( 2024-08-20T22:34:21.8870586Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:34:21.8871480Z return self._call_impl(*args, **kwargs) 2024-08-20T22:34:21.8872421Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:34:21.8873261Z return forward_call(*args, **kwargs) 2024-08-20T22:34:21.8874349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 397, in forward 2024-08-20T22:34:21.8875488Z hidden_states, self_attn_weights, present_key_value = self.self_attn( 2024-08-20T22:34:21.8876628Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:34:21.8877525Z return self._call_impl(*args, **kwargs) 2024-08-20T22:34:21.8878516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:34:21.8879356Z return forward_call(*args, **kwargs) 2024-08-20T22:34:21.8880438Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 152, in forward 2024-08-20T22:34:21.8881486Z query_states = self.q_proj(hidden_states) * self.scaling 2024-08-20T22:34:21.8882564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:34:21.8883599Z return self._call_impl(*args, **kwargs) 2024-08-20T22:34:21.8884548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:34:21.8885394Z return forward_call(*args, **kwargs) 2024-08-20T22:34:21.8886290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 125, in forward 2024-08-20T22:34:21.8887150Z return F.linear(input, self.weight, self.bias) 2024-08-20T22:34:21.8890500Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 14.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 6.44 MiB is free. Process 187528 has 21.96 GiB memory in use. Of the allocated memory 21.61 GiB is allocated by PyTorch, and 27.77 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-08-20T22:34:21.8893518Z 2024-08-20T22:34:21.8893818Z The above exception was the direct cause of the following exception: 2024-08-20T22:34:21.8894277Z 2024-08-20T22:34:21.8894423Z Traceback (most recent call last): 2024-08-20T22:34:21.8895087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T22:34:21.8895767Z ) = runner.load_model( 2024-08-20T22:34:21.8896441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 454, in load_model 2024-08-20T22:34:21.8897222Z self.validate_model(model, example_inputs) 2024-08-20T22:34:21.8897988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-20T22:34:21.8898766Z raise RuntimeError("Eager run failed") from e 2024-08-20T22:34:21.8899235Z RuntimeError: Eager run failed 2024-08-20T22:34:21.8899498Z 2024-08-20T22:34:21.8899615Z eager_fail_to_run 2024-08-20T22:34:25.5765828Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:34:25.5767506Z warnings.warn( 2024-08-20T22:34:25.8127330Z 2024-08-20T22:34:27.3378795Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:34:27.3379283Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:34:27.3379766Z cuda train BlenderbotSmallForCausalLM 2024-08-20T22:34:27.3589945Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForCausalLM. Setting accuracy check to cosine 2024-08-20T22:34:28.8412242Z pass 2024-08-20T22:34:28.8548940Z TIMING: entire_frame_compile:0.84223 backend_compile:0.00035 total_wall_time:0.84223 2024-08-20T22:34:28.8551023Z STATS: call_* op count: 58 | FakeTensorMode.__torch_dispatch__:653 | FakeTensor.__torch_dispatch__:46 2024-08-20T22:34:28.8552815Z Dynamo produced 6 graphs covering 58 ops with 6 graph breaks (5 unique) 2024-08-20T22:34:32.5980611Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:34:32.5983934Z warnings.warn( 2024-08-20T22:34:32.8058939Z 2024-08-20T22:34:34.8202635Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:34:34.8203337Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:34:34.8203956Z cuda train BlenderbotSmallForConditionalGeneration 2024-08-20T22:34:34.8679209Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForConditionalGeneration. Setting accuracy check to cosine 2024-08-20T22:34:37.6164928Z pass 2024-08-20T22:34:37.6484123Z TIMING: entire_frame_compile:1.57296 backend_compile:0.00048 total_wall_time:1.57296 2024-08-20T22:34:37.6485321Z STATS: call_* op count: 121 | FakeTensorMode.__torch_dispatch__:1227 | FakeTensor.__torch_dispatch__:114 2024-08-20T22:34:37.6486432Z Dynamo produced 7 graphs covering 121 ops with 8 graph breaks (5 unique) 2024-08-20T22:34:41.4311198Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:34:41.4312897Z warnings.warn( 2024-08-20T22:34:41.6058533Z 2024-08-20T22:34:43.7356628Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:34:43.7357238Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:34:43.7357748Z cuda train CamemBert 2024-08-20T22:34:52.0145636Z W0820 22:34:52.013000 4083 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:35:00.2346892Z pass 2024-08-20T22:35:00.3223896Z TIMING: entire_frame_compile:13.65066 backend_compile:0.00237 total_wall_time:13.65066 2024-08-20T22:35:00.3226248Z STATS: call_* op count: 1003 | FakeTensorMode.__torch_dispatch__:7195 | FakeTensor.__torch_dispatch__:3647 2024-08-20T22:35:00.3228116Z Dynamo produced 2 graphs covering 1003 ops with 5 graph breaks (4 unique) 2024-08-20T22:35:04.3294623Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:35:04.3296292Z warnings.warn( 2024-08-20T22:35:04.5447767Z 2024-08-20T22:35:07.2447612Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:35:07.2448072Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:35:07.2448548Z cuda train DebertaForMaskedLM 2024-08-20T22:35:16.4462172Z W0820 22:35:16.445000 4126 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:35:23.1506094Z pass 2024-08-20T22:35:23.2120675Z TIMING: entire_frame_compile:12.80528 backend_compile:0.00207 total_wall_time:12.80528 2024-08-20T22:35:23.2121938Z STATS: call_* op count: 1318 | FakeTensorMode.__torch_dispatch__:8414 | FakeTensor.__torch_dispatch__:2981 2024-08-20T22:35:23.2123055Z Dynamo produced 2 graphs covering 1318 ops with 5 graph breaks (4 unique) 2024-08-20T22:35:27.1853133Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:35:27.1854824Z warnings.warn( 2024-08-20T22:35:27.4639369Z 2024-08-20T22:35:29.5920252Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:35:29.5920746Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:35:29.5921257Z cuda train DebertaForQuestionAnswering 2024-08-20T22:35:38.2047733Z W0820 22:35:38.203000 4169 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:35:44.7714437Z pass 2024-08-20T22:35:44.8421259Z TIMING: entire_frame_compile:12.6111 backend_compile:0.002 total_wall_time:12.6111 2024-08-20T22:35:44.8422237Z STATS: call_* op count: 1316 | FakeTensorMode.__torch_dispatch__:8453 | FakeTensor.__torch_dispatch__:2928 2024-08-20T22:35:44.8423233Z Dynamo produced 2 graphs covering 1316 ops with 5 graph breaks (4 unique) 2024-08-20T22:35:48.8432578Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:35:48.8434316Z warnings.warn( 2024-08-20T22:35:49.0227785Z 2024-08-20T22:36:02.2229768Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:36:02.2230387Z loading model: 0it [00:13, ?it/s] 2024-08-20T22:36:02.2230865Z cuda train DebertaV2ForMaskedLM 2024-08-20T22:36:02.2231696Z pass_due_to_skip 2024-08-20T22:36:02.4472264Z TIMING: 2024-08-20T22:36:02.4472590Z STATS: call_* op count: 0 2024-08-20T22:36:02.4473153Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:36:06.1041083Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:36:06.1042984Z warnings.warn( 2024-08-20T22:36:06.3071292Z 2024-08-20T22:36:17.1024839Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:36:17.1025337Z loading model: 0it [00:10, ?it/s] 2024-08-20T22:36:17.1025884Z cuda train DebertaV2ForQuestionAnswering 2024-08-20T22:36:27.7537892Z ERROR:common: 2024-08-20T22:36:27.7538339Z Traceback (most recent call last): 2024-08-20T22:36:27.7539412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2732, in check_accuracy 2024-08-20T22:36:27.7540170Z correct_result = self.run_n_iterations( 2024-08-20T22:36:27.7541733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2545, in run_n_iterations 2024-08-20T22:36:27.7542557Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-08-20T22:36:27.7543477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in forward_and_backward_pass 2024-08-20T22:36:27.7544304Z pred = mod(**cloned_inputs) 2024-08-20T22:36:27.7545459Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:36:27.7546485Z return self._call_impl(*args, **kwargs) 2024-08-20T22:36:27.7547541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:36:27.7548757Z return forward_call(*args, **kwargs) 2024-08-20T22:36:27.7550137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1486, in forward 2024-08-20T22:36:27.7551380Z outputs = self.deberta( 2024-08-20T22:36:27.7552369Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:36:27.7553255Z return self._call_impl(*args, **kwargs) 2024-08-20T22:36:27.7554188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:36:27.7555067Z return forward_call(*args, **kwargs) 2024-08-20T22:36:27.7556120Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1070, in forward 2024-08-20T22:36:27.7557084Z encoder_outputs = self.encoder( 2024-08-20T22:36:27.7558062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:36:27.7558944Z return self._call_impl(*args, **kwargs) 2024-08-20T22:36:27.7560240Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:36:27.7561147Z return forward_call(*args, **kwargs) 2024-08-20T22:36:27.7562344Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 514, in forward 2024-08-20T22:36:27.7563519Z output_states = layer_module( 2024-08-20T22:36:27.7564605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:36:27.7565557Z return self._call_impl(*args, **kwargs) 2024-08-20T22:36:27.7566571Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:36:27.7567481Z return forward_call(*args, **kwargs) 2024-08-20T22:36:27.7568648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 362, in forward 2024-08-20T22:36:27.7569683Z attention_output = self.attention( 2024-08-20T22:36:27.7570747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:36:27.7571703Z return self._call_impl(*args, **kwargs) 2024-08-20T22:36:27.7572717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:36:27.7573613Z return forward_call(*args, **kwargs) 2024-08-20T22:36:27.7574809Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 293, in forward 2024-08-20T22:36:27.7575812Z self_output = self.self( 2024-08-20T22:36:27.7576815Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:36:27.7577966Z return self._call_impl(*args, **kwargs) 2024-08-20T22:36:27.7578967Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:36:27.7579804Z return forward_call(*args, **kwargs) 2024-08-20T22:36:27.7580867Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 733, in forward 2024-08-20T22:36:27.7582117Z attention_probs = XSoftmax.apply(attention_scores, attention_mask, -1) 2024-08-20T22:36:27.7583155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/function.py", line 575, in apply 2024-08-20T22:36:27.7584037Z return super().apply(*args, **kwargs) # type: ignore[misc] 2024-08-20T22:36:27.7585258Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 113, in forward 2024-08-20T22:36:27.7586380Z output = torch.softmax(output, self.dim) 2024-08-20T22:36:27.7589676Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 24.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 10.44 MiB is free. Process 187893 has 21.96 GiB memory in use. Of the allocated memory 21.29 GiB is allocated by PyTorch, and 318.25 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-08-20T22:36:27.7864628Z eager_1st_run_OOM 2024-08-20T22:36:27.7865009Z TIMING: 2024-08-20T22:36:27.7865301Z STATS: call_* op count: 0 2024-08-20T22:36:27.7865856Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:36:31.4562360Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:36:31.4564154Z warnings.warn( 2024-08-20T22:36:31.6316270Z 2024-08-20T22:36:33.1696825Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:36:33.1697304Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:36:33.1697756Z cuda train DistilBertForMaskedLM 2024-08-20T22:36:36.6164877Z W0820 22:36:36.615000 4296 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:36:40.8618489Z pass 2024-08-20T22:36:40.9106840Z TIMING: entire_frame_compile:6.67916 backend_compile:0.00119 total_wall_time:6.67916 2024-08-20T22:36:40.9107914Z STATS: call_* op count: 540 | FakeTensorMode.__torch_dispatch__:3744 | FakeTensor.__torch_dispatch__:1901 2024-08-20T22:36:40.9108837Z Dynamo produced 2 graphs covering 540 ops with 5 graph breaks (4 unique) 2024-08-20T22:36:44.7344094Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:36:44.7345744Z warnings.warn( 2024-08-20T22:36:44.8996130Z 2024-08-20T22:36:46.1617232Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:36:46.1617847Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:36:46.1618341Z cuda train DistilBertForQuestionAnswering 2024-08-20T22:36:49.5077214Z W0820 22:36:49.506000 4339 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:36:53.6713547Z pass 2024-08-20T22:36:53.7211300Z TIMING: entire_frame_compile:6.58917 backend_compile:0.00116 total_wall_time:6.58917 2024-08-20T22:36:53.7212447Z STATS: call_* op count: 539 | FakeTensorMode.__torch_dispatch__:3795 | FakeTensor.__torch_dispatch__:1848 2024-08-20T22:36:53.7213394Z Dynamo produced 2 graphs covering 539 ops with 5 graph breaks (4 unique) 2024-08-20T22:36:57.5687572Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:36:57.5690811Z warnings.warn( 2024-08-20T22:36:57.7763272Z 2024-08-20T22:36:59.8506483Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:36:59.8506958Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:36:59.8509110Z cuda train DistillGPT2 2024-08-20T22:37:04.1454057Z W0820 22:37:04.144000 4382 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:37:07.2940269Z pass 2024-08-20T22:37:07.3372620Z TIMING: entire_frame_compile:5.5432 backend_compile:0.00091 total_wall_time:5.5432 2024-08-20T22:37:07.3373974Z STATS: call_* op count: 571 | FakeTensorMode.__torch_dispatch__:3500 | FakeTensor.__torch_dispatch__:1376 2024-08-20T22:37:07.3374919Z Dynamo produced 2 graphs covering 571 ops with 5 graph breaks (4 unique) 2024-08-20T22:37:11.1668983Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:37:11.1671784Z warnings.warn( 2024-08-20T22:37:11.3760142Z 2024-08-20T22:37:11.3769387Z loading model: 0it [00:00, ?it/s]If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2024-08-20T22:37:12.1877667Z 2024-08-20T22:37:12.1878157Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:37:12.1878813Z cuda train ElectraForCausalLM 2024-08-20T22:37:18.9277919Z pass 2024-08-20T22:37:18.9493180Z TIMING: entire_frame_compile:5.70968 backend_compile:0.00046 total_wall_time:5.70968 2024-08-20T22:37:18.9494291Z STATS: call_* op count: 377 | FakeTensorMode.__torch_dispatch__:7277 | FakeTensor.__torch_dispatch__:11 2024-08-20T22:37:18.9495284Z Dynamo produced 1 graphs covering 377 ops with 4 graph breaks (4 unique) 2024-08-20T22:37:22.7504884Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:37:22.7506618Z warnings.warn( 2024-08-20T22:37:22.9619784Z 2024-08-20T22:37:23.7406070Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:37:23.7406684Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:37:23.7407288Z cuda train ElectraForQuestionAnswering 2024-08-20T22:37:30.2881558Z W0820 22:37:30.287000 4468 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:37:38.4030413Z pass 2024-08-20T22:37:38.4172950Z TIMING: entire_frame_compile:13.58653 backend_compile:0.0024 total_wall_time:13.58653 2024-08-20T22:37:38.4174236Z STATS: call_* op count: 1000 | FakeTensorMode.__torch_dispatch__:7269 | FakeTensor.__torch_dispatch__:3630 2024-08-20T22:37:38.4175173Z Dynamo produced 2 graphs covering 1000 ops with 5 graph breaks (4 unique) 2024-08-20T22:37:42.3947397Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:37:42.3949073Z warnings.warn( 2024-08-20T22:37:42.5998652Z 2024-08-20T22:37:45.0345299Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:37:45.0346364Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:37:45.0350108Z cuda train GPT2ForSequenceClassification 2024-08-20T22:37:55.9792438Z W0820 22:37:55.978000 4511 torch/_logging/_internal.py:1058] [7/0] Profiler function will be ignored 2024-08-20T22:38:02.1183401Z pass 2024-08-20T22:38:02.2223811Z TIMING: entire_frame_compile:12.81435 backend_compile:0.00202 total_wall_time:12.81435 2024-08-20T22:38:02.2224857Z STATS: call_* op count: 1101 | FakeTensorMode.__torch_dispatch__:9657 | FakeTensor.__torch_dispatch__:2717 2024-08-20T22:38:02.2225786Z Dynamo produced 5 graphs covering 1101 ops with 7 graph breaks (5 unique) 2024-08-20T22:38:06.1640680Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:38:06.1642369Z warnings.warn( 2024-08-20T22:38:06.3446728Z 2024-08-20T22:38:07.8417173Z loading model: 0it [00:00, ?it/s]WARNING:common:Model GoogleFnet supports float32 only 2024-08-20T22:38:08.0604808Z 2024-08-20T22:38:08.0605327Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:38:08.0605870Z WARNING:common:Model GoogleFnet supports float32 only 2024-08-20T22:38:08.0616269Z cuda train GoogleFnet 2024-08-20T22:38:09.4003572Z WARNING:common:Model GoogleFnet supports float32 only 2024-08-20T22:38:12.6616970Z W0820 22:38:12.660000 4554 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:38:16.9099308Z pass 2024-08-20T22:38:16.9520920Z TIMING: entire_frame_compile:6.96788 backend_compile:0.00123 total_wall_time:6.96788 2024-08-20T22:38:16.9521937Z STATS: call_* op count: 573 | FakeTensorMode.__torch_dispatch__:2194 | FakeTensor.__torch_dispatch__:1955 2024-08-20T22:38:16.9522857Z Dynamo produced 2 graphs covering 573 ops with 5 graph breaks (4 unique) 2024-08-20T22:38:20.7981594Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:38:20.7983290Z warnings.warn( 2024-08-20T22:38:20.9716147Z 2024-08-20T22:38:23.2114519Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:38:23.2115129Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:38:23.2118327Z cuda train LayoutLMForMaskedLM 2024-08-20T22:38:31.6945191Z W0820 22:38:31.693000 4597 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:38:40.0602409Z pass 2024-08-20T22:38:40.1540511Z TIMING: entire_frame_compile:13.97785 backend_compile:0.00245 total_wall_time:13.97785 2024-08-20T22:38:40.1541508Z STATS: call_* op count: 1033 | FakeTensorMode.__torch_dispatch__:7450 | FakeTensor.__torch_dispatch__:3719 2024-08-20T22:38:40.1542458Z Dynamo produced 2 graphs covering 1033 ops with 5 graph breaks (4 unique) 2024-08-20T22:38:44.1457984Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:38:44.1459667Z warnings.warn( 2024-08-20T22:38:44.3270394Z 2024-08-20T22:38:46.2648612Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:38:46.2649079Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:38:46.2649559Z cuda train LayoutLMForSequenceClassification 2024-08-20T22:38:56.7181365Z W0820 22:38:56.717000 4640 torch/_logging/_internal.py:1058] [7/0] Profiler function will be ignored 2024-08-20T22:39:05.1239121Z pass 2024-08-20T22:39:05.2139322Z TIMING: entire_frame_compile:16.32606 backend_compile:0.00268 total_wall_time:16.32606 2024-08-20T22:39:05.2140896Z STATS: call_* op count: 1030 | FakeTensorMode.__torch_dispatch__:10895 | FakeTensor.__torch_dispatch__:3726 2024-08-20T22:39:05.2142184Z Dynamo produced 5 graphs covering 1030 ops with 7 graph breaks (5 unique) 2024-08-20T22:39:09.2701153Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:39:09.2702944Z warnings.warn( 2024-08-20T22:39:09.4444186Z 2024-08-20T22:39:21.0080850Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:39:21.0081417Z loading model: 0it [00:11, ?it/s] 2024-08-20T22:39:21.0081891Z cuda train M2M100ForConditionalGeneration 2024-08-20T22:39:21.0853604Z WARNING:common:fp64 golden ref were not generated for M2M100ForConditionalGeneration. Setting accuracy check to cosine 2024-08-20T22:39:35.9203867Z pass 2024-08-20T22:39:36.1275939Z TIMING: entire_frame_compile:12.85771 backend_compile:0.0012 total_wall_time:12.85771 2024-08-20T22:39:36.1276959Z STATS: call_* op count: 1296 | FakeTensorMode.__torch_dispatch__:16961 | FakeTensor.__torch_dispatch__:12 2024-08-20T22:39:36.1277889Z Dynamo produced 1 graphs covering 1296 ops with 4 graph breaks (4 unique) 2024-08-20T22:39:40.0374164Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:39:40.0375811Z warnings.warn( 2024-08-20T22:39:40.2761089Z 2024-08-20T22:39:44.3190621Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:39:44.3191093Z loading model: 0it [00:04, ?it/s] 2024-08-20T22:39:44.3191541Z cuda train MBartForCausalLM 2024-08-20T22:39:44.3494788Z WARNING:common:fp64 golden ref were not generated for MBartForCausalLM. Setting accuracy check to cosine 2024-08-20T22:39:46.3530764Z pass 2024-08-20T22:39:46.3975724Z TIMING: entire_frame_compile:0.86819 backend_compile:0.00036 total_wall_time:0.86819 2024-08-20T22:39:46.3976994Z STATS: call_* op count: 60 | FakeTensorMode.__torch_dispatch__:677 | FakeTensor.__torch_dispatch__:46 2024-08-20T22:39:46.3978009Z Dynamo produced 6 graphs covering 60 ops with 6 graph breaks (5 unique) 2024-08-20T22:39:50.1176622Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:39:50.1178327Z warnings.warn( 2024-08-20T22:39:50.3547363Z 2024-08-20T22:39:57.4729815Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:39:57.4730275Z loading model: 0it [00:07, ?it/s] 2024-08-20T22:39:57.4730752Z cuda train MBartForConditionalGeneration 2024-08-20T22:39:57.5715919Z WARNING:common:fp64 golden ref were not generated for MBartForConditionalGeneration. Setting accuracy check to cosine 2024-08-20T22:40:02.4924794Z pass 2024-08-20T22:40:02.8165263Z TIMING: entire_frame_compile:1.72727 backend_compile:0.00052 total_wall_time:1.72727 2024-08-20T22:40:02.8166245Z STATS: call_* op count: 136 | FakeTensorMode.__torch_dispatch__:1326 | FakeTensor.__torch_dispatch__:114 2024-08-20T22:40:02.8167171Z Dynamo produced 8 graphs covering 136 ops with 8 graph breaks (5 unique) 2024-08-20T22:40:06.5714553Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:40:06.5716196Z warnings.warn( 2024-08-20T22:40:06.7334807Z 2024-08-20T22:40:10.9132691Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:40:10.9133301Z loading model: 0it [00:04, ?it/s] 2024-08-20T22:40:10.9133862Z cuda train MT5ForConditionalGeneration 2024-08-20T22:40:22.4006775Z W0820 22:40:22.399000 4812 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:40:31.5246999Z pass 2024-08-20T22:40:31.7094152Z TIMING: entire_frame_compile:17.83737 backend_compile:0.00241 total_wall_time:17.83737 2024-08-20T22:40:31.7095390Z STATS: call_* op count: 1763 | FakeTensorMode.__torch_dispatch__:14179 | FakeTensor.__torch_dispatch__:3432 2024-08-20T22:40:31.7096302Z Dynamo produced 2 graphs covering 1763 ops with 5 graph breaks (4 unique) 2024-08-20T22:40:35.7816308Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:40:35.7818025Z warnings.warn( 2024-08-20T22:40:35.9845728Z 2024-08-20T22:40:35.9885479Z loading model: 0it [00:00, ?it/s]If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2024-08-20T22:40:40.8493903Z 2024-08-20T22:40:40.8494391Z loading model: 0it [00:04, ?it/s] 2024-08-20T22:40:40.8496586Z cuda train MegatronBertForCausalLM 2024-08-20T22:40:58.7907215Z W0820 22:40:58.789000 4855 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:41:14.9559227Z pass 2024-08-20T22:41:15.2105741Z TIMING: entire_frame_compile:26.5647 backend_compile:0.0051 total_wall_time:26.5647 2024-08-20T22:41:15.2108577Z STATS: call_* op count: 1922 | FakeTensorMode.__torch_dispatch__:13896 | FakeTensor.__torch_dispatch__:7103 2024-08-20T22:41:15.2109536Z Dynamo produced 2 graphs covering 1922 ops with 5 graph breaks (4 unique) 2024-08-20T22:41:19.4757126Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:41:19.4760746Z warnings.warn( 2024-08-20T22:41:19.6873299Z 2024-08-20T22:41:24.2051373Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:41:24.2051981Z loading model: 0it [00:04, ?it/s] 2024-08-20T22:41:24.2052532Z cuda train MegatronBertForQuestionAnswering 2024-08-20T22:41:41.6944064Z W0820 22:41:41.693000 4898 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:41:57.8260774Z pass 2024-08-20T22:41:58.0739455Z TIMING: entire_frame_compile:26.50299 backend_compile:0.00488 total_wall_time:26.50299 2024-08-20T22:41:58.0740461Z STATS: call_* op count: 1916 | FakeTensorMode.__torch_dispatch__:13893 | FakeTensor.__torch_dispatch__:7050 2024-08-20T22:41:58.0741396Z Dynamo produced 2 graphs covering 1916 ops with 5 graph breaks (4 unique) 2024-08-20T22:42:02.3471722Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:42:02.3473420Z warnings.warn( 2024-08-20T22:42:02.5855433Z 2024-08-20T22:42:04.0442790Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:42:04.0443808Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:42:04.0444710Z cuda train MobileBertForMaskedLM 2024-08-20T22:42:33.7997136Z pass 2024-08-20T22:42:33.8500827Z TIMING: entire_frame_compile:26.50587 backend_compile:0.00252 total_wall_time:26.50587 2024-08-20T22:42:33.8501808Z STATS: call_* op count: 1449 | FakeTensorMode.__torch_dispatch__:26824 | FakeTensor.__torch_dispatch__:8 2024-08-20T22:42:33.8502736Z Dynamo produced 1 graphs covering 1449 ops with 3 graph breaks (3 unique) 2024-08-20T22:42:37.9764106Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:42:37.9766174Z warnings.warn( 2024-08-20T22:42:38.1786368Z 2024-08-20T22:42:39.4505331Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:42:39.4506218Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:42:39.4507166Z cuda train MobileBertForQuestionAnswering 2024-08-20T22:43:09.2780556Z pass 2024-08-20T22:43:09.3133934Z TIMING: entire_frame_compile:26.61658 backend_compile:0.00251 total_wall_time:26.61658 2024-08-20T22:43:09.3136466Z STATS: call_* op count: 1453 | FakeTensorMode.__torch_dispatch__:26856 | FakeTensor.__torch_dispatch__:12 2024-08-20T22:43:09.3137419Z Dynamo produced 1 graphs covering 1453 ops with 3 graph breaks (3 unique) 2024-08-20T22:43:13.4232501Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:43:13.4234182Z warnings.warn( 2024-08-20T22:43:13.6342605Z 2024-08-20T22:43:16.2183265Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:43:16.2183747Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:43:16.2184206Z cuda train OPTForCausalLM 2024-08-20T22:43:16.2375136Z WARNING:common:fp64 golden ref were not generated for OPTForCausalLM. Setting accuracy check to cosine 2024-08-20T22:43:19.0627225Z pass 2024-08-20T22:43:19.1104355Z TIMING: entire_frame_compile:0.87327 backend_compile:0.00037 total_wall_time:0.87327 2024-08-20T22:43:19.1107217Z STATS: call_* op count: 82 | FakeTensorMode.__torch_dispatch__:772 | FakeTensor.__torch_dispatch__:44 2024-08-20T22:43:19.1108345Z Dynamo produced 6 graphs covering 82 ops with 6 graph breaks (5 unique) 2024-08-20T22:43:22.8574029Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:43:22.8575718Z warnings.warn( 2024-08-20T22:43:23.0306873Z 2024-08-20T22:43:25.0655060Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:43:25.0655542Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:43:25.0655994Z cuda train PLBartForCausalLM 2024-08-20T22:43:25.0817562Z WARNING:common:fp64 golden ref were not generated for PLBartForCausalLM. Setting accuracy check to cosine 2024-08-20T22:43:26.5674019Z pass 2024-08-20T22:43:26.5990219Z TIMING: entire_frame_compile:0.85214 backend_compile:0.00035 total_wall_time:0.85214 2024-08-20T22:43:26.5991219Z STATS: call_* op count: 60 | FakeTensorMode.__torch_dispatch__:681 | FakeTensor.__torch_dispatch__:46 2024-08-20T22:43:26.5992112Z Dynamo produced 6 graphs covering 60 ops with 6 graph breaks (5 unique) 2024-08-20T22:43:30.3417811Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:43:30.3419456Z warnings.warn( 2024-08-20T22:43:30.5196132Z 2024-08-20T22:43:34.0011421Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:43:34.0011892Z loading model: 0it [00:03, ?it/s] 2024-08-20T22:43:34.0015049Z cuda train PLBartForConditionalGeneration 2024-08-20T22:43:34.0546831Z WARNING:common:fp64 golden ref were not generated for PLBartForConditionalGeneration. Setting accuracy check to cosine 2024-08-20T22:43:37.0434904Z pass 2024-08-20T22:43:37.0991945Z TIMING: entire_frame_compile:1.59047 backend_compile:0.00053 total_wall_time:1.59047 2024-08-20T22:43:37.0993912Z STATS: call_* op count: 137 | FakeTensorMode.__torch_dispatch__:1340 | FakeTensor.__torch_dispatch__:114 2024-08-20T22:43:37.0995489Z Dynamo produced 8 graphs covering 137 ops with 8 graph breaks (5 unique) 2024-08-20T22:43:40.8788646Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:43:40.8790340Z warnings.warn( 2024-08-20T22:43:41.0832490Z 2024-08-20T22:43:47.2696351Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:43:47.2696822Z loading model: 0it [00:06, ?it/s] 2024-08-20T22:43:47.2697291Z cuda train PegasusForCausalLM 2024-08-20T22:43:47.2993027Z WARNING:common:fp64 golden ref were not generated for PegasusForCausalLM. Setting accuracy check to cosine 2024-08-20T22:43:49.0250146Z pass 2024-08-20T22:43:49.0478774Z TIMING: entire_frame_compile:0.8216 backend_compile:0.00036 total_wall_time:0.8216 2024-08-20T22:43:49.0480071Z STATS: call_* op count: 58 | FakeTensorMode.__torch_dispatch__:624 | FakeTensor.__torch_dispatch__:46 2024-08-20T22:43:49.0480997Z Dynamo produced 6 graphs covering 58 ops with 6 graph breaks (5 unique) 2024-08-20T22:43:52.8198017Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:43:52.8199704Z warnings.warn( 2024-08-20T22:43:52.9975292Z 2024-08-20T22:44:03.2860680Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:44:03.2861143Z loading model: 0it [00:10, ?it/s] 2024-08-20T22:44:03.2865725Z cuda train PegasusForConditionalGeneration 2024-08-20T22:44:03.3608135Z WARNING:common:fp64 golden ref were not generated for PegasusForConditionalGeneration. Setting accuracy check to cosine 2024-08-20T22:44:06.8138384Z pass 2024-08-20T22:44:07.0206661Z TIMING: entire_frame_compile:1.52303 backend_compile:0.00047 total_wall_time:1.52303 2024-08-20T22:44:07.0207686Z STATS: call_* op count: 121 | FakeTensorMode.__torch_dispatch__:1171 | FakeTensor.__torch_dispatch__:114 2024-08-20T22:44:07.0208624Z Dynamo produced 7 graphs covering 121 ops with 7 graph breaks (4 unique) 2024-08-20T22:44:10.8198860Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:44:10.8200511Z warnings.warn( 2024-08-20T22:44:11.0166898Z 2024-08-20T22:44:11.0195062Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2024-08-20T22:44:13.1789692Z We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2024-08-20T22:44:13.1791681Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2024-08-20T22:44:13.4585095Z 2024-08-20T22:44:13.4585768Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:44:13.4586327Z cuda train RobertaForCausalLM 2024-08-20T22:44:21.9638394Z W0820 22:44:21.963000 5242 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:44:30.2004466Z pass 2024-08-20T22:44:30.2737579Z TIMING: entire_frame_compile:13.65693 backend_compile:0.00243 total_wall_time:13.65693 2024-08-20T22:44:30.2738885Z STATS: call_* op count: 1007 | FakeTensorMode.__torch_dispatch__:7232 | FakeTensor.__torch_dispatch__:3647 2024-08-20T22:44:30.2739950Z Dynamo produced 2 graphs covering 1007 ops with 5 graph breaks (4 unique) 2024-08-20T22:44:34.2685250Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:44:34.2689204Z warnings.warn( 2024-08-20T22:44:34.4656488Z 2024-08-20T22:44:36.1952862Z loading model: 0it [00:00, ?it/s]We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2024-08-20T22:44:36.1954905Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2024-08-20T22:44:36.5167050Z 2024-08-20T22:44:36.5167771Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:44:36.5168320Z cuda train RobertaForQuestionAnswering 2024-08-20T22:44:44.4435921Z W0820 22:44:44.442000 5285 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:44:52.5965161Z pass 2024-08-20T22:44:52.6888083Z TIMING: entire_frame_compile:13.5384 backend_compile:0.00234 total_wall_time:13.5384 2024-08-20T22:44:52.6890469Z STATS: call_* op count: 1000 | FakeTensorMode.__torch_dispatch__:7225 | FakeTensor.__torch_dispatch__:3594 2024-08-20T22:44:52.6891685Z Dynamo produced 2 graphs covering 1000 ops with 5 graph breaks (4 unique) 2024-08-20T22:44:56.6657560Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:44:56.6659333Z warnings.warn( 2024-08-20T22:44:56.8530899Z 2024-08-20T22:44:57.4685655Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:44:57.4686118Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:44:57.4686606Z cuda train Speech2Text2ForCausalLM 2024-08-20T22:44:57.5769915Z WARNING:common:fp64 golden ref were not generated for Speech2Text2ForCausalLM. Setting accuracy check to cosine 2024-08-20T22:44:58.8737938Z pass 2024-08-20T22:44:58.8795114Z TIMING: entire_frame_compile:0.84126 backend_compile:0.00036 total_wall_time:0.84126 2024-08-20T22:44:58.8796108Z STATS: call_* op count: 68 | FakeTensorMode.__torch_dispatch__:696 | FakeTensor.__torch_dispatch__:48 2024-08-20T22:44:58.8797014Z Dynamo produced 6 graphs covering 68 ops with 6 graph breaks (5 unique) 2024-08-20T22:45:02.6062655Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:45:02.6064308Z warnings.warn( 2024-08-20T22:45:02.8482859Z 2024-08-20T22:45:04.3576349Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:45:04.3576848Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:45:04.3578495Z cuda train T5ForConditionalGeneration 2024-08-20T22:45:14.4329952Z W0820 22:45:14.432000 5371 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:45:20.6627930Z pass 2024-08-20T22:45:20.7220423Z TIMING: entire_frame_compile:12.56612 backend_compile:0.0017 total_wall_time:12.56612 2024-08-20T22:45:20.7221420Z STATS: call_* op count: 1223 | FakeTensorMode.__torch_dispatch__:9919 | FakeTensor.__torch_dispatch__:2370 2024-08-20T22:45:20.7222345Z Dynamo produced 2 graphs covering 1223 ops with 5 graph breaks (4 unique) 2024-08-20T22:45:24.7140910Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:45:24.7142591Z warnings.warn( 2024-08-20T22:45:25.0398430Z 2024-08-20T22:45:26.5447143Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:45:26.5448068Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:45:26.5448507Z cuda train T5Small 2024-08-20T22:45:36.6224491Z W0820 22:45:36.621000 5414 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:45:42.8492051Z pass 2024-08-20T22:45:42.9083433Z TIMING: entire_frame_compile:12.57172 backend_compile:0.00164 total_wall_time:12.57172 2024-08-20T22:45:42.9084576Z STATS: call_* op count: 1223 | FakeTensorMode.__torch_dispatch__:9919 | FakeTensor.__torch_dispatch__:2370 2024-08-20T22:45:42.9085785Z Dynamo produced 2 graphs covering 1223 ops with 5 graph breaks (4 unique) 2024-08-20T22:45:46.9089053Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:45:46.9090733Z warnings.warn( 2024-08-20T22:45:47.1291847Z 2024-08-20T22:45:51.1614106Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:45:51.1614713Z loading model: 0it [00:04, ?it/s] 2024-08-20T22:45:51.1615259Z cuda train TrOCRForCausalLM 2024-08-20T22:45:51.1914498Z WARNING:common:fp64 golden ref were not generated for TrOCRForCausalLM. Setting accuracy check to cosine 2024-08-20T22:45:52.9695134Z pass 2024-08-20T22:45:52.9918711Z TIMING: entire_frame_compile:0.86669 backend_compile:0.00037 total_wall_time:0.86669 2024-08-20T22:45:52.9919754Z STATS: call_* op count: 59 | FakeTensorMode.__torch_dispatch__:669 | FakeTensor.__torch_dispatch__:46 2024-08-20T22:45:52.9920658Z Dynamo produced 6 graphs covering 59 ops with 6 graph breaks (5 unique) 2024-08-20T22:45:56.7417880Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:45:56.7420698Z warnings.warn( 2024-08-20T22:45:56.9103468Z 2024-08-20T22:46:06.8376425Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:46:06.8377031Z loading model: 0it [00:09, ?it/s] 2024-08-20T22:46:06.8377594Z cuda train XGLMForCausalLM 2024-08-20T22:46:06.8895476Z WARNING:common:fp64 golden ref were not generated for XGLMForCausalLM. Setting accuracy check to cosine 2024-08-20T22:46:09.2232313Z fail_accuracy 2024-08-20T22:46:09.2666830Z TIMING: entire_frame_compile:0.88548 backend_compile:0.00037 total_wall_time:0.88548 2024-08-20T22:46:09.2668172Z STATS: call_* op count: 67 | FakeTensorMode.__torch_dispatch__:672 | FakeTensor.__torch_dispatch__:48 2024-08-20T22:46:09.2669081Z Dynamo produced 6 graphs covering 67 ops with 6 graph breaks (5 unique) 2024-08-20T22:46:13.0591622Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:46:13.0593319Z warnings.warn( 2024-08-20T22:46:13.2611060Z 2024-08-20T22:46:18.0749692Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:46:18.0750162Z loading model: 0it [00:04, ?it/s] 2024-08-20T22:46:18.0750625Z cuda train XLNetLMHeadModel 2024-08-20T22:46:40.1339691Z W0820 22:46:40.133000 5543 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:46:58.4010859Z pass 2024-08-20T22:46:58.6671952Z TIMING: entire_frame_compile:31.05844 backend_compile:0.00472 total_wall_time:31.05844 2024-08-20T22:46:58.6672953Z STATS: call_* op count: 1873 | FakeTensorMode.__torch_dispatch__:25064 | FakeTensor.__torch_dispatch__:6527 2024-08-20T22:46:58.6673908Z Dynamo produced 2 graphs covering 1873 ops with 5 graph breaks (4 unique) 2024-08-20T22:47:03.0490521Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:47:03.0492189Z warnings.warn( 2024-08-20T22:47:03.2700627Z 2024-08-20T22:47:05.5949732Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:47:05.5950601Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:47:05.5951493Z cuda train YituTechConvBert 2024-08-20T22:47:16.9283381Z W0820 22:47:16.927000 5586 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:47:29.2412398Z pass 2024-08-20T22:47:29.3128480Z TIMING: entire_frame_compile:20.62362 backend_compile:0.00363 total_wall_time:20.62362 2024-08-20T22:47:29.3129894Z STATS: call_* op count: 1511 | FakeTensorMode.__torch_dispatch__:10843 | FakeTensor.__torch_dispatch__:5159 2024-08-20T22:47:29.3130854Z Dynamo produced 2 graphs covering 1511 ops with 5 graph breaks (4 unique) 2024-08-20T22:47:30.6208257Z accuracy pass_rate=91.30% 2024-08-20T22:47:30.6212118Z calls_captured gmean=0.00x mean=750.522x 2024-08-20T22:47:30.6215658Z unique_graphs gmean=0.00x mean=3.348x 2024-08-20T22:47:30.6219632Z graph_breaks gmean=0.00x mean=5.196x 2024-08-20T22:47:30.6223421Z unique_graph_breaks gmean=0.00x mean=4.043x 2024-08-20T22:47:30.6227429Z autograd_captures gmean=0.00x mean=0.000x 2024-08-20T22:47:30.6231203Z autograd_compiles gmean=0.00x mean=0.000x 2024-08-20T22:47:30.6235086Z cudagraph_skips gmean=0.00x mean=0.000x 2024-08-20T22:47:31.2489342Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamo_eager_huggingface_training.csv 2024-08-20T22:47:31.5118823Z AlbertForMaskedLM PASS 2024-08-20T22:47:31.5121615Z AlbertForQuestionAnswering PASS 2024-08-20T22:47:31.5127278Z AllenaiLongformerBase PASS 2024-08-20T22:47:31.5130967Z BartForCausalLM PASS 2024-08-20T22:47:31.5135583Z BartForConditionalGeneration PASS 2024-08-20T22:47:31.5139999Z BertForMaskedLM PASS 2024-08-20T22:47:31.5144380Z BertForQuestionAnswering PASS 2024-08-20T22:47:31.5148967Z BlenderbotForCausalLM XFAIL 2024-08-20T22:47:31.5153437Z BlenderbotSmallForCausalLM PASS 2024-08-20T22:47:31.5158398Z BlenderbotSmallForConditionalGeneration PASS 2024-08-20T22:47:31.5164169Z CamemBert PASS 2024-08-20T22:47:31.5168911Z DebertaForMaskedLM PASS 2024-08-20T22:47:31.5173110Z DebertaForQuestionAnswering PASS 2024-08-20T22:47:31.5177575Z DebertaV2ForMaskedLM XFAIL 2024-08-20T22:47:31.5182212Z DebertaV2ForQuestionAnswering XFAIL 2024-08-20T22:47:31.5186601Z DistilBertForMaskedLM PASS 2024-08-20T22:47:31.5191230Z DistilBertForQuestionAnswering PASS 2024-08-20T22:47:31.5195518Z DistillGPT2 PASS 2024-08-20T22:47:31.5200033Z ElectraForCausalLM PASS 2024-08-20T22:47:31.5204840Z ElectraForQuestionAnswering PASS 2024-08-20T22:47:31.5209428Z GPT2ForSequenceClassification PASS 2024-08-20T22:47:31.5213793Z GoogleFnet PASS 2024-08-20T22:47:31.5218270Z LayoutLMForMaskedLM PASS 2024-08-20T22:47:31.5222756Z LayoutLMForSequenceClassification PASS 2024-08-20T22:47:31.5227173Z M2M100ForConditionalGeneration PASS 2024-08-20T22:47:31.5231548Z MBartForCausalLM PASS 2024-08-20T22:47:31.5236186Z MBartForConditionalGeneration PASS 2024-08-20T22:47:31.5240497Z MT5ForConditionalGeneration PASS 2024-08-20T22:47:31.5245349Z MegatronBertForCausalLM PASS 2024-08-20T22:47:31.5249656Z MegatronBertForQuestionAnswering PASS 2024-08-20T22:47:31.5254213Z MobileBertForMaskedLM PASS 2024-08-20T22:47:31.5258554Z MobileBertForQuestionAnswering PASS 2024-08-20T22:47:31.5263757Z OPTForCausalLM PASS 2024-08-20T22:47:31.5268036Z PLBartForCausalLM PASS 2024-08-20T22:47:31.5272422Z PLBartForConditionalGeneration PASS 2024-08-20T22:47:31.5276744Z PegasusForCausalLM PASS 2024-08-20T22:47:31.5281444Z PegasusForConditionalGeneration PASS 2024-08-20T22:47:31.5286543Z RobertaForCausalLM PASS 2024-08-20T22:47:31.5290779Z RobertaForQuestionAnswering PASS 2024-08-20T22:47:31.5295094Z Speech2Text2ForCausalLM PASS 2024-08-20T22:47:31.5299703Z T5ForConditionalGeneration PASS 2024-08-20T22:47:31.5303935Z T5Small PASS 2024-08-20T22:47:31.5308635Z TrOCRForCausalLM PASS 2024-08-20T22:47:31.5313492Z XGLMForCausalLM FAIL_BUT_FLAKY: accuracy=fail_accuracy, expected=pass 2024-08-20T22:47:31.5317662Z XLNetLMHeadModel PASS 2024-08-20T22:47:31.5322043Z YituTechConvBert PASS 2024-08-20T22:47:31.5774536Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamo_eager_huggingface_training.csv 2024-08-20T22:47:31.8404080Z AlbertForMaskedLM PASS 2024-08-20T22:47:31.8408985Z AlbertForQuestionAnswering PASS 2024-08-20T22:47:31.8414379Z AllenaiLongformerBase PASS 2024-08-20T22:47:31.8419896Z BartForCausalLM PASS 2024-08-20T22:47:31.8425437Z BartForConditionalGeneration PASS 2024-08-20T22:47:31.8430908Z BertForMaskedLM PASS 2024-08-20T22:47:31.8436312Z BertForQuestionAnswering PASS 2024-08-20T22:47:31.8441789Z BlenderbotForCausalLM PASS 2024-08-20T22:47:31.8448055Z BlenderbotSmallForCausalLM PASS 2024-08-20T22:47:31.8453118Z BlenderbotSmallForConditionalGeneration PASS 2024-08-20T22:47:31.8457831Z CamemBert PASS 2024-08-20T22:47:31.8464161Z DebertaForMaskedLM PASS 2024-08-20T22:47:31.8469011Z DebertaForQuestionAnswering PASS 2024-08-20T22:47:31.8473545Z DebertaV2ForMaskedLM PASS 2024-08-20T22:47:31.8478066Z DebertaV2ForQuestionAnswering PASS 2024-08-20T22:47:31.8482667Z DistilBertForMaskedLM PASS 2024-08-20T22:47:31.8487823Z DistilBertForQuestionAnswering PASS 2024-08-20T22:47:31.8492148Z DistillGPT2 PASS 2024-08-20T22:47:31.8496727Z ElectraForCausalLM PASS 2024-08-20T22:47:31.8501187Z ElectraForQuestionAnswering PASS 2024-08-20T22:47:31.8506150Z GPT2ForSequenceClassification PASS 2024-08-20T22:47:31.8511194Z GoogleFnet PASS 2024-08-20T22:47:31.8516426Z LayoutLMForMaskedLM PASS 2024-08-20T22:47:31.8521708Z LayoutLMForSequenceClassification PASS 2024-08-20T22:47:31.8527435Z M2M100ForConditionalGeneration PASS 2024-08-20T22:47:31.8532715Z MBartForCausalLM PASS 2024-08-20T22:47:31.8538282Z MBartForConditionalGeneration PASS 2024-08-20T22:47:31.8543570Z MT5ForConditionalGeneration PASS 2024-08-20T22:47:31.8548942Z MegatronBertForCausalLM PASS 2024-08-20T22:47:31.8554355Z MegatronBertForQuestionAnswering PASS 2024-08-20T22:47:31.8559741Z MobileBertForMaskedLM PASS 2024-08-20T22:47:31.8565313Z MobileBertForQuestionAnswering PASS 2024-08-20T22:47:31.8570390Z OPTForCausalLM PASS 2024-08-20T22:47:31.8574935Z PLBartForCausalLM PASS 2024-08-20T22:47:31.8579585Z PLBartForConditionalGeneration PASS 2024-08-20T22:47:31.8584155Z PegasusForCausalLM PASS 2024-08-20T22:47:31.8588890Z PegasusForConditionalGeneration PASS 2024-08-20T22:47:31.8593262Z RobertaForCausalLM PASS 2024-08-20T22:47:31.8598004Z RobertaForQuestionAnswering PASS 2024-08-20T22:47:31.8602440Z Speech2Text2ForCausalLM PASS 2024-08-20T22:47:31.8607392Z T5ForConditionalGeneration PASS 2024-08-20T22:47:31.8611791Z T5Small PASS 2024-08-20T22:47:31.8616510Z TrOCRForCausalLM PASS 2024-08-20T22:47:31.8621210Z XGLMForCausalLM PASS 2024-08-20T22:47:31.8625804Z XLNetLMHeadModel PASS 2024-08-20T22:47:31.8630432Z YituTechConvBert PASS 2024-08-20T22:47:31.9073528Z + cleanup_workspace 2024-08-20T22:47:31.9074413Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-20T22:47:31.9075515Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-20T22:47:31.9076465Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-20T22:47:31.9077294Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-20T22:47:31.9078152Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-20T22:47:31.9079055Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-20T22:47:31.9079782Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-20T22:47:32.6186593Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-20T22:47:32.6187116Z cat test/**/*_toprint.log || true 2024-08-20T22:47:32.6200362Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:47:32.6200866Z env: 2024-08-20T22:47:32.6201138Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:32.6201597Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:47:32.6202324Z DOCKER_CONTAINER_ID: c5602275dd27032945979a763aa9348ea9f7fd583e459e7a824e50356e8b3793 2024-08-20T22:47:32.6203045Z ##[endgroup] 2024-08-20T22:47:32.6294201Z cat: 'test/**/*_toprint.log': No such file or directory 2024-08-20T22:47:32.6329157Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-20T22:47:32.6329647Z kill "$MONITOR_SCRIPT_PID" 2024-08-20T22:47:32.6338505Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:47:32.6339012Z env: 2024-08-20T22:47:32.6339289Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:32.6339747Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:47:32.6340475Z DOCKER_CONTAINER_ID: c5602275dd27032945979a763aa9348ea9f7fd583e459e7a824e50356e8b3793 2024-08-20T22:47:32.6341123Z MONITOR_SCRIPT_PID: 182547 2024-08-20T22:47:32.6341479Z ##[endgroup] 2024-08-20T22:47:32.6542475Z Prepare all required actions 2024-08-20T22:47:32.6542939Z Getting action download info 2024-08-20T22:47:32.8368239Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-20T22:47:33.0022479Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-20T22:47:33.0022959Z with: 2024-08-20T22:47:33.0023529Z file-suffix: test-dynamo_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026298293 2024-08-20T22:47:33.0024253Z s3-bucket: gha-artifacts 2024-08-20T22:47:33.0024598Z env: 2024-08-20T22:47:33.0024887Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:33.0025352Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:47:33.0026091Z DOCKER_CONTAINER_ID: c5602275dd27032945979a763aa9348ea9f7fd583e459e7a824e50356e8b3793 2024-08-20T22:47:33.0026723Z ##[endgroup] 2024-08-20T22:47:33.0053308Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-20T22:47:33.0053961Z # Remove any previous test jsons if they exist 2024-08-20T22:47:33.0054475Z rm -f test-jsons-*.zip 2024-08-20T22:47:33.0055001Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-20T22:47:33.0064502Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:47:33.0064998Z env: 2024-08-20T22:47:33.0065275Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:33.0065731Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:47:33.0066619Z DOCKER_CONTAINER_ID: c5602275dd27032945979a763aa9348ea9f7fd583e459e7a824e50356e8b3793 2024-08-20T22:47:33.0067532Z FILE_SUFFIX: test-dynamo_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026298293 2024-08-20T22:47:33.0068198Z ##[endgroup] 2024-08-20T22:47:33.0453226Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-20T22:47:33.0483461Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-20T22:47:33.0484859Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-20T22:47:33.0489789Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-20T22:47:33.0495343Z adding: test/slow_tests.json (deflated 82%) 2024-08-20T22:47:33.0527980Z ##[group]Run # Remove any previous test reports if they exist 2024-08-20T22:47:33.0528625Z # Remove any previous test reports if they exist 2024-08-20T22:47:33.0529160Z rm -f test-reports-*.zip 2024-08-20T22:47:33.0529756Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-20T22:47:33.0538577Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:47:33.0539073Z env: 2024-08-20T22:47:33.0539354Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:33.0539818Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:47:33.0540549Z DOCKER_CONTAINER_ID: c5602275dd27032945979a763aa9348ea9f7fd583e459e7a824e50356e8b3793 2024-08-20T22:47:33.0541458Z FILE_SUFFIX: test-dynamo_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026298293 2024-08-20T22:47:33.0542140Z ##[endgroup] 2024-08-20T22:47:33.0798684Z adding: test/test-reports/inference_huggingface.csv (deflated 76%) 2024-08-20T22:47:33.0799612Z adding: test/test-reports/inference_huggingface_graph_breaks.csv (deflated 86%) 2024-08-20T22:47:33.0800591Z adding: test/test-reports/inference_huggingface_graph_break_deduped.csv (deflated 72%) 2024-08-20T22:47:33.0801505Z adding: test/test-reports/training_huggingface.csv (deflated 74%) 2024-08-20T22:47:33.0808752Z adding: test/test-reports/training_huggingface_graph_breaks.csv (deflated 97%) 2024-08-20T22:47:33.0809736Z adding: test/test-reports/training_huggingface_graph_break_deduped.csv (deflated 74%) 2024-08-20T22:47:33.0842179Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-20T22:47:33.0843313Z # Remove any previous usage logs if they exist 2024-08-20T22:47:33.0843807Z rm -f logs-*.zip 2024-08-20T22:47:33.0844670Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-20T22:47:33.0845462Z # so check to see if the file exists first 2024-08-20T22:47:33.0845968Z if [ -f 'usage_log.txt' ]; then 2024-08-20T22:47:33.0846491Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-20T22:47:33.0846966Z fi 2024-08-20T22:47:33.0847329Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-20T22:47:33.0847901Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-20T22:47:33.0848407Z fi 2024-08-20T22:47:33.0856963Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:47:33.0857459Z env: 2024-08-20T22:47:33.0857740Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:33.0858188Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:47:33.0858961Z DOCKER_CONTAINER_ID: c5602275dd27032945979a763aa9348ea9f7fd583e459e7a824e50356e8b3793 2024-08-20T22:47:33.0860194Z FILE_SUFFIX: test-dynamo_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026298293 2024-08-20T22:47:33.0860904Z ##[endgroup] 2024-08-20T22:47:33.0932800Z adding: usage_log.txt (deflated 92%) 2024-08-20T22:47:33.1021665Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-20T22:47:33.1022384Z # Remove any previous debugging artifacts if they exist 2024-08-20T22:47:33.1022932Z rm -f debug-*.zip 2024-08-20T22:47:33.1023312Z if [ -d 'test/debug' ]; then 2024-08-20T22:47:33.1023812Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-20T22:47:33.1024489Z fi 2024-08-20T22:47:33.1033934Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:47:33.1034439Z env: 2024-08-20T22:47:33.1034706Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:33.1035156Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:47:33.1035877Z DOCKER_CONTAINER_ID: c5602275dd27032945979a763aa9348ea9f7fd583e459e7a824e50356e8b3793 2024-08-20T22:47:33.1036792Z FILE_SUFFIX: test-dynamo_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026298293 2024-08-20T22:47:33.1037453Z ##[endgroup] 2024-08-20T22:47:33.1135992Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:47:33.1136429Z with: 2024-08-20T22:47:33.1136712Z s3-bucket: gha-artifacts 2024-08-20T22:47:33.1137145Z s3-prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T22:47:33.1137621Z retention-days: 14 2024-08-20T22:47:33.1137951Z if-no-files-found: warn 2024-08-20T22:47:33.1138310Z path: test-jsons-*.zip 2024-08-20T22:47:33.1138667Z name: artifact 2024-08-20T22:47:33.1138966Z region: us-east-1 2024-08-20T22:47:33.1139267Z env: 2024-08-20T22:47:33.1139545Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:33.1139997Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:47:33.1140717Z DOCKER_CONTAINER_ID: c5602275dd27032945979a763aa9348ea9f7fd583e459e7a824e50356e8b3793 2024-08-20T22:47:33.1141347Z ##[endgroup] 2024-08-20T22:47:33.4741524Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T22:47:33.4742151Z With the provided path, there will be 1 file uploaded 2024-08-20T22:47:33.4742790Z Uploading to s3 prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T22:47:33.6408894Z Starting upload of test-jsons-test-dynamo_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026298293.zip 2024-08-20T22:47:33.7504387Z Finished upload of test-jsons-test-dynamo_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026298293.zip 2024-08-20T22:47:33.7792451Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:47:33.7792906Z with: 2024-08-20T22:47:33.7793194Z s3-bucket: gha-artifacts 2024-08-20T22:47:33.7793632Z s3-prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T22:47:33.7794112Z retention-days: 14 2024-08-20T22:47:33.7794447Z if-no-files-found: error 2024-08-20T22:47:33.7794817Z path: test-reports-*.zip 2024-08-20T22:47:33.7795171Z name: artifact 2024-08-20T22:47:33.7795471Z region: us-east-1 2024-08-20T22:47:33.7795777Z env: 2024-08-20T22:47:33.7796056Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:33.7796718Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:47:33.7797450Z DOCKER_CONTAINER_ID: c5602275dd27032945979a763aa9348ea9f7fd583e459e7a824e50356e8b3793 2024-08-20T22:47:33.7798088Z ##[endgroup] 2024-08-20T22:47:34.1406251Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T22:47:34.1406939Z With the provided path, there will be 1 file uploaded 2024-08-20T22:47:34.1407678Z Uploading to s3 prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T22:47:34.1464794Z Starting upload of test-reports-test-dynamo_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026298293.zip 2024-08-20T22:47:34.2778284Z Finished upload of test-reports-test-dynamo_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026298293.zip 2024-08-20T22:47:34.3066033Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:47:34.3066478Z with: 2024-08-20T22:47:34.3066763Z s3-bucket: gha-artifacts 2024-08-20T22:47:34.3067196Z s3-prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T22:47:34.3067685Z retention-days: 14 2024-08-20T22:47:34.3068015Z if-no-files-found: ignore 2024-08-20T22:47:34.3068372Z path: logs-*.zip 2024-08-20T22:47:34.3068679Z name: artifact 2024-08-20T22:47:34.3068974Z region: us-east-1 2024-08-20T22:47:34.3069279Z env: 2024-08-20T22:47:34.3069742Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:34.3070197Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:47:34.3070911Z DOCKER_CONTAINER_ID: c5602275dd27032945979a763aa9348ea9f7fd583e459e7a824e50356e8b3793 2024-08-20T22:47:34.3071731Z ##[endgroup] 2024-08-20T22:47:34.6681015Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T22:47:34.6681620Z With the provided path, there will be 1 file uploaded 2024-08-20T22:47:34.6682242Z Uploading to s3 prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T22:47:34.6737964Z Starting upload of logs-test-dynamo_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026298293.zip 2024-08-20T22:47:34.8221102Z Finished upload of logs-test-dynamo_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026298293.zip 2024-08-20T22:47:34.8507408Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:47:34.8507845Z with: 2024-08-20T22:47:34.8508133Z s3-bucket: gha-artifacts 2024-08-20T22:47:34.8508569Z s3-prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T22:47:34.8509045Z retention-days: 14 2024-08-20T22:47:34.8509379Z if-no-files-found: ignore 2024-08-20T22:47:34.8509741Z path: debug-*.zip 2024-08-20T22:47:34.8510071Z name: artifact 2024-08-20T22:47:34.8510369Z region: us-east-1 2024-08-20T22:47:34.8510668Z env: 2024-08-20T22:47:34.8510945Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:34.8511401Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:47:34.8512118Z DOCKER_CONTAINER_ID: c5602275dd27032945979a763aa9348ea9f7fd583e459e7a824e50356e8b3793 2024-08-20T22:47:34.8512748Z ##[endgroup] 2024-08-20T22:47:35.1985098Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-20T22:47:35.2275041Z ##[group]Run # shellcheck disable=SC2156 2024-08-20T22:47:35.2275537Z # shellcheck disable=SC2156 2024-08-20T22:47:35.2276340Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-20T22:47:35.2285600Z shell: /usr/bin/bash -e {0} 2024-08-20T22:47:35.2285954Z env: 2024-08-20T22:47:35.2286232Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:35.2286688Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:47:35.2287436Z DOCKER_CONTAINER_ID: c5602275dd27032945979a763aa9348ea9f7fd583e459e7a824e50356e8b3793 2024-08-20T22:47:35.2288072Z ##[endgroup] 2024-08-20T22:47:35.4898528Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-20T22:47:35.4899106Z with: 2024-08-20T22:47:35.4899370Z env: 2024-08-20T22:47:35.4899652Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:35.4900107Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:47:35.4900835Z DOCKER_CONTAINER_ID: c5602275dd27032945979a763aa9348ea9f7fd583e459e7a824e50356e8b3793 2024-08-20T22:47:35.4901465Z ##[endgroup] 2024-08-20T22:47:35.4920489Z ##[group]Run set -eou pipefail 2024-08-20T22:47:35.4920900Z set -eou pipefail 2024-08-20T22:47:35.4921251Z  2024-08-20T22:47:35.4921765Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-20T22:47:35.4922399Z for _ in $(seq 1440); do 2024-08-20T22:47:35.4922972Z  # Break if no ssh session exists anymore 2024-08-20T22:47:35.4923475Z  if [ "$(who)" = "" ]; then 2024-08-20T22:47:35.4923884Z  break 2024-08-20T22:47:35.4924221Z  fi 2024-08-20T22:47:35.4924517Z  echo "." 2024-08-20T22:47:35.4924844Z  sleep 5 2024-08-20T22:47:35.4925161Z done 2024-08-20T22:47:35.4934042Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:47:35.4934549Z env: 2024-08-20T22:47:35.4934825Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:35.4935280Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:47:35.4935999Z DOCKER_CONTAINER_ID: c5602275dd27032945979a763aa9348ea9f7fd583e459e7a824e50356e8b3793 2024-08-20T22:47:35.4936625Z ##[endgroup] 2024-08-20T22:47:35.4965222Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-20T22:47:35.5045073Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:47:35.5046041Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:47:35.5046643Z # shellcheck disable=SC2046 2024-08-20T22:47:35.5047129Z docker stop $(docker ps -q) || true 2024-08-20T22:47:35.5047617Z # Prune all of the docker images 2024-08-20T22:47:35.5048072Z docker system prune -af 2024-08-20T22:47:35.5056916Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:47:35.5057420Z env: 2024-08-20T22:47:35.5057697Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:47:35.5058147Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:47:35.5058867Z DOCKER_CONTAINER_ID: c5602275dd27032945979a763aa9348ea9f7fd583e459e7a824e50356e8b3793 2024-08-20T22:47:35.5059493Z ##[endgroup] 2024-08-20T22:47:36.0454783Z c5602275dd27 2024-08-20T22:47:36.8065683Z Deleted Containers: 2024-08-20T22:47:36.8066237Z c5602275dd27032945979a763aa9348ea9f7fd583e459e7a824e50356e8b3793 2024-08-20T22:47:36.8066658Z 2024-08-20T22:47:46.0444170Z Deleted Images: 2024-08-20T22:47:46.0446301Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:47:46.0448774Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T22:47:46.0450749Z deleted: sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb 2024-08-20T22:47:46.0451865Z deleted: sha256:c2ce10014dd98f6abbfbf2265f9780a8204c2a7a27463e61ee565573a23d141d 2024-08-20T22:47:46.0452832Z deleted: sha256:ae249c0e6e6013bd6ffea2ca0b0160c015c9ebf7c03070bc2f983a9ff2415bf5 2024-08-20T22:47:46.0453949Z deleted: sha256:8a73883c1067c0e29ff128f553518682b47ed9707d2f35e7d5138e2caf029d61 2024-08-20T22:47:46.0454803Z deleted: sha256:9c040db729fa084584aaf5573082408a2196bc64e5952270c79dc610ed4680b6 2024-08-20T22:47:46.0455631Z deleted: sha256:17311eeeb8b71b16699880c0f646e185107496c4d41da5d23370603768ec030c 2024-08-20T22:47:46.0456549Z deleted: sha256:07cc10bece8501b1c8035d4385ad9e9867dd6e92fa32218a4dcb3e42af7e0411 2024-08-20T22:47:46.0457709Z deleted: sha256:f3e32d05099d1a3115a4722bc2e94a48752c3e965446773673808252671bb967 2024-08-20T22:47:46.0458545Z deleted: sha256:be7c3f41475e3e72ffe3ac2f92ab2f4cf49d5496b2aae738942ac3bb687d742c 2024-08-20T22:47:46.0459389Z deleted: sha256:4fcac490be825129b468cf458089f1fcf8fd56afc44ef4ab27c2eff41ab1ac90 2024-08-20T22:47:46.0460915Z deleted: sha256:31c66d80a2a4792897f267d363c0cf84fb821c5d4a275ab1636f778fd0284c08 2024-08-20T22:47:46.0461748Z deleted: sha256:8a8bcb6085e7e438e5533e8a5613e2ae5888543853568d023af3e2623a1593a9 2024-08-20T22:47:46.0462578Z deleted: sha256:2f6cb60029c79d11567b738f5bdbfb9cb1838cc0f194c4a5e6ae685178d2ed34 2024-08-20T22:47:46.0463430Z deleted: sha256:e546391804d3313f558e1ac16e93cb29a417a92e4eccaf0d4bd12af3bbcf39e3 2024-08-20T22:47:46.0464369Z deleted: sha256:1623b2aa16f2bdc6b572f6799498d9a92fa8114d07c440599b17aea72e22f8b1 2024-08-20T22:47:46.0465493Z deleted: sha256:c0386ca58b7cb7afee015b7e58d2f70525eaf73d6da2cc63868b20e14b1f9497 2024-08-20T22:47:46.0466401Z deleted: sha256:dcbf534ff289186e246531925fed90f72516eb3d63663bad234d611cdc025d62 2024-08-20T22:47:46.0467231Z deleted: sha256:0bab537af579f56c9cd51c7073fd53635fffb424b363236e0de38a7dd0e60ae8 2024-08-20T22:47:46.0468051Z deleted: sha256:d85a852ae1ee027e2fc17240ed68b87957b0a6d210c9119752d8979a0dcd55cf 2024-08-20T22:47:46.0468870Z deleted: sha256:f3821d761b555362e44cf9a1447d8fd2d95b37d54aea67c7519cd9e5db0c4d7e 2024-08-20T22:47:46.0469715Z deleted: sha256:c05e84a545db5dfcb0847c73d2b0f4749fa67aabce05705632ff7f76c84d4688 2024-08-20T22:47:46.0470559Z deleted: sha256:27054eae7ca67b2bd5ca0fd3f4488bf5c5f78e95487d8c45616f3d753d48a8b5 2024-08-20T22:47:46.0471383Z deleted: sha256:9804139f6ae97796aaa993fe4b89ea0cf111eb4d30a79f4c87c339c2e1e04fa0 2024-08-20T22:47:46.0472217Z deleted: sha256:ad6ae2fc594aaa359ff4e287323836570454eb24dd162930655e414d1d77c6dd 2024-08-20T22:47:46.0473331Z deleted: sha256:9cfbb36e328af52a248853a46aab88e266c09e2b44a03e0a42194a031b9b9453 2024-08-20T22:47:46.0474172Z deleted: sha256:a8a842bd7f37b8d50b6d12d6f95f46bbba57953db466fcbde9ba522625f024ef 2024-08-20T22:47:46.0475010Z deleted: sha256:da9738654b3c86f89ebeed2560f1bae093f1219f1eff90b0394b7f211ecabd04 2024-08-20T22:47:46.0475846Z deleted: sha256:cc683bf0cb245394e0f568984b2dd4fce1b6a3a7fe7c44734bae1aa1caaa3251 2024-08-20T22:47:46.0476682Z deleted: sha256:965f220e71c7d1ec19aaf41b1529ab287beb9d86eb8482c4c14094485ab34475 2024-08-20T22:47:46.0477517Z deleted: sha256:da32ecec41a73ce7dd29464f90c7951510a18cd378753659a78cd7fabd10dc36 2024-08-20T22:47:46.0478354Z deleted: sha256:d828d187cee9ad48dbc3702fb61d359619dd8a1ed582878680b51686bf117e5a 2024-08-20T22:47:46.0479172Z deleted: sha256:712111f4b12ff8e2787597d5fdcb043bd0824e55ddd97f262ef550506ca195f0 2024-08-20T22:47:46.0479992Z deleted: sha256:a064eea405958ad14dc6c794e5df9fd1d4515424272801ec559d45a59317309a 2024-08-20T22:47:46.0480831Z deleted: sha256:e124382d136bd5e9a8b409ca79c1f1ef76458deaf444b1b2060d3fa99dbeb506 2024-08-20T22:47:46.0481657Z deleted: sha256:6e10fe757cc190a1b31b47010f4f7610e02cb1b4b438f190dcd301aa4fe2278e 2024-08-20T22:47:46.0482475Z deleted: sha256:4cea352e10f46ca4729f96ac6c4600fc53d4c36e7658c2e0f19f4f309cd4a2a7 2024-08-20T22:47:46.0483417Z deleted: sha256:518bba00fe1d0933053066d21cac0235c9fb485abbc41557ec184b293095c924 2024-08-20T22:47:46.0484258Z deleted: sha256:a64f9ca42e9f27ff2690c15a5c84312e6efaccad45668d9b496cb542165cfb5a 2024-08-20T22:47:46.0485091Z deleted: sha256:096ea0fa14a6c420658f5276b8a8f5d264c0de7e18161ceda2cfc2f4a92cd9bf 2024-08-20T22:47:46.0485912Z deleted: sha256:6f91c94ce6e3f8034c91c66ee25c66a3d85dba9d813b749bd6d2000942a99edc 2024-08-20T22:47:46.0486797Z deleted: sha256:8f8d255eeae546cead8f9a94d237fa57a99940c0f5d51bc29a9d41c7df9cffd9 2024-08-20T22:47:46.0487642Z deleted: sha256:fe05276f4a49f91eef56bfbc3da87cfff5eb116a827fa2d30efe42ce1c134b40 2024-08-20T22:47:46.0488481Z deleted: sha256:5f83cd75d0669c813d4f58978c92e14305cd8f6a07eb9a60665a583460fa7b59 2024-08-20T22:47:46.0489292Z deleted: sha256:8f7ab6366169d39d8953129f650f1927d2fb1328b3f11e2fd9076adb5f46b93a 2024-08-20T22:47:46.0490405Z deleted: sha256:4146cfd5dba0f5308954749f98596b14a1d8f6d23bd93f7ad86c6c3d72239328 2024-08-20T22:47:46.0491495Z deleted: sha256:c6d83cb9ccc606ff63bb3c4d5a5e1ff1450c619b5aae494303a94ae25971cca0 2024-08-20T22:47:46.0492338Z deleted: sha256:71e037e0a3e0963c4565534a7387c082afcb04475d3ac629acf7a0ab2f49fadc 2024-08-20T22:47:46.0493153Z deleted: sha256:4b31135c564686bdaf23262b901237880aea635655721cc7e8c95dc42b05e06d 2024-08-20T22:47:46.0493965Z deleted: sha256:d09bd5c2f35cedcb846543941ee64ce1f684b52e4285696c93b3fb2032da8579 2024-08-20T22:47:46.0494812Z deleted: sha256:4de7fb7d4fc22212aae5fc1ceb637526916d0cced240d7c93d5edd6b0ea28013 2024-08-20T22:47:46.0495648Z deleted: sha256:69890e3e82746e10aff2f1084f11afb4b95a4a6c26b30f64b7001bc0a5fe32bf 2024-08-20T22:47:46.0496461Z deleted: sha256:0156df53b3224e645336ea52b4bb816f1e8d8c9b591748cd645e5e5649fa507b 2024-08-20T22:47:46.0497313Z deleted: sha256:0664624cb9cc4a386f31bf49919a8d8a11d9fea428d05585d9e6228f754bc7cd 2024-08-20T22:47:46.0498216Z deleted: sha256:8c2ddd5fa535c768c058a23f5cc4b6c64cdd71bb7458491282aabc7f7bb94d69 2024-08-20T22:47:46.0499057Z deleted: sha256:8dd03dbe0e1c5f47dd333074dff4bd00415f2f63727a65bf17866774e2ebbd0d 2024-08-20T22:47:46.0499889Z deleted: sha256:3c56b63e094bd18fad9da7a541d537210bca3b730c590d3641213259f6908d6f 2024-08-20T22:47:46.0500702Z deleted: sha256:51484fef07bd1266d4d6ff3d767751172660fc4cc6a1ffe883523a63c985e31e 2024-08-20T22:47:46.0501524Z deleted: sha256:03e83c94023982ab7dd8dc65a6df2024e342f1b760c91ec7a680ee45824869c7 2024-08-20T22:47:46.0502351Z deleted: sha256:58440ab3e8175b5de94a76485c8689a6dc186015475ebf4b386ebba309c5da34 2024-08-20T22:47:46.0503177Z deleted: sha256:9a980f3383e8aa511f0ef68982cb6dfc53b38d1ac8c3d673cb3ba3d45158c91e 2024-08-20T22:47:46.0504009Z deleted: sha256:cb0269c9cf1c47b6c4188ca7be4db261b5d3fc4b4a5153629e222dba7b1f9f98 2024-08-20T22:47:46.0504976Z deleted: sha256:41f1ba870cdffc05e0bf8d87b640fb9d4c153d7f4f3b775f37f94e9f3d584aad 2024-08-20T22:47:46.0505811Z deleted: sha256:7b502ab352f48e8b6eb5e6824a29f308952a5ee7c7d987562e1df7e47639d7f1 2024-08-20T22:47:46.0506627Z deleted: sha256:6ba5217e18738660e6ca8b7a7ec88845e70b0fd781301520bf46c71d99df3144 2024-08-20T22:47:46.0507447Z deleted: sha256:dd5a1c233594780163726bdbbf5de00f7b6b8dca1dace8a001340714c54baedb 2024-08-20T22:47:46.0508269Z deleted: sha256:c27f0db27da73e01f67f6cb78d10f355e350d829479ba4351d82055bb29f6597 2024-08-20T22:47:46.0509099Z deleted: sha256:4e3ce3c854f35c08be3dd95e0d6e718512a885e4614291d7d8b7cd23f48b7dd8 2024-08-20T22:47:46.0509935Z deleted: sha256:c2d6fb088bbec16e60cf0ad74b016f7287c92e659ec2db01271bc309b4dc6d83 2024-08-20T22:47:46.0510774Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-08-20T22:47:46.0511596Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-08-20T22:47:46.0512413Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-08-20T22:47:46.0513232Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-08-20T22:47:46.0514065Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-08-20T22:47:46.0514879Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-08-20T22:47:46.0515722Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-08-20T22:47:46.0516578Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-08-20T22:47:46.0517470Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-08-20T22:47:46.0518311Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-08-20T22:47:46.0519135Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-08-20T22:47:46.0519635Z 2024-08-20T22:47:46.0519787Z Total reclaimed space: 39.92GB 2024-08-20T22:47:46.0586844Z Post job cleanup. 2024-08-20T22:47:46.0637919Z Post job cleanup. 2024-08-20T22:47:46.1518562Z [command]/usr/bin/git version 2024-08-20T22:47:46.1570827Z git version 2.40.1 2024-08-20T22:47:46.1602025Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/e4cd2366-7c72-4de6-89f2-f53681f0e87a/.gitconfig' 2024-08-20T22:47:46.1613008Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/e4cd2366-7c72-4de6-89f2-f53681f0e87a' before making global git config changes 2024-08-20T22:47:46.1614281Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T22:47:46.1617602Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:47:46.1671655Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T22:47:46.1716162Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T22:47:46.2091356Z Entering 'android/libs/fbjni' 2024-08-20T22:47:46.2161569Z Entering 'third_party/FP16' 2024-08-20T22:47:46.2230235Z Entering 'third_party/FXdiv' 2024-08-20T22:47:46.2299505Z Entering 'third_party/NNPACK' 2024-08-20T22:47:46.2369533Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:47:46.2438629Z Entering 'third_party/XNNPACK' 2024-08-20T22:47:46.2525044Z Entering 'third_party/benchmark' 2024-08-20T22:47:46.2593975Z Entering 'third_party/cpp-httplib' 2024-08-20T22:47:46.2662208Z Entering 'third_party/cpuinfo' 2024-08-20T22:47:46.2730404Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:47:46.2800215Z Entering 'third_party/cutlass' 2024-08-20T22:47:46.2875437Z Entering 'third_party/eigen' 2024-08-20T22:47:46.2945171Z Entering 'third_party/fbgemm' 2024-08-20T22:47:46.3017510Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:47:46.3087054Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:47:46.3153341Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:47:46.3226066Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:47:46.3297288Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:47:46.3365507Z Entering 'third_party/flatbuffers' 2024-08-20T22:47:46.3438405Z Entering 'third_party/fmt' 2024-08-20T22:47:46.3510234Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:47:46.3579841Z Entering 'third_party/gloo' 2024-08-20T22:47:46.3649300Z Entering 'third_party/googletest' 2024-08-20T22:47:46.3719052Z Entering 'third_party/ideep' 2024-08-20T22:47:46.3784381Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:47:46.3859660Z Entering 'third_party/ittapi' 2024-08-20T22:47:46.3930035Z Entering 'third_party/kineto' 2024-08-20T22:47:46.3996899Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:47:46.4063089Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:47:46.4134821Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:47:46.4201573Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:47:46.4270403Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:47:46.4336165Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:47:46.4408571Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:47:46.4481596Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:47:46.4549288Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:47:46.4617939Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:47:46.4689176Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:47:46.4755826Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:47:46.4826727Z Entering 'third_party/mimalloc' 2024-08-20T22:47:46.4904924Z Entering 'third_party/nccl/nccl' 2024-08-20T22:47:46.4969599Z Entering 'third_party/nlohmann' 2024-08-20T22:47:46.5038571Z Entering 'third_party/onnx' 2024-08-20T22:47:46.5119801Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:47:46.5188481Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:47:46.5261926Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:47:46.5332319Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:47:46.5398879Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:47:46.5465106Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:47:46.5530599Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:47:46.5598074Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:47:46.5664648Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:47:46.5729989Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:47:46.5794613Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:47:46.5867307Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:47:46.5937122Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:47:46.6025980Z Entering 'third_party/pocketfft' 2024-08-20T22:47:46.6093823Z Entering 'third_party/protobuf' 2024-08-20T22:47:46.6163063Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:47:46.6229907Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:47:46.6301415Z Entering 'third_party/psimd' 2024-08-20T22:47:46.6369924Z Entering 'third_party/pthreadpool' 2024-08-20T22:47:46.6441937Z Entering 'third_party/pybind11' 2024-08-20T22:47:46.6511792Z Entering 'third_party/python-peachpy' 2024-08-20T22:47:46.6579801Z Entering 'third_party/sleef' 2024-08-20T22:47:46.6646983Z Entering 'third_party/tensorpipe' 2024-08-20T22:47:46.6713799Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:47:46.6780677Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:47:46.6846447Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:47:46.6912199Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:47:46.6975777Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:47:46.7071459Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T22:47:46.7107914Z http.https://github.com/.extraheader 2024-08-20T22:47:46.7118638Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-20T22:47:46.7169482Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T22:47:46.7532121Z Entering 'android/libs/fbjni' 2024-08-20T22:47:46.7575671Z http.https://github.com/.extraheader 2024-08-20T22:47:46.7620697Z Entering 'third_party/FP16' 2024-08-20T22:47:46.7664668Z http.https://github.com/.extraheader 2024-08-20T22:47:46.7708640Z Entering 'third_party/FXdiv' 2024-08-20T22:47:46.7752527Z http.https://github.com/.extraheader 2024-08-20T22:47:46.7796524Z Entering 'third_party/NNPACK' 2024-08-20T22:47:46.7840656Z http.https://github.com/.extraheader 2024-08-20T22:47:46.7885422Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:47:46.7933868Z http.https://github.com/.extraheader 2024-08-20T22:47:46.7978606Z Entering 'third_party/XNNPACK' 2024-08-20T22:47:46.8022577Z http.https://github.com/.extraheader 2024-08-20T22:47:46.8083282Z Entering 'third_party/benchmark' 2024-08-20T22:47:46.8127640Z http.https://github.com/.extraheader 2024-08-20T22:47:46.8171866Z Entering 'third_party/cpp-httplib' 2024-08-20T22:47:46.8215892Z http.https://github.com/.extraheader 2024-08-20T22:47:46.8259468Z Entering 'third_party/cpuinfo' 2024-08-20T22:47:46.8303947Z http.https://github.com/.extraheader 2024-08-20T22:47:46.8347812Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:47:46.8392118Z http.https://github.com/.extraheader 2024-08-20T22:47:46.8435293Z Entering 'third_party/cutlass' 2024-08-20T22:47:46.8480669Z http.https://github.com/.extraheader 2024-08-20T22:47:46.8532480Z Entering 'third_party/eigen' 2024-08-20T22:47:46.8576640Z http.https://github.com/.extraheader 2024-08-20T22:47:46.8621600Z Entering 'third_party/fbgemm' 2024-08-20T22:47:46.8665728Z http.https://github.com/.extraheader 2024-08-20T22:47:46.8709545Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:47:46.8751417Z http.https://github.com/.extraheader 2024-08-20T22:47:46.8794433Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:47:46.8841929Z http.https://github.com/.extraheader 2024-08-20T22:47:46.8885579Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:47:46.8927423Z http.https://github.com/.extraheader 2024-08-20T22:47:46.8978858Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:47:46.9021191Z http.https://github.com/.extraheader 2024-08-20T22:47:46.9066822Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:47:46.9109267Z http.https://github.com/.extraheader 2024-08-20T22:47:46.9154992Z Entering 'third_party/flatbuffers' 2024-08-20T22:47:46.9205448Z http.https://github.com/.extraheader 2024-08-20T22:47:46.9251412Z Entering 'third_party/fmt' 2024-08-20T22:47:46.9295383Z http.https://github.com/.extraheader 2024-08-20T22:47:46.9338830Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:47:46.9383619Z http.https://github.com/.extraheader 2024-08-20T22:47:46.9428100Z Entering 'third_party/gloo' 2024-08-20T22:47:46.9473010Z http.https://github.com/.extraheader 2024-08-20T22:47:46.9516736Z Entering 'third_party/googletest' 2024-08-20T22:47:46.9560756Z http.https://github.com/.extraheader 2024-08-20T22:47:46.9604866Z Entering 'third_party/ideep' 2024-08-20T22:47:46.9647815Z http.https://github.com/.extraheader 2024-08-20T22:47:46.9690142Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:47:46.9732051Z http.https://github.com/.extraheader 2024-08-20T22:47:46.9784744Z Entering 'third_party/ittapi' 2024-08-20T22:47:46.9829046Z http.https://github.com/.extraheader 2024-08-20T22:47:46.9874938Z Entering 'third_party/kineto' 2024-08-20T22:47:46.9918964Z http.https://github.com/.extraheader 2024-08-20T22:47:46.9964117Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:47:47.0005653Z http.https://github.com/.extraheader 2024-08-20T22:47:47.0048882Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:47:47.0092667Z http.https://github.com/.extraheader 2024-08-20T22:47:47.0138346Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:47:47.0182055Z http.https://github.com/.extraheader 2024-08-20T22:47:47.0226302Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:47:47.0269584Z http.https://github.com/.extraheader 2024-08-20T22:47:47.0314760Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:47:47.0358808Z http.https://github.com/.extraheader 2024-08-20T22:47:47.0405479Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:47:47.0448605Z http.https://github.com/.extraheader 2024-08-20T22:47:47.0496684Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:47:47.0539466Z http.https://github.com/.extraheader 2024-08-20T22:47:47.0584605Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:47:47.0626233Z http.https://github.com/.extraheader 2024-08-20T22:47:47.0674807Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:47:47.0716953Z http.https://github.com/.extraheader 2024-08-20T22:47:47.0763698Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:47:47.0805929Z http.https://github.com/.extraheader 2024-08-20T22:47:47.0853664Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:47:47.0896468Z http.https://github.com/.extraheader 2024-08-20T22:47:47.0938080Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:47:47.0979611Z http.https://github.com/.extraheader 2024-08-20T22:47:47.1027104Z Entering 'third_party/mimalloc' 2024-08-20T22:47:47.1073439Z http.https://github.com/.extraheader 2024-08-20T22:47:47.1119356Z Entering 'third_party/nccl/nccl' 2024-08-20T22:47:47.1164398Z http.https://github.com/.extraheader 2024-08-20T22:47:47.1207174Z Entering 'third_party/nlohmann' 2024-08-20T22:47:47.1253014Z http.https://github.com/.extraheader 2024-08-20T22:47:47.1298977Z Entering 'third_party/onnx' 2024-08-20T22:47:47.1342946Z http.https://github.com/.extraheader 2024-08-20T22:47:47.1402230Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:47:47.1447117Z http.https://github.com/.extraheader 2024-08-20T22:47:47.1491907Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:47:47.1539090Z http.https://github.com/.extraheader 2024-08-20T22:47:47.1589656Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:47:47.1636221Z http.https://github.com/.extraheader 2024-08-20T22:47:47.1682136Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:47:47.1728615Z http.https://github.com/.extraheader 2024-08-20T22:47:47.1771958Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:47:47.1814718Z http.https://github.com/.extraheader 2024-08-20T22:47:47.1857273Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:47:47.1900741Z http.https://github.com/.extraheader 2024-08-20T22:47:47.1943168Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:47:47.1986048Z http.https://github.com/.extraheader 2024-08-20T22:47:47.2031172Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:47:47.2073460Z http.https://github.com/.extraheader 2024-08-20T22:47:47.2116365Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:47:47.2168163Z http.https://github.com/.extraheader 2024-08-20T22:47:47.2210658Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:47:47.2253051Z http.https://github.com/.extraheader 2024-08-20T22:47:47.2295899Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:47:47.2338988Z http.https://github.com/.extraheader 2024-08-20T22:47:47.2385820Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:47:47.2428703Z http.https://github.com/.extraheader 2024-08-20T22:47:47.2476204Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:47:47.2518469Z http.https://github.com/.extraheader 2024-08-20T22:47:47.2582633Z Entering 'third_party/pocketfft' 2024-08-20T22:47:47.2626847Z http.https://github.com/.extraheader 2024-08-20T22:47:47.2674719Z Entering 'third_party/protobuf' 2024-08-20T22:47:47.2718979Z http.https://github.com/.extraheader 2024-08-20T22:47:47.2766032Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:47:47.2808155Z http.https://github.com/.extraheader 2024-08-20T22:47:47.2851546Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:47:47.2898611Z http.https://github.com/.extraheader 2024-08-20T22:47:47.2945941Z Entering 'third_party/psimd' 2024-08-20T22:47:47.2990958Z http.https://github.com/.extraheader 2024-08-20T22:47:47.3035139Z Entering 'third_party/pthreadpool' 2024-08-20T22:47:47.3079988Z http.https://github.com/.extraheader 2024-08-20T22:47:47.3123928Z Entering 'third_party/pybind11' 2024-08-20T22:47:47.3168870Z http.https://github.com/.extraheader 2024-08-20T22:47:47.3212977Z Entering 'third_party/python-peachpy' 2024-08-20T22:47:47.3257095Z http.https://github.com/.extraheader 2024-08-20T22:47:47.3301449Z Entering 'third_party/sleef' 2024-08-20T22:47:47.3345315Z http.https://github.com/.extraheader 2024-08-20T22:47:47.3390141Z Entering 'third_party/tensorpipe' 2024-08-20T22:47:47.3433886Z http.https://github.com/.extraheader 2024-08-20T22:47:47.3476183Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:47:47.3519080Z http.https://github.com/.extraheader 2024-08-20T22:47:47.3563569Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:47:47.3605704Z http.https://github.com/.extraheader 2024-08-20T22:47:47.3648489Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:47:47.3691636Z http.https://github.com/.extraheader 2024-08-20T22:47:47.3735269Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:47:47.3778251Z http.https://github.com/.extraheader 2024-08-20T22:47:47.3819530Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:47:47.3866828Z http.https://github.com/.extraheader 2024-08-20T22:47:47.4031574Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-20T22:47:47.4052643Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-20T22:47:47.4061372Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:47:47.4062041Z ##[endgroup] 2024-08-20T22:47:54.7952075Z Cleaning up orphan processes