2024-06-26T07:02:33.1987954Z Current runner version: '2.317.0' 2024-06-26T07:02:33.1994182Z Runner name: 'i-0a96abc6f4fd8966c' 2024-06-26T07:02:33.1994951Z Runner group name: 'Default' 2024-06-26T07:02:33.1995799Z Machine name: 'ip-10-0-76-149' 2024-06-26T07:02:33.2000560Z ##[group]GITHUB_TOKEN Permissions 2024-06-26T07:02:33.2002486Z Actions: read 2024-06-26T07:02:33.2002994Z Attestations: read 2024-06-26T07:02:33.2003472Z Checks: read 2024-06-26T07:02:33.2004132Z Contents: read 2024-06-26T07:02:33.2004663Z Deployments: read 2024-06-26T07:02:33.2005178Z Discussions: read 2024-06-26T07:02:33.2005656Z Issues: read 2024-06-26T07:02:33.2006252Z Metadata: read 2024-06-26T07:02:33.2006739Z Packages: read 2024-06-26T07:02:33.2007208Z Pages: read 2024-06-26T07:02:33.2007722Z PullRequests: read 2024-06-26T07:02:33.2008249Z RepositoryProjects: read 2024-06-26T07:02:33.2008846Z SecurityEvents: read 2024-06-26T07:02:33.2009342Z Statuses: read 2024-06-26T07:02:33.2009816Z ##[endgroup] 2024-06-26T07:02:33.2013257Z Secret source: Actions 2024-06-26T07:02:33.2014035Z Prepare workflow directory 2024-06-26T07:02:33.5637297Z Prepare all required actions 2024-06-26T07:02:33.5822852Z Getting action download info 2024-06-26T07:02:33.7890144Z Download action repository 'pytorch/test-infra@main' (SHA:43a2ce341cc31288e9a38b65ce600a7f43021bd5) 2024-06-26T07:02:34.0964933Z Download action repository 'pytorch/pytorch@main' (SHA:c422a9549d2b320b47a9391f6cbcf592b2876c88) 2024-06-26T07:02:36.8626812Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-06-26T07:02:36.9937856Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-06-26T07:02:37.3546152Z Getting action download info 2024-06-26T07:02:37.4507223Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-06-26T07:02:37.6330084Z Getting action download info 2024-06-26T07:02:37.7618746Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-06-26T07:02:37.9680558Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/129470 (b8c4c54d347aa776934c60784e35936878ef18dc) 2024-06-26T07:02:37.9682823Z ##[group] Inputs 2024-06-26T07:02:37.9683312Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:02:37.9690363Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:02:37.9697994Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:37.9699158Z sync-tag: 2024-06-26T07:02:37.9700035Z timeout-minutes: 240 2024-06-26T07:02:37.9700421Z use-gha: 2024-06-26T07:02:37.9700706Z dashboard-tag: 2024-06-26T07:02:37.9701023Z s3-bucket: gha-artifacts 2024-06-26T07:02:37.9701373Z aws-role-to-assume: 2024-06-26T07:02:37.9701726Z ##[endgroup] 2024-06-26T07:02:37.9702786Z Complete job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:02:38.0277313Z A job started hook has been configured by the self-hosted runner administrator 2024-06-26T07:02:38.0417296Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T07:02:38.0428015Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:38.0428576Z ##[endgroup] 2024-06-26T07:02:38.9221324Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-06-26T07:02:38.9221871Z with: 2024-06-26T07:02:38.9222566Z github-secret: *** 2024-06-26T07:02:38.9223511Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-06-26T07:02:38.9224498Z activate-with-label: false 2024-06-26T07:02:38.9224870Z label: with-ssh 2024-06-26T07:02:38.9225195Z remove-existing-keys: true 2024-06-26T07:02:38.9225560Z fail-silently: true 2024-06-26T07:02:38.9225873Z env: 2024-06-26T07:02:38.9226149Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:38.9226490Z ##[endgroup] 2024-06-26T07:02:39.0103041Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-06-26T07:02:39.0104066Z ciflow reference detected, attempting to extract PR number 2024-06-26T07:02:39.3369448Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-06-26T07:02:39.4137377Z No SSH keys found for user pytorch-bot[bot] 2024-06-26T07:02:39.4138108Z Grabbing public ssh keys from https://github.com/leslie-fang-intel.keys 2024-06-26T07:02:39.4669523Z No SSH keys found for user leslie-fang-intel 2024-06-26T07:02:39.4772049Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-06-26T07:02:39.4772606Z with: 2024-06-26T07:02:39.4772927Z submodules: recursive 2024-06-26T07:02:39.4773292Z fetch-depth: 0 2024-06-26T07:02:39.4773619Z env: 2024-06-26T07:02:39.4773994Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:39.4774368Z ##[endgroup] 2024-06-26T07:02:39.4967912Z ##[group]Run retry () { 2024-06-26T07:02:39.4968389Z retry () { 2024-06-26T07:02:39.4968948Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-06-26T07:02:39.4969539Z } 2024-06-26T07:02:39.4969898Z echo "${GITHUB_WORKSPACE}" 2024-06-26T07:02:39.4970402Z if [ -z "${NO_SUDO}" ]; then 2024-06-26T07:02:39.4970986Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:02:39.4971477Z else 2024-06-26T07:02:39.4971863Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:02:39.4972349Z fi 2024-06-26T07:02:39.4972759Z mkdir "${GITHUB_WORKSPACE}" 2024-06-26T07:02:39.4980835Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:39.4981385Z env: 2024-06-26T07:02:39.4981745Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:39.4982138Z NO_SUDO: 2024-06-26T07:02:39.5010379Z ##[endgroup] 2024-06-26T07:02:39.5033212Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:02:42.4983890Z ##[group]Run malfet/checkout@silent-checkout 2024-06-26T07:02:42.4984335Z with: 2024-06-26T07:02:42.4984653Z ref: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:02:42.4985094Z fetch-depth: 0 2024-06-26T07:02:42.4985409Z submodules: recursive 2024-06-26T07:02:42.4985752Z quiet-checkout: true 2024-06-26T07:02:42.4986222Z repository: pytorch/pytorch 2024-06-26T07:02:42.4986698Z token: *** 2024-06-26T07:02:42.4986987Z ssh-strict: true 2024-06-26T07:02:42.4987317Z persist-credentials: true 2024-06-26T07:02:42.4987737Z clean: true 2024-06-26T07:02:42.4988073Z sparse-checkout-cone-mode: true 2024-06-26T07:02:42.4988456Z lfs: false 2024-06-26T07:02:42.4988744Z set-safe-directory: true 2024-06-26T07:02:42.4989082Z env: 2024-06-26T07:02:42.4989349Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:42.4989691Z ##[endgroup] 2024-06-26T07:02:42.5906308Z Syncing repository: pytorch/pytorch 2024-06-26T07:02:42.5908082Z ##[group]Getting Git version info 2024-06-26T07:02:42.5908842Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:02:42.5909686Z [command]/usr/bin/git version 2024-06-26T07:02:42.5910052Z git version 2.40.1 2024-06-26T07:02:42.5911386Z ##[endgroup] 2024-06-26T07:02:42.5922429Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/a14722b8-87e9-455d-8766-dd3498473b82/.gitconfig' 2024-06-26T07:02:42.5924168Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/a14722b8-87e9-455d-8766-dd3498473b82' before making global git config changes 2024-06-26T07:02:42.5925386Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T07:02:42.5926468Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:02:42.5946694Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:02:42.5950458Z ##[group]Initializing the repository 2024-06-26T07:02:42.5953125Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:02:42.5977587Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-06-26T07:02:42.5978448Z hint: is subject to change. To configure the initial branch name to use in all 2024-06-26T07:02:42.5979240Z hint: of your new repositories, which will suppress this warning, call: 2024-06-26T07:02:42.5979974Z hint: 2024-06-26T07:02:42.5980410Z hint: git config --global init.defaultBranch 2024-06-26T07:02:42.5980884Z hint: 2024-06-26T07:02:42.5981575Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-06-26T07:02:42.5982419Z hint: 'development'. The just-created branch can be renamed via this command: 2024-06-26T07:02:42.5982990Z hint: 2024-06-26T07:02:42.5983304Z hint: git branch -m 2024-06-26T07:02:42.5984032Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-06-26T07:02:42.5985945Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-06-26T07:02:42.6012676Z ##[endgroup] 2024-06-26T07:02:42.6013272Z ##[group]Disabling automatic garbage collection 2024-06-26T07:02:42.6015260Z [command]/usr/bin/git config --local gc.auto 0 2024-06-26T07:02:42.6038873Z ##[endgroup] 2024-06-26T07:02:42.6039434Z ##[group]Setting up auth 2024-06-26T07:02:42.6044085Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T07:02:42.6069597Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T07:02:42.6260384Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T07:02:42.6281366Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T07:02:42.6476819Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:02:42.6520388Z ##[endgroup] 2024-06-26T07:02:42.6520949Z ##[group]Fetching the repository 2024-06-26T07:02:42.6527145Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-06-26T07:02:44.8038976Z remote: Enumerating objects: 987392 2024-06-26T07:02:44.8039517Z remote: Enumerating objects: 990209, done. 2024-06-26T07:02:44.8044622Z remote: Counting objects: 0% (1/2817) 2024-06-26T07:02:44.8045490Z remote: Counting objects: 1% (29/2817) 2024-06-26T07:02:44.8046349Z remote: Counting objects: 2% (57/2817) 2024-06-26T07:02:44.8047113Z remote: Counting objects: 3% (85/2817) 2024-06-26T07:02:44.8047866Z remote: Counting objects: 4% (113/2817) 2024-06-26T07:02:44.8048778Z remote: Counting objects: 5% (141/2817) 2024-06-26T07:02:44.8049542Z remote: Counting objects: 6% (170/2817) 2024-06-26T07:02:44.8050236Z remote: Counting objects: 7% (198/2817) 2024-06-26T07:02:44.8050914Z remote: Counting objects: 8% (226/2817) 2024-06-26T07:02:44.8051447Z remote: Counting objects: 9% (254/2817) 2024-06-26T07:02:44.8052015Z remote: Counting objects: 10% (282/2817) 2024-06-26T07:02:44.8052508Z remote: Counting objects: 11% (310/2817) 2024-06-26T07:02:44.8053167Z remote: Counting objects: 12% (339/2817) 2024-06-26T07:02:44.8053714Z remote: Counting objects: 13% (367/2817) 2024-06-26T07:02:44.8054208Z remote: Counting objects: 14% (395/2817) 2024-06-26T07:02:44.8054699Z remote: Counting objects: 15% (423/2817) 2024-06-26T07:02:44.8055189Z remote: Counting objects: 16% (451/2817) 2024-06-26T07:02:44.8055831Z remote: Counting objects: 17% (479/2817) 2024-06-26T07:02:44.8056363Z remote: Counting objects: 18% (508/2817) 2024-06-26T07:02:44.8056851Z remote: Counting objects: 19% (536/2817) 2024-06-26T07:02:44.8057341Z remote: Counting objects: 20% (564/2817) 2024-06-26T07:02:44.8057827Z remote: Counting objects: 21% (592/2817) 2024-06-26T07:02:44.8058473Z remote: Counting objects: 22% (620/2817) 2024-06-26T07:02:44.8059012Z remote: Counting objects: 23% (648/2817) 2024-06-26T07:02:44.8059502Z remote: Counting objects: 24% (677/2817) 2024-06-26T07:02:44.8059980Z remote: Counting objects: 25% (705/2817) 2024-06-26T07:02:44.8060866Z remote: Counting objects: 26% (733/2817) 2024-06-26T07:02:44.8061444Z remote: Counting objects: 27% (761/2817) 2024-06-26T07:02:44.8061927Z remote: Counting objects: 28% (789/2817) 2024-06-26T07:02:44.8062409Z remote: Counting objects: 29% (817/2817) 2024-06-26T07:02:44.8062896Z remote: Counting objects: 30% (846/2817) 2024-06-26T07:02:44.8063494Z remote: Counting objects: 31% (874/2817) 2024-06-26T07:02:44.8064039Z remote: Counting objects: 32% (902/2817) 2024-06-26T07:02:44.8064516Z remote: Counting objects: 33% (930/2817) 2024-06-26T07:02:44.8065034Z remote: Counting objects: 34% (958/2817) 2024-06-26T07:02:44.8065610Z remote: Counting objects: 35% (986/2817) 2024-06-26T07:02:44.8066210Z remote: Counting objects: 36% (1015/2817) 2024-06-26T07:02:44.8066788Z remote: Counting objects: 37% (1043/2817) 2024-06-26T07:02:44.8067308Z remote: Counting objects: 38% (1071/2817) 2024-06-26T07:02:44.8067878Z remote: Counting objects: 39% (1099/2817) 2024-06-26T07:02:44.8068576Z remote: Counting objects: 40% (1127/2817) 2024-06-26T07:02:44.8069136Z remote: Counting objects: 41% (1155/2817) 2024-06-26T07:02:44.8069636Z remote: Counting objects: 42% (1184/2817) 2024-06-26T07:02:44.8070151Z remote: Counting objects: 43% (1212/2817) 2024-06-26T07:02:44.8070652Z remote: Counting objects: 44% (1240/2817) 2024-06-26T07:02:44.8071203Z remote: Counting objects: 45% (1268/2817) 2024-06-26T07:02:44.8071830Z remote: Counting objects: 46% (1296/2817) 2024-06-26T07:02:44.8072363Z remote: Counting objects: 47% (1324/2817) 2024-06-26T07:02:44.8073015Z remote: Counting objects: 48% (1353/2817) 2024-06-26T07:02:44.8073505Z remote: Counting objects: 49% (1381/2817) 2024-06-26T07:02:44.8074120Z remote: Counting objects: 50% (1409/2817) 2024-06-26T07:02:44.8074684Z remote: Counting objects: 51% (1437/2817) 2024-06-26T07:02:44.8075178Z remote: Counting objects: 52% (1465/2817) 2024-06-26T07:02:44.8075722Z remote: Counting objects: 53% (1494/2817) 2024-06-26T07:02:44.8076310Z remote: Counting objects: 54% (1522/2817) 2024-06-26T07:02:44.8076960Z remote: Counting objects: 55% (1550/2817) 2024-06-26T07:02:44.8077463Z remote: Counting objects: 56% (1578/2817) 2024-06-26T07:02:44.8077952Z remote: Counting objects: 57% (1606/2817) 2024-06-26T07:02:44.8078451Z remote: Counting objects: 58% (1634/2817) 2024-06-26T07:02:44.8079054Z remote: Counting objects: 59% (1663/2817) 2024-06-26T07:02:44.8079613Z remote: Counting objects: 60% (1691/2817) 2024-06-26T07:02:44.8080119Z remote: Counting objects: 61% (1719/2817) 2024-06-26T07:02:44.8080621Z remote: Counting objects: 62% (1747/2817) 2024-06-26T07:02:44.8081145Z remote: Counting objects: 63% (1775/2817) 2024-06-26T07:02:44.8081778Z remote: Counting objects: 64% (1803/2817) 2024-06-26T07:02:44.8082356Z remote: Counting objects: 65% (1832/2817) 2024-06-26T07:02:44.8082845Z remote: Counting objects: 66% (1860/2817) 2024-06-26T07:02:44.8083354Z remote: Counting objects: 67% (1888/2817) 2024-06-26T07:02:44.8084125Z remote: Counting objects: 68% (1916/2817) 2024-06-26T07:02:44.8084683Z remote: Counting objects: 69% (1944/2817) 2024-06-26T07:02:44.8085173Z remote: Counting objects: 70% (1972/2817) 2024-06-26T07:02:44.8085672Z remote: Counting objects: 71% (2001/2817) 2024-06-26T07:02:44.8086168Z remote: Counting objects: 72% (2029/2817) 2024-06-26T07:02:44.8086744Z remote: Counting objects: 73% (2057/2817) 2024-06-26T07:02:44.8087318Z remote: Counting objects: 74% (2085/2817) 2024-06-26T07:02:44.8087818Z remote: Counting objects: 75% (2113/2817) 2024-06-26T07:02:44.8088316Z remote: Counting objects: 76% (2141/2817) 2024-06-26T07:02:44.8088922Z remote: Counting objects: 77% (2170/2817) 2024-06-26T07:02:44.8089525Z remote: Counting objects: 78% (2198/2817) 2024-06-26T07:02:44.8090085Z remote: Counting objects: 79% (2226/2817) 2024-06-26T07:02:44.8090585Z remote: Counting objects: 80% (2254/2817) 2024-06-26T07:02:44.8091134Z remote: Counting objects: 81% (2282/2817) 2024-06-26T07:02:44.8091686Z remote: Counting objects: 82% (2310/2817) 2024-06-26T07:02:44.8092236Z remote: Counting objects: 83% (2339/2817) 2024-06-26T07:02:44.8092840Z remote: Counting objects: 84% (2367/2817) 2024-06-26T07:02:44.8093338Z remote: Counting objects: 85% (2395/2817) 2024-06-26T07:02:44.8093831Z remote: Counting objects: 86% (2423/2817) 2024-06-26T07:02:44.8094341Z remote: Counting objects: 87% (2451/2817) 2024-06-26T07:02:44.8094841Z remote: Counting objects: 88% (2479/2817) 2024-06-26T07:02:44.8095356Z remote: Counting objects: 89% (2508/2817) 2024-06-26T07:02:44.8095860Z remote: Counting objects: 90% (2536/2817) 2024-06-26T07:02:44.8096364Z remote: Counting objects: 91% (2564/2817) 2024-06-26T07:02:44.8096872Z remote: Counting objects: 92% (2592/2817) 2024-06-26T07:02:44.8097377Z remote: Counting objects: 93% (2620/2817) 2024-06-26T07:02:44.8097879Z remote: Counting objects: 94% (2648/2817) 2024-06-26T07:02:44.8098386Z remote: Counting objects: 95% (2677/2817) 2024-06-26T07:02:44.8098892Z remote: Counting objects: 96% (2705/2817) 2024-06-26T07:02:44.8099402Z remote: Counting objects: 97% (2733/2817) 2024-06-26T07:02:44.8099898Z remote: Counting objects: 98% (2761/2817) 2024-06-26T07:02:44.8100397Z remote: Counting objects: 99% (2789/2817) 2024-06-26T07:02:44.8100997Z remote: Counting objects: 100% (2817/2817) 2024-06-26T07:02:44.8101537Z remote: Counting objects: 100% (2817/2817), done. 2024-06-26T07:02:44.8268259Z remote: Compressing objects: 0% (1/1140) 2024-06-26T07:02:44.8839938Z remote: Compressing objects: 1% (12/1140) 2024-06-26T07:02:44.9259077Z remote: Compressing objects: 2% (23/1140) 2024-06-26T07:02:44.9577919Z remote: Compressing objects: 3% (35/1140) 2024-06-26T07:02:44.9799109Z remote: Compressing objects: 4% (46/1140) 2024-06-26T07:02:45.0099100Z remote: Compressing objects: 5% (57/1140) 2024-06-26T07:02:45.0320040Z remote: Compressing objects: 6% (69/1140) 2024-06-26T07:02:45.1296689Z remote: Compressing objects: 7% (80/1140) 2024-06-26T07:02:45.1916548Z remote: Compressing objects: 8% (92/1140) 2024-06-26T07:02:45.2379869Z remote: Compressing objects: 9% (103/1140) 2024-06-26T07:02:45.2549309Z remote: Compressing objects: 10% (114/1140) 2024-06-26T07:02:45.2643366Z remote: Compressing objects: 11% (126/1140) 2024-06-26T07:02:45.2722953Z remote: Compressing objects: 12% (137/1140) 2024-06-26T07:02:45.2760301Z remote: Compressing objects: 13% (149/1140) 2024-06-26T07:02:45.2760849Z remote: Compressing objects: 14% (160/1140) 2024-06-26T07:02:45.2820314Z remote: Compressing objects: 15% (171/1140) 2024-06-26T07:02:45.2820841Z remote: Compressing objects: 16% (183/1140) 2024-06-26T07:02:45.2823483Z remote: Compressing objects: 17% (194/1140) 2024-06-26T07:02:45.2849002Z remote: Compressing objects: 18% (206/1140) 2024-06-26T07:02:45.2868624Z remote: Compressing objects: 19% (217/1140) 2024-06-26T07:02:45.2896206Z remote: Compressing objects: 20% (228/1140) 2024-06-26T07:02:45.2917050Z remote: Compressing objects: 21% (240/1140) 2024-06-26T07:02:45.2935100Z remote: Compressing objects: 22% (251/1140) 2024-06-26T07:02:45.2948606Z remote: Compressing objects: 23% (263/1140) 2024-06-26T07:02:45.2953115Z remote: Compressing objects: 24% (274/1140) 2024-06-26T07:02:45.2964595Z remote: Compressing objects: 25% (285/1140) 2024-06-26T07:02:45.2972182Z remote: Compressing objects: 26% (297/1140) 2024-06-26T07:02:45.2974787Z remote: Compressing objects: 27% (308/1140) 2024-06-26T07:02:45.2984700Z remote: Compressing objects: 28% (320/1140) 2024-06-26T07:02:45.2992475Z remote: Compressing objects: 29% (331/1140) 2024-06-26T07:02:45.2998925Z remote: Compressing objects: 30% (342/1140) 2024-06-26T07:02:45.3001224Z remote: Compressing objects: 31% (354/1140) 2024-06-26T07:02:45.3019808Z remote: Compressing objects: 32% (365/1140) 2024-06-26T07:02:45.3032102Z remote: Compressing objects: 33% (377/1140) 2024-06-26T07:02:45.3046603Z remote: Compressing objects: 34% (388/1140) 2024-06-26T07:02:45.3060807Z remote: Compressing objects: 35% (399/1140) 2024-06-26T07:02:45.3082048Z remote: Compressing objects: 36% (411/1140) 2024-06-26T07:02:45.3108623Z remote: Compressing objects: 37% (422/1140) 2024-06-26T07:02:45.3124481Z remote: Compressing objects: 38% (434/1140) 2024-06-26T07:02:45.3135750Z remote: Compressing objects: 39% (445/1140) 2024-06-26T07:02:45.3165844Z remote: Compressing objects: 40% (456/1140) 2024-06-26T07:02:45.3188260Z remote: Compressing objects: 41% (468/1140) 2024-06-26T07:02:45.3207134Z remote: Compressing objects: 42% (479/1140) 2024-06-26T07:02:45.3217341Z remote: Compressing objects: 43% (491/1140) 2024-06-26T07:02:45.3225246Z remote: Compressing objects: 44% (502/1140) 2024-06-26T07:02:45.3239263Z remote: Compressing objects: 45% (513/1140) 2024-06-26T07:02:45.3244898Z remote: Compressing objects: 46% (525/1140) 2024-06-26T07:02:45.3253867Z remote: Compressing objects: 47% (536/1140) 2024-06-26T07:02:45.3261279Z remote: Compressing objects: 48% (548/1140) 2024-06-26T07:02:45.3270737Z remote: Compressing objects: 49% (559/1140) 2024-06-26T07:02:45.3271283Z remote: Compressing objects: 50% (570/1140) 2024-06-26T07:02:45.3284465Z remote: Compressing objects: 51% (582/1140) 2024-06-26T07:02:45.3304330Z remote: Compressing objects: 52% (593/1140) 2024-06-26T07:02:45.3318286Z remote: Compressing objects: 53% (605/1140) 2024-06-26T07:02:45.3336842Z remote: Compressing objects: 54% (616/1140) 2024-06-26T07:02:45.3350405Z remote: Compressing objects: 55% (627/1140) 2024-06-26T07:02:45.3358653Z remote: Compressing objects: 56% (639/1140) 2024-06-26T07:02:45.3370492Z remote: Compressing objects: 57% (650/1140) 2024-06-26T07:02:45.3388269Z remote: Compressing objects: 58% (662/1140) 2024-06-26T07:02:45.3399049Z remote: Compressing objects: 59% (673/1140) 2024-06-26T07:02:45.3406679Z remote: Compressing objects: 60% (684/1140) 2024-06-26T07:02:45.3414332Z remote: Compressing objects: 61% (696/1140) 2024-06-26T07:02:45.3430233Z remote: Compressing objects: 62% (707/1140) 2024-06-26T07:02:45.3442574Z remote: Compressing objects: 63% (719/1140) 2024-06-26T07:02:45.3451744Z remote: Compressing objects: 64% (730/1140) 2024-06-26T07:02:45.3462522Z remote: Compressing objects: 65% (741/1140) 2024-06-26T07:02:45.3470612Z remote: Compressing objects: 66% (753/1140) 2024-06-26T07:02:45.3475704Z remote: Compressing objects: 67% (764/1140) 2024-06-26T07:02:45.3477726Z remote: Compressing objects: 68% (776/1140) 2024-06-26T07:02:45.3487978Z remote: Compressing objects: 69% (787/1140) 2024-06-26T07:02:45.3492528Z remote: Compressing objects: 70% (798/1140) 2024-06-26T07:02:45.3493946Z remote: Compressing objects: 71% (810/1140) 2024-06-26T07:02:45.3496106Z remote: Compressing objects: 72% (821/1140) 2024-06-26T07:02:45.3499435Z remote: Compressing objects: 73% (833/1140) 2024-06-26T07:02:45.3509032Z remote: Compressing objects: 74% (844/1140) 2024-06-26T07:02:45.3509907Z remote: Compressing objects: 75% (855/1140) 2024-06-26T07:02:45.3510448Z remote: Compressing objects: 76% (867/1140) 2024-06-26T07:02:45.3510991Z remote: Compressing objects: 77% (878/1140) 2024-06-26T07:02:45.3511533Z remote: Compressing objects: 78% (890/1140) 2024-06-26T07:02:45.3512062Z remote: Compressing objects: 79% (901/1140) 2024-06-26T07:02:45.3512593Z remote: Compressing objects: 80% (912/1140) 2024-06-26T07:02:45.3522020Z remote: Compressing objects: 81% (924/1140) 2024-06-26T07:02:45.3522704Z remote: Compressing objects: 82% (935/1140) 2024-06-26T07:02:45.3523322Z remote: Compressing objects: 83% (947/1140) 2024-06-26T07:02:45.3523973Z remote: Compressing objects: 84% (958/1140) 2024-06-26T07:02:45.3524511Z remote: Compressing objects: 85% (969/1140) 2024-06-26T07:02:45.3525044Z remote: Compressing objects: 86% (981/1140) 2024-06-26T07:02:45.3525580Z remote: Compressing objects: 87% (992/1140) 2024-06-26T07:02:45.3526116Z remote: Compressing objects: 88% (1004/1140) 2024-06-26T07:02:45.3526668Z remote: Compressing objects: 89% (1015/1140) 2024-06-26T07:02:45.3527629Z remote: Compressing objects: 90% (1026/1140) 2024-06-26T07:02:45.3529129Z remote: Compressing objects: 91% (1038/1140) 2024-06-26T07:02:45.3529965Z remote: Compressing objects: 92% (1049/1140) 2024-06-26T07:02:45.3531582Z remote: Compressing objects: 93% (1061/1140) 2024-06-26T07:02:45.3532543Z remote: Compressing objects: 94% (1072/1140) 2024-06-26T07:02:45.3534082Z remote: Compressing objects: 95% (1083/1140) 2024-06-26T07:02:45.3535075Z remote: Compressing objects: 96% (1095/1140) 2024-06-26T07:02:45.3538458Z remote: Compressing objects: 97% (1106/1140) 2024-06-26T07:02:45.3539116Z remote: Compressing objects: 98% (1118/1140) 2024-06-26T07:02:45.3539651Z remote: Compressing objects: 99% (1129/1140) 2024-06-26T07:02:45.3540198Z remote: Compressing objects: 100% (1140/1140) 2024-06-26T07:02:45.3540787Z remote: Compressing objects: 100% (1140/1140), done. 2024-06-26T07:03:04.7605461Z remote: Total 990209 (delta 2280), reused 2016 (delta 1676), pack-reused 987392 2024-06-26T07:03:29.3107039Z [command]/usr/bin/git rev-parse --verify --quiet b8c4c54d347aa776934c60784e35936878ef18dc^{object} 2024-06-26T07:03:29.3125185Z b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:03:29.3130593Z ##[endgroup] 2024-06-26T07:03:29.3131308Z ##[group]Determining the checkout info 2024-06-26T07:03:29.3132064Z ##[endgroup] 2024-06-26T07:03:29.3137779Z ##[group]Checking out the ref 2024-06-26T07:03:29.3138868Z [command]/usr/bin/git checkout --quiet --force b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:03:30.5521400Z ##[endgroup] 2024-06-26T07:03:30.5522015Z ##[group]Setting up auth for fetching submodules 2024-06-26T07:03:30.5526550Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:03:30.5566092Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-06-26T07:03:30.5586152Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-06-26T07:03:30.5606100Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-06-26T07:03:30.5622922Z ##[endgroup] 2024-06-26T07:03:30.5623444Z ##[group]Fetching submodules 2024-06-26T07:03:30.5626515Z [command]/usr/bin/git submodule sync --recursive 2024-06-26T07:03:30.5838226Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-06-26T07:03:30.6043099Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-06-26T07:03:30.6046326Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-06-26T07:03:30.6048570Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-06-26T07:03:30.6050630Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-06-26T07:03:30.6053128Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-06-26T07:03:30.6055543Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-06-26T07:03:30.6057576Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-06-26T07:03:30.6060033Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-06-26T07:03:30.6063233Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-06-26T07:03:30.6066681Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-06-26T07:03:30.6069631Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-06-26T07:03:30.6073263Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-06-26T07:03:30.6076596Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-06-26T07:03:30.6080220Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-06-26T07:03:30.6083818Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-06-26T07:03:30.6087730Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-06-26T07:03:30.6091574Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:03:30.6095216Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-06-26T07:03:30.6099167Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-06-26T07:03:30.6102960Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-06-26T07:03:30.6107017Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-06-26T07:03:30.6111273Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-06-26T07:03:30.6115503Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-06-26T07:03:30.6119721Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-06-26T07:03:30.6124308Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-06-26T07:03:30.6128743Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-06-26T07:03:30.6133442Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-06-26T07:03:30.6137782Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-06-26T07:03:30.6142379Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-06-26T07:03:30.6147037Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-06-26T07:03:30.6152187Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-06-26T07:03:30.6156832Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-06-26T07:03:30.6161738Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-06-26T07:03:30.6166824Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-06-26T07:03:30.6171781Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-06-26T07:03:30.6192426Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-06-26T07:03:30.9133022Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-06-26T07:03:31.0804092Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-06-26T07:03:31.2480911Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-06-26T07:03:31.4765050Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-06-26T07:03:33.3967111Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-06-26T07:03:43.2276012Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-06-26T07:03:43.6002280Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-06-26T07:03:44.1022805Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-06-26T07:03:44.6647913Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-06-26T07:03:45.8187414Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-06-26T07:03:47.6224433Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-06-26T07:03:53.4933998Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-06-26T07:03:54.7323328Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-06-26T07:03:56.4544453Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-06-26T07:03:57.6281804Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-06-26T07:03:57.7973241Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-06-26T07:03:58.1848053Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-06-26T07:03:58.5658127Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-06-26T07:03:59.6191883Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-06-26T07:03:59.9514513Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-06-26T07:04:00.2312602Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-06-26T07:04:01.7691854Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-06-26T07:04:02.5055711Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-06-26T07:04:03.2013058Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-06-26T07:04:08.8038425Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-06-26T07:04:11.2987246Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-06-26T07:04:16.8057155Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-06-26T07:04:17.0241449Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-06-26T07:04:25.0832668Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-06-26T07:04:25.2453632Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-06-26T07:04:25.6663832Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-06-26T07:04:26.6264211Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-06-26T07:04:26.9002251Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-06-26T07:04:27.5313750Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-06-26T07:04:27.9428340Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-06-26T07:04:27.9514040Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-06-26T07:04:27.9575813Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-06-26T07:04:27.9758680Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-06-26T07:04:28.0066317Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-06-26T07:04:28.7880817Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-06-26T07:04:28.8058317Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-06-26T07:04:28.8400198Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-06-26T07:04:28.9247490Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-06-26T07:04:28.9518424Z Submodule path 'third_party/cudnn_frontend': checked out 'aa3abd4bc689d6412979c7f55f9cd132848c9c6a' 2024-06-26T07:04:29.3811969Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-06-26T07:04:29.5965809Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-06-26T07:04:29.6593392Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-06-26T07:04:29.6606624Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:04:29.6609522Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:04:29.6610992Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:04:29.6612471Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:04:29.6614311Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:04:29.6633072Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-06-26T07:04:30.6632681Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-06-26T07:04:31.2343837Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-06-26T07:04:32.9482912Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-06-26T07:04:33.9180838Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-06-26T07:04:34.2559937Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-06-26T07:04:34.3401327Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-06-26T07:04:34.6836166Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-06-26T07:04:34.7381966Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-06-26T07:04:34.7469968Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-06-26T07:04:34.8411388Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-06-26T07:04:34.8694701Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-06-26T07:04:34.8757500Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-06-26T07:04:34.9062231Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-06-26T07:04:34.9257780Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-06-26T07:04:34.9627567Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:04:34.9723925Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-06-26T07:04:34.9735571Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-06-26T07:04:34.9752051Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-06-26T07:04:47.1182928Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-06-26T07:04:47.1323518Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-06-26T07:04:47.2105653Z Submodule path 'third_party/kineto': checked out '8681ff11e1fa54da39023076c5c43eddd87b7a8a' 2024-06-26T07:04:47.2118710Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:04:47.2121242Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:04:47.2123411Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:04:47.2140639Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-06-26T07:04:47.7518457Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-06-26T07:04:48.9702451Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-06-26T07:04:50.0109389Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-06-26T07:04:50.0120372Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:04:50.0122355Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:04:50.0124839Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:04:50.0128002Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:04:50.0130239Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:04:50.0133108Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:04:50.0135436Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:04:50.0138360Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:04:50.0155988Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-06-26T07:04:50.9980838Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-06-26T07:04:51.3922135Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-06-26T07:04:52.6511413Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-06-26T07:04:52.9318114Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-06-26T07:04:53.4122793Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-06-26T07:04:54.3894723Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-06-26T07:04:59.8662795Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-06-26T07:05:00.2075200Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-06-26T07:05:00.2216080Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-06-26T07:05:00.2506362Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-06-26T07:05:00.2605756Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-06-26T07:05:00.2620624Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:05:00.2637588Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-06-26T07:05:00.5406345Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-06-26T07:05:00.5543582Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-06-26T07:05:00.5882198Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-06-26T07:05:00.6694208Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-06-26T07:05:00.6814505Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-06-26T07:05:00.7113446Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-06-26T07:05:00.7610856Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-06-26T07:05:00.7906937Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-06-26T07:05:00.8105459Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-06-26T07:05:00.8967259Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-06-26T07:05:01.1706604Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-06-26T07:05:01.1730009Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-06-26T07:05:01.1731505Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-06-26T07:05:01.1748067Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-06-26T07:05:01.5785559Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-06-26T07:05:02.6520759Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-06-26T07:05:02.6802296Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-06-26T07:05:02.7326166Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-06-26T07:05:02.7337484Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:05:02.7344399Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:05:02.7346550Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:05:02.7348262Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:05:02.7350443Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:05:02.7352730Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:05:02.7354584Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:05:02.7356193Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:05:02.7369203Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-06-26T07:05:03.1962194Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-06-26T07:05:04.2123902Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-06-26T07:05:04.5203495Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-06-26T07:05:10.6321961Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-06-26T07:05:10.9166882Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-06-26T07:05:11.1223017Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-06-26T07:05:11.4550685Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-06-26T07:05:19.0087984Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-06-26T07:05:19.0419374Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-06-26T07:05:19.0545585Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-06-26T07:05:19.1381983Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-06-26T07:05:19.1476281Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-06-26T07:05:19.1587037Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-06-26T07:05:19.1701225Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-06-26T07:05:19.1715557Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:05:19.1717346Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:05:19.1734193Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-06-26T07:05:20.9547275Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-06-26T07:05:22.1926348Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-06-26T07:05:22.2294561Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:05:22.5960440Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-06-26T07:05:22.6041271Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-06-26T07:05:22.8213307Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-06-26T07:05:22.8227317Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:05:22.8229396Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-06-26T07:05:22.8245000Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-06-26T07:05:23.2085664Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-06-26T07:05:24.2320083Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-06-26T07:05:24.2949456Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-06-26T07:05:24.3012198Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-06-26T07:05:24.3099828Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-06-26T07:05:24.3381397Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-06-26T07:05:24.3606005Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-06-26T07:05:24.3937637Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-06-26T07:05:24.4136503Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-06-26T07:05:24.4148092Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:05:24.4150081Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:05:24.4152576Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:05:24.4154996Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:05:24.4170867Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-06-26T07:05:25.4100134Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-06-26T07:05:25.6838871Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-06-26T07:05:26.7099300Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-06-26T07:05:27.7505974Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-06-26T07:05:27.7619242Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-06-26T07:05:27.8184153Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-06-26T07:05:27.8416626Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-06-26T07:05:27.8426793Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:05:27.8442380Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-06-26T07:05:28.0949756Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-06-26T07:05:28.0973902Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-06-26T07:05:28.1177087Z Entering 'android/libs/fbjni' 2024-06-26T07:05:28.1202454Z Entering 'third_party/FP16' 2024-06-26T07:05:28.1229483Z Entering 'third_party/FXdiv' 2024-06-26T07:05:28.1257493Z Entering 'third_party/NNPACK' 2024-06-26T07:05:28.1284860Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:05:28.1313637Z Entering 'third_party/XNNPACK' 2024-06-26T07:05:28.1351540Z Entering 'third_party/benchmark' 2024-06-26T07:05:28.1378057Z Entering 'third_party/cpp-httplib' 2024-06-26T07:05:28.1404869Z Entering 'third_party/cpuinfo' 2024-06-26T07:05:28.1434278Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:05:28.1460755Z Entering 'third_party/cutlass' 2024-06-26T07:05:28.1493427Z Entering 'third_party/eigen' 2024-06-26T07:05:28.1523453Z Entering 'third_party/fbgemm' 2024-06-26T07:05:28.1553077Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:05:28.1578377Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:05:28.1607282Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:05:28.1638734Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:05:28.1667646Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:05:28.1695345Z Entering 'third_party/flatbuffers' 2024-06-26T07:05:28.1726248Z Entering 'third_party/fmt' 2024-06-26T07:05:28.1752149Z Entering 'third_party/foxi' 2024-06-26T07:05:28.1778717Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:05:28.1806989Z Entering 'third_party/gloo' 2024-06-26T07:05:28.1832332Z Entering 'third_party/googletest' 2024-06-26T07:05:28.1860779Z Entering 'third_party/ideep' 2024-06-26T07:05:28.1886580Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:05:28.1919178Z Entering 'third_party/ittapi' 2024-06-26T07:05:28.1945116Z Entering 'third_party/kineto' 2024-06-26T07:05:28.1972550Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:05:28.1999235Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:05:28.2028363Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:05:28.2055960Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:05:28.2084190Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:05:28.2111644Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:05:28.2139539Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:05:28.2168594Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:05:28.2197380Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:05:28.2224118Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:05:28.2254009Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:05:28.2280361Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:05:28.2307352Z Entering 'third_party/mimalloc' 2024-06-26T07:05:28.2336425Z Entering 'third_party/nccl/nccl' 2024-06-26T07:05:28.2364714Z Entering 'third_party/nlohmann' 2024-06-26T07:05:28.2393545Z Entering 'third_party/onnx' 2024-06-26T07:05:28.2432595Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:05:28.2459566Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:05:28.2490790Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:05:28.2521559Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:05:28.2549790Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:05:28.2578172Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:05:28.2607636Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:05:28.2634357Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:05:28.2661251Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:05:28.2686683Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:05:28.2712235Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:05:28.2741981Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:05:28.2771444Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:05:28.2813599Z Entering 'third_party/pocketfft' 2024-06-26T07:05:28.2843192Z Entering 'third_party/protobuf' 2024-06-26T07:05:28.2872872Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:05:28.2902041Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:05:28.2930116Z Entering 'third_party/psimd' 2024-06-26T07:05:28.2959575Z Entering 'third_party/pthreadpool' 2024-06-26T07:05:28.2989564Z Entering 'third_party/pybind11' 2024-06-26T07:05:28.3017910Z Entering 'third_party/python-peachpy' 2024-06-26T07:05:28.3045619Z Entering 'third_party/sleef' 2024-06-26T07:05:28.3073110Z Entering 'third_party/tensorpipe' 2024-06-26T07:05:28.3099652Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:05:28.3128409Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:05:28.3154573Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:05:28.3182196Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:05:28.3208624Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:05:28.3248985Z ##[endgroup] 2024-06-26T07:05:28.3250933Z ##[group]Persisting credentials for submodules 2024-06-26T07:05:28.3253634Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-06-26T07:05:28.3458734Z Entering 'android/libs/fbjni' 2024-06-26T07:05:28.3493655Z Entering 'third_party/FP16' 2024-06-26T07:05:28.3530677Z Entering 'third_party/FXdiv' 2024-06-26T07:05:28.3566766Z Entering 'third_party/NNPACK' 2024-06-26T07:05:28.3601468Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:05:28.3636255Z Entering 'third_party/XNNPACK' 2024-06-26T07:05:28.3682758Z Entering 'third_party/benchmark' 2024-06-26T07:05:28.3717677Z Entering 'third_party/cpp-httplib' 2024-06-26T07:05:28.3753600Z Entering 'third_party/cpuinfo' 2024-06-26T07:05:28.3789480Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:05:28.3824222Z Entering 'third_party/cutlass' 2024-06-26T07:05:28.3863860Z Entering 'third_party/eigen' 2024-06-26T07:05:28.3900574Z Entering 'third_party/fbgemm' 2024-06-26T07:05:28.3935290Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:05:28.3969396Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:05:28.4003400Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:05:28.4038960Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:05:28.4073268Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:05:28.4109437Z Entering 'third_party/flatbuffers' 2024-06-26T07:05:28.4144163Z Entering 'third_party/fmt' 2024-06-26T07:05:28.4177842Z Entering 'third_party/foxi' 2024-06-26T07:05:28.4213905Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:05:28.4248999Z Entering 'third_party/gloo' 2024-06-26T07:05:28.4283822Z Entering 'third_party/googletest' 2024-06-26T07:05:28.4318436Z Entering 'third_party/ideep' 2024-06-26T07:05:28.4353662Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:05:28.4392555Z Entering 'third_party/ittapi' 2024-06-26T07:05:28.4428002Z Entering 'third_party/kineto' 2024-06-26T07:05:28.4464262Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:05:28.4497063Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:05:28.4530250Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:05:28.4562291Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:05:28.4595781Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:05:28.4629317Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:05:28.4664857Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:05:28.4697946Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:05:28.4733244Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:05:28.4766395Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:05:28.4801264Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:05:28.4835443Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:05:28.4870169Z Entering 'third_party/mimalloc' 2024-06-26T07:05:28.4904696Z Entering 'third_party/nccl/nccl' 2024-06-26T07:05:28.4939849Z Entering 'third_party/nlohmann' 2024-06-26T07:05:28.4974809Z Entering 'third_party/onnx' 2024-06-26T07:05:28.5021849Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:05:28.5056917Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:05:28.5095258Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:05:28.5131560Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:05:28.5164793Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:05:28.5197800Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:05:28.5232021Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:05:28.5267307Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:05:28.5301556Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:05:28.5335759Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:05:28.5369106Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:05:28.5401880Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:05:28.5437709Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:05:28.5489123Z Entering 'third_party/pocketfft' 2024-06-26T07:05:28.5525785Z Entering 'third_party/protobuf' 2024-06-26T07:05:28.5562597Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:05:28.5597381Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:05:28.5631211Z Entering 'third_party/psimd' 2024-06-26T07:05:28.5665446Z Entering 'third_party/pthreadpool' 2024-06-26T07:05:28.5700448Z Entering 'third_party/pybind11' 2024-06-26T07:05:28.5733463Z Entering 'third_party/python-peachpy' 2024-06-26T07:05:28.5766462Z Entering 'third_party/sleef' 2024-06-26T07:05:28.5801943Z Entering 'third_party/tensorpipe' 2024-06-26T07:05:28.5835632Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:05:28.5870730Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:05:28.5903859Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:05:28.5938169Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:05:28.5972380Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:05:28.6019754Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-06-26T07:05:28.6225958Z Entering 'android/libs/fbjni' 2024-06-26T07:05:28.6256920Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-06-26T07:05:28.6269612Z Entering 'third_party/FP16' 2024-06-26T07:05:28.6302307Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-06-26T07:05:28.6315713Z Entering 'third_party/FXdiv' 2024-06-26T07:05:28.6348045Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-06-26T07:05:28.6361816Z Entering 'third_party/NNPACK' 2024-06-26T07:05:28.6393178Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-06-26T07:05:28.6404554Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:05:28.6436185Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-06-26T07:05:28.6448618Z Entering 'third_party/XNNPACK' 2024-06-26T07:05:28.6480625Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-06-26T07:05:28.6502898Z Entering 'third_party/benchmark' 2024-06-26T07:05:28.6535594Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:05:28.6549047Z Entering 'third_party/cpp-httplib' 2024-06-26T07:05:28.6581205Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-06-26T07:05:28.6595761Z Entering 'third_party/cpuinfo' 2024-06-26T07:05:28.6626095Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:05:28.6637848Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:05:28.6670523Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-06-26T07:05:28.6684964Z Entering 'third_party/cutlass' 2024-06-26T07:05:28.6715870Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:05:28.6736066Z Entering 'third_party/eigen' 2024-06-26T07:05:28.6769057Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-06-26T07:05:28.6783856Z Entering 'third_party/fbgemm' 2024-06-26T07:05:28.6815462Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-06-26T07:05:28.6829032Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:05:28.6860110Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-06-26T07:05:28.6873485Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:05:28.6903828Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:05:28.6915140Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:05:28.6946090Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:05:28.6964137Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:05:28.6996076Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:05:28.7009576Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:05:28.7041005Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-06-26T07:05:28.7053535Z Entering 'third_party/flatbuffers' 2024-06-26T07:05:28.7085372Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-06-26T07:05:28.7099883Z Entering 'third_party/fmt' 2024-06-26T07:05:28.7133100Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:05:28.7146961Z Entering 'third_party/foxi' 2024-06-26T07:05:28.7179380Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-06-26T07:05:28.7192665Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:05:28.7225107Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-06-26T07:05:28.7237184Z Entering 'third_party/gloo' 2024-06-26T07:05:28.7268777Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-06-26T07:05:28.7280688Z Entering 'third_party/googletest' 2024-06-26T07:05:28.7311406Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:05:28.7323924Z Entering 'third_party/ideep' 2024-06-26T07:05:28.7354549Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-06-26T07:05:28.7367173Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:05:28.7398065Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-06-26T07:05:28.7418293Z Entering 'third_party/ittapi' 2024-06-26T07:05:28.7449626Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-06-26T07:05:28.7462892Z Entering 'third_party/kineto' 2024-06-26T07:05:28.7503741Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-06-26T07:05:28.7509479Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:05:28.7541928Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-06-26T07:05:28.7554085Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:05:28.7586486Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-06-26T07:05:28.7600099Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:05:28.7633672Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-06-26T07:05:28.7646276Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:05:28.7676849Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:05:28.7689994Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:05:28.7721814Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-06-26T07:05:28.7734403Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:05:28.7766441Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-06-26T07:05:28.7779781Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:05:28.7812623Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-06-26T07:05:28.7825052Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:05:28.7857803Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:05:28.7870825Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:05:28.7902620Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-06-26T07:05:28.7916804Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:05:28.7949297Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-06-26T07:05:28.7964254Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:05:28.7995156Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-06-26T07:05:28.8007022Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:05:28.8038504Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-06-26T07:05:28.8051479Z Entering 'third_party/mimalloc' 2024-06-26T07:05:28.8084848Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-06-26T07:05:28.8099572Z Entering 'third_party/nccl/nccl' 2024-06-26T07:05:28.8131615Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-06-26T07:05:28.8145724Z Entering 'third_party/nlohmann' 2024-06-26T07:05:28.8176439Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-06-26T07:05:28.8191319Z Entering 'third_party/onnx' 2024-06-26T07:05:28.8222235Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-06-26T07:05:28.8248462Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:05:28.8279521Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:05:28.8292252Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:05:28.8323278Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:05:28.8339255Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:05:28.8371264Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-06-26T07:05:28.8385094Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:05:28.8417473Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:05:28.8430142Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:05:28.8461562Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:05:28.8474320Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:05:28.8506885Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-06-26T07:05:28.8519673Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:05:28.8550784Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-06-26T07:05:28.8564858Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:05:28.8596874Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-06-26T07:05:28.8609359Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:05:28.8639094Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-06-26T07:05:28.8650073Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:05:28.8680742Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-06-26T07:05:28.8692995Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:05:28.8725453Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-06-26T07:05:28.8738759Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:05:28.8771308Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-06-26T07:05:28.8785203Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:05:28.8816441Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-06-26T07:05:28.8845365Z Entering 'third_party/pocketfft' 2024-06-26T07:05:28.8878014Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-06-26T07:05:28.8890750Z Entering 'third_party/protobuf' 2024-06-26T07:05:28.8921953Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-06-26T07:05:28.8938499Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:05:28.8970105Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:05:28.8982561Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:05:28.9014733Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:05:28.9028819Z Entering 'third_party/psimd' 2024-06-26T07:05:28.9061586Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-06-26T07:05:28.9075554Z Entering 'third_party/pthreadpool' 2024-06-26T07:05:28.9106766Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-06-26T07:05:28.9120537Z Entering 'third_party/pybind11' 2024-06-26T07:05:28.9153423Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:05:28.9166261Z Entering 'third_party/python-peachpy' 2024-06-26T07:05:28.9195965Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-06-26T07:05:28.9207525Z Entering 'third_party/sleef' 2024-06-26T07:05:28.9239027Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-06-26T07:05:28.9251820Z Entering 'third_party/tensorpipe' 2024-06-26T07:05:28.9282762Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-06-26T07:05:28.9296913Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:05:28.9328030Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:05:28.9340317Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:05:28.9372204Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-06-26T07:05:28.9384461Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:05:28.9416121Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-06-26T07:05:28.9429284Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:05:28.9460553Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:05:28.9472891Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:05:28.9504464Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-06-26T07:05:29.0364100Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-06-26T07:05:29.0571311Z Entering 'android/libs/fbjni' 2024-06-26T07:05:29.0596866Z Entering 'third_party/FP16' 2024-06-26T07:05:29.0626914Z Entering 'third_party/FXdiv' 2024-06-26T07:05:29.0653670Z Entering 'third_party/NNPACK' 2024-06-26T07:05:29.0684276Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:05:29.0712405Z Entering 'third_party/XNNPACK' 2024-06-26T07:05:29.0752875Z Entering 'third_party/benchmark' 2024-06-26T07:05:29.0780739Z Entering 'third_party/cpp-httplib' 2024-06-26T07:05:29.0810236Z Entering 'third_party/cpuinfo' 2024-06-26T07:05:29.0839419Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:05:29.0865418Z Entering 'third_party/cutlass' 2024-06-26T07:05:29.0899987Z Entering 'third_party/eigen' 2024-06-26T07:05:29.0932401Z Entering 'third_party/fbgemm' 2024-06-26T07:05:29.0959704Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:05:29.0987693Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:05:29.1016717Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:05:29.1047913Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:05:29.1075294Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:05:29.1104542Z Entering 'third_party/flatbuffers' 2024-06-26T07:05:29.1134964Z Entering 'third_party/fmt' 2024-06-26T07:05:29.1164847Z Entering 'third_party/foxi' 2024-06-26T07:05:29.1190708Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:05:29.1220351Z Entering 'third_party/gloo' 2024-06-26T07:05:29.1250131Z Entering 'third_party/googletest' 2024-06-26T07:05:29.1279904Z Entering 'third_party/ideep' 2024-06-26T07:05:29.1309222Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:05:29.1341578Z Entering 'third_party/ittapi' 2024-06-26T07:05:29.1368340Z Entering 'third_party/kineto' 2024-06-26T07:05:29.1395082Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:05:29.1424194Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:05:29.1453219Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:05:29.1480058Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:05:29.1508432Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:05:29.1536053Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:05:29.1565920Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:05:29.1593898Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:05:29.1622435Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:05:29.1650209Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:05:29.1678794Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:05:29.1707603Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:05:29.1736637Z Entering 'third_party/mimalloc' 2024-06-26T07:05:29.1765641Z Entering 'third_party/nccl/nccl' 2024-06-26T07:05:29.1791180Z Entering 'third_party/nlohmann' 2024-06-26T07:05:29.1818107Z Entering 'third_party/onnx' 2024-06-26T07:05:29.1859354Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:05:29.1887290Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:05:29.1917059Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:05:29.1947963Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:05:29.1975129Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:05:29.2003402Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:05:29.2032581Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:05:29.2059700Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:05:29.2085051Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:05:29.2111979Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:05:29.2138676Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:05:29.2166663Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:05:29.2195485Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:05:29.2237760Z Entering 'third_party/pocketfft' 2024-06-26T07:05:29.2268436Z Entering 'third_party/protobuf' 2024-06-26T07:05:29.2296694Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:05:29.2322610Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:05:29.2352948Z Entering 'third_party/psimd' 2024-06-26T07:05:29.2380474Z Entering 'third_party/pthreadpool' 2024-06-26T07:05:29.2408647Z Entering 'third_party/pybind11' 2024-06-26T07:05:29.2436886Z Entering 'third_party/python-peachpy' 2024-06-26T07:05:29.2464532Z Entering 'third_party/sleef' 2024-06-26T07:05:29.2493657Z Entering 'third_party/tensorpipe' 2024-06-26T07:05:29.2522520Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:05:29.2550576Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:05:29.2578223Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:05:29.2605985Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:05:29.2632763Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:05:29.2673493Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-06-26T07:05:29.2882623Z Entering 'android/libs/fbjni' 2024-06-26T07:05:29.2911624Z Entering 'third_party/FP16' 2024-06-26T07:05:29.2939052Z Entering 'third_party/FXdiv' 2024-06-26T07:05:29.2969703Z Entering 'third_party/NNPACK' 2024-06-26T07:05:29.2995964Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:05:29.3025249Z Entering 'third_party/XNNPACK' 2024-06-26T07:05:29.3066398Z Entering 'third_party/benchmark' 2024-06-26T07:05:29.3094210Z Entering 'third_party/cpp-httplib' 2024-06-26T07:05:29.3122029Z Entering 'third_party/cpuinfo' 2024-06-26T07:05:29.3151488Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:05:29.3178644Z Entering 'third_party/cutlass' 2024-06-26T07:05:29.3209768Z Entering 'third_party/eigen' 2024-06-26T07:05:29.3238574Z Entering 'third_party/fbgemm' 2024-06-26T07:05:29.3267120Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:05:29.3293308Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:05:29.3320017Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:05:29.3351930Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:05:29.3378934Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:05:29.3406216Z Entering 'third_party/flatbuffers' 2024-06-26T07:05:29.3434880Z Entering 'third_party/fmt' 2024-06-26T07:05:29.3463673Z Entering 'third_party/foxi' 2024-06-26T07:05:29.3493093Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:05:29.3519713Z Entering 'third_party/gloo' 2024-06-26T07:05:29.3547068Z Entering 'third_party/googletest' 2024-06-26T07:05:29.3574521Z Entering 'third_party/ideep' 2024-06-26T07:05:29.3600628Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:05:29.3634986Z Entering 'third_party/ittapi' 2024-06-26T07:05:29.3662377Z Entering 'third_party/kineto' 2024-06-26T07:05:29.3690729Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:05:29.3719264Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:05:29.3749815Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:05:29.3776321Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:05:29.3804174Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:05:29.3831750Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:05:29.3861221Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:05:29.3889286Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:05:29.3917164Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:05:29.3944074Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:05:29.3972674Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:05:29.3998668Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:05:29.4028862Z Entering 'third_party/mimalloc' 2024-06-26T07:05:29.4056355Z Entering 'third_party/nccl/nccl' 2024-06-26T07:05:29.4085316Z Entering 'third_party/nlohmann' 2024-06-26T07:05:29.4113417Z Entering 'third_party/onnx' 2024-06-26T07:05:29.4151768Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:05:29.4180188Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:05:29.4208535Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:05:29.4237550Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:05:29.4264753Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:05:29.4291036Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:05:29.4316489Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:05:29.4345058Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:05:29.4371708Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:05:29.4399372Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:05:29.4427221Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:05:29.4455205Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:05:29.4483852Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:05:29.4525182Z Entering 'third_party/pocketfft' 2024-06-26T07:05:29.4554460Z Entering 'third_party/protobuf' 2024-06-26T07:05:29.4585920Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:05:29.4613559Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:05:29.4643104Z Entering 'third_party/psimd' 2024-06-26T07:05:29.4671368Z Entering 'third_party/pthreadpool' 2024-06-26T07:05:29.4699031Z Entering 'third_party/pybind11' 2024-06-26T07:05:29.4724803Z Entering 'third_party/python-peachpy' 2024-06-26T07:05:29.4751879Z Entering 'third_party/sleef' 2024-06-26T07:05:29.4779975Z Entering 'third_party/tensorpipe' 2024-06-26T07:05:29.4805669Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:05:29.4832174Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:05:29.4860485Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:05:29.4886330Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:05:29.4913069Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:05:29.4950290Z ##[endgroup] 2024-06-26T07:05:29.4983180Z [command]/usr/bin/git log -1 --format='%H' 2024-06-26T07:05:29.5005194Z 'b8c4c54d347aa776934c60784e35936878ef18dc' 2024-06-26T07:05:29.5173442Z Prepare all required actions 2024-06-26T07:05:29.5173899Z Getting action download info 2024-06-26T07:05:29.6940497Z ##[group]Run ./.github/actions/setup-linux 2024-06-26T07:05:29.6940929Z env: 2024-06-26T07:05:29.6941204Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:29.6941574Z ##[endgroup] 2024-06-26T07:05:29.6999878Z ##[group]Run set -euo pipefail 2024-06-26T07:05:29.7000306Z set -euo pipefail 2024-06-26T07:05:29.7000695Z function get_ec2_metadata() { 2024-06-26T07:05:29.7001229Z  # Pulled from instance metadata endpoint for EC2 2024-06-26T07:05:29.7002107Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-06-26T07:05:29.7002866Z  category=$1 2024-06-26T07:05:29.7003384Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-06-26T07:05:29.7004172Z  runner_name_str=i-0a96abc6f4fd8966c 2024-06-26T07:05:29.7004681Z  if [[ -f /.inarc ]]; then 2024-06-26T07:05:29.7005169Z  echo "ARC Runner, no info on ec2 metadata" 2024-06-26T07:05:29.7005728Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-06-26T07:05:29.7006405Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-06-26T07:05:29.7007000Z  else 2024-06-26T07:05:29.7007478Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-06-26T07:05:29.7008052Z  fi 2024-06-26T07:05:29.7008334Z } 2024-06-26T07:05:29.7008696Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-06-26T07:05:29.7009268Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-06-26T07:05:29.7009918Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-06-26T07:05:29.7010482Z echo "system info $(uname -a)" 2024-06-26T07:05:29.7018355Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:29.7018850Z env: 2024-06-26T07:05:29.7019148Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:29.7019501Z ##[endgroup] 2024-06-26T07:05:29.7087949Z ami-id: ami-0ce0c36d7a00b20e2 2024-06-26T07:05:29.7135061Z instance-id: i-0a96abc6f4fd8966c 2024-06-26T07:05:29.7177925Z instance-type: g5.4xlarge 2024-06-26T07:05:29.7183067Z system info Linux ip-10-0-76-149.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-06-26T07:05:29.7202705Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:05:29.7203974Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:05:29.7211558Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:29.7212126Z env: 2024-06-26T07:05:29.7212408Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:29.7212758Z ##[endgroup] 2024-06-26T07:05:29.7278224Z ##[group]Run if systemctl is-active --quiet docker; then 2024-06-26T07:05:29.7278981Z if systemctl is-active --quiet docker; then 2024-06-26T07:05:29.7279515Z  echo "Docker daemon is running..."; 2024-06-26T07:05:29.7279959Z else 2024-06-26T07:05:29.7280439Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-06-26T07:05:29.7281016Z fi 2024-06-26T07:05:29.7288808Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:29.7289291Z env: 2024-06-26T07:05:29.7289569Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:29.7289923Z ##[endgroup] 2024-06-26T07:05:29.7325164Z Docker daemon is running... 2024-06-26T07:05:29.7368474Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:05:29.7369041Z with: 2024-06-26T07:05:29.7369325Z shell: bash 2024-06-26T07:05:29.7369631Z timeout_minutes: 5 2024-06-26T07:05:29.7369966Z max_attempts: 3 2024-06-26T07:05:29.7370296Z retry_wait_seconds: 30 2024-06-26T07:05:29.7371812Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-06-26T07:05:29.7373220Z polling_interval_seconds: 1 2024-06-26T07:05:29.7373609Z warning_on_retry: true 2024-06-26T07:05:29.7373965Z continue_on_error: false 2024-06-26T07:05:29.7374319Z env: 2024-06-26T07:05:29.7374595Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:29.7374963Z AWS_RETRY_MODE: standard 2024-06-26T07:05:29.7375329Z AWS_MAX_ATTEMPTS: 5 2024-06-26T07:05:29.7375671Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:05:29.7376056Z ##[endgroup] 2024-06-26T07:05:30.6406777Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:05:30.6407569Z Configure a credential helper to remove this warning. See 2024-06-26T07:05:30.6408393Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:05:30.6408929Z 2024-06-26T07:05:30.6409068Z Login Succeeded 2024-06-26T07:05:30.7863182Z Command completed after 1 attempt(s). 2024-06-26T07:05:30.7902971Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:05:30.7903674Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:05:30.7904321Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:05:30.7912247Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:30.7912735Z env: 2024-06-26T07:05:30.7913014Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:30.7913370Z ##[endgroup] 2024-06-26T07:05:30.7970499Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:05:30.7971247Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:05:30.7971829Z # shellcheck disable=SC2046 2024-06-26T07:05:30.7972290Z docker stop $(docker ps -q) || true 2024-06-26T07:05:30.7972773Z # Prune all of the docker images 2024-06-26T07:05:30.7973228Z docker system prune -af 2024-06-26T07:05:30.7980230Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:30.7980726Z env: 2024-06-26T07:05:30.7981008Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:30.7981356Z ##[endgroup] 2024-06-26T07:05:30.8227053Z "docker stop" requires at least 1 argument. 2024-06-26T07:05:30.8228071Z See 'docker stop --help'. 2024-06-26T07:05:30.8228411Z 2024-06-26T07:05:30.8228707Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-06-26T07:05:30.8229244Z 2024-06-26T07:05:30.8229450Z Stop one or more running containers 2024-06-26T07:05:30.8377861Z Total reclaimed space: 0B 2024-06-26T07:05:30.8414985Z ##[group]Run set +e 2024-06-26T07:05:30.8415499Z set +e 2024-06-26T07:05:30.8415943Z set -x 2024-06-26T07:05:30.8416375Z  2024-06-26T07:05:30.8416860Z PT_DOMAIN=download.pytorch.org 2024-06-26T07:05:30.8418049Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-06-26T07:05:30.8419912Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-06-26T07:05:30.8421100Z # one is returned at random 2024-06-26T07:05:30.8421921Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-06-26T07:05:30.8422687Z  2024-06-26T07:05:30.8423143Z if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:05:30.8424073Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-06-26T07:05:30.8425229Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-06-26T07:05:30.8426050Z  2024-06-26T07:05:30.8426522Z  if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:05:30.8427336Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-06-26T07:05:30.8428181Z  exit 1 2024-06-26T07:05:30.8428640Z  fi 2024-06-26T07:05:30.8429071Z fi 2024-06-26T07:05:30.8429656Z  2024-06-26T07:05:30.8430186Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-06-26T07:05:30.8430949Z  # Clean up any old records first 2024-06-26T07:05:30.8431703Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-06-26T07:05:30.8432384Z fi 2024-06-26T07:05:30.8432807Z  2024-06-26T07:05:30.8433431Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-06-26T07:05:30.8434264Z cat /etc/hosts 2024-06-26T07:05:30.8444616Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:30.8445358Z env: 2024-06-26T07:05:30.8445770Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:30.8446288Z ##[endgroup] 2024-06-26T07:05:30.8474123Z + PT_DOMAIN=download.pytorch.org 2024-06-26T07:05:30.8478063Z ++ dig -4 +short download.pytorch.org 2024-06-26T07:05:30.8478502Z ++ tail -n1 2024-06-26T07:05:30.8615476Z + RESOLVED_IP=18.160.10.22 2024-06-26T07:05:30.8615916Z + '[' -z 18.160.10.22 ']' 2024-06-26T07:05:30.8616381Z + grep -r download.pytorch.org /etc/hosts 2024-06-26T07:05:30.8621421Z 18.67.76.37 download.pytorch.org 2024-06-26T07:05:30.8622540Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-06-26T07:05:30.8720841Z + echo '18.160.10.22 download.pytorch.org' 2024-06-26T07:05:30.8721343Z + sudo tee -a /etc/hosts 2024-06-26T07:05:30.8956628Z 18.160.10.22 download.pytorch.org 2024-06-26T07:05:30.8969898Z + cat /etc/hosts 2024-06-26T07:05:30.8974764Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-06-26T07:05:30.8987055Z ::1 localhost6 localhost6.localdomain6 2024-06-26T07:05:30.8987582Z 18.160.10.22 download.pytorch.org 2024-06-26T07:05:30.9193002Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-06-26T07:05:30.9193597Z with: 2024-06-26T07:05:30.9194664Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:30.9195854Z docker-build-dir: .ci/docker 2024-06-26T07:05:30.9196217Z working-directory: . 2024-06-26T07:05:30.9196668Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:30.9197179Z force-push: false 2024-06-26T07:05:30.9197465Z env: 2024-06-26T07:05:30.9197728Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:30.9198057Z ##[endgroup] 2024-06-26T07:05:30.9216596Z ##[group]Run set -ex 2024-06-26T07:05:30.9216948Z set -ex 2024-06-26T07:05:30.9217231Z  2024-06-26T07:05:30.9217764Z # If the docker build directory or the build script doesn't exist, the action will 2024-06-26T07:05:30.9218709Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-06-26T07:05:30.9219496Z # job could then download the pre-built image as usual 2024-06-26T07:05:30.9220218Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-06-26T07:05:30.9221009Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:30.9221629Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:30.9222181Z  2024-06-26T07:05:30.9222677Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-06-26T07:05:30.9223290Z  exit 0 2024-06-26T07:05:30.9223577Z else 2024-06-26T07:05:30.9223932Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:30.9224362Z fi 2024-06-26T07:05:30.9224640Z  2024-06-26T07:05:30.9225102Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-06-26T07:05:30.9225927Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-06-26T07:05:30.9226677Z  # use it as it is, but first let's extract the tag 2024-06-26T07:05:30.9227359Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-06-26T07:05:30.9228142Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:30.9228818Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:30.9229366Z else 2024-06-26T07:05:30.9229783Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-06-26T07:05:30.9230432Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:30.9231302Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:30.9232037Z fi 2024-06-26T07:05:30.9239679Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:30.9240164Z env: 2024-06-26T07:05:30.9240431Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:30.9240771Z REPO_NAME: pytorch 2024-06-26T07:05:30.9241871Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:30.9243065Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:05:30.9243553Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:30.9244307Z ##[endgroup] 2024-06-26T07:05:30.9264053Z + [[ ! -d .ci/docker ]] 2024-06-26T07:05:30.9264465Z + [[ ! -f .ci/docker/build.sh ]] 2024-06-26T07:05:30.9264845Z + echo skip=false 2024-06-26T07:05:30.9266780Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-06-26T07:05:30.9269670Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:30.9270838Z ++ awk -F '[:,]' '{print $2}' 2024-06-26T07:05:30.9276505Z + DOCKER_TAG=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:30.9277149Z + echo docker-tag=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:30.9278788Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:30.9305422Z ##[group]Run set +e 2024-06-26T07:05:30.9305805Z set +e 2024-06-26T07:05:30.9306108Z set -x 2024-06-26T07:05:30.9306401Z  2024-06-26T07:05:30.9306690Z login() { 2024-06-26T07:05:30.9307363Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:05:30.9308207Z } 2024-06-26T07:05:30.9308484Z  2024-06-26T07:05:30.9308773Z retry () { 2024-06-26T07:05:30.9309167Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:05:30.9309613Z } 2024-06-26T07:05:30.9309888Z  2024-06-26T07:05:30.9310334Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:05:30.9310745Z  2024-06-26T07:05:30.9311212Z # Check if image already exists, if it does then skip building it 2024-06-26T07:05:30.9311923Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-06-26T07:05:30.9312431Z  exit 0 2024-06-26T07:05:30.9312741Z fi 2024-06-26T07:05:30.9313015Z  2024-06-26T07:05:30.9313515Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-06-26T07:05:30.9314335Z # be empty. The default action would be to continue rebuild the image 2024-06-26T07:05:30.9315078Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-06-26T07:05:30.9315752Z  # if we're on the base branch then use the parent commit 2024-06-26T07:05:30.9316350Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-06-26T07:05:30.9316787Z else 2024-06-26T07:05:30.9317254Z  # otherwise we're on a PR, so use the most recent base commit 2024-06-26T07:05:30.9317947Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-06-26T07:05:30.9318453Z fi 2024-06-26T07:05:30.9318740Z  2024-06-26T07:05:30.9319066Z if [[ -z "${MERGE_BASE}" ]]; then 2024-06-26T07:05:30.9319569Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:30.9320030Z  2024-06-26T07:05:30.9320685Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-06-26T07:05:30.9321439Z  exit 0 2024-06-26T07:05:30.9321737Z fi 2024-06-26T07:05:30.9322015Z  2024-06-26T07:05:30.9322454Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-06-26T07:05:30.9323427Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-06-26T07:05:30.9324450Z  exit 1 2024-06-26T07:05:30.9324777Z fi 2024-06-26T07:05:30.9325067Z  2024-06-26T07:05:30.9325561Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-06-26T07:05:30.9326495Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-06-26T07:05:30.9327357Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-06-26T07:05:30.9328307Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-06-26T07:05:30.9329383Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-06-26T07:05:30.9330118Z fi 2024-06-26T07:05:30.9330397Z  2024-06-26T07:05:30.9330766Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:05:30.9338593Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:30.9339094Z env: 2024-06-26T07:05:30.9339388Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:30.9339771Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:05:30.9340249Z BASE_REVISION: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:05:30.9341502Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:30.9342720Z DOCKER_TAG: 91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:30.9343332Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:30.9343857Z ##[endgroup] 2024-06-26T07:05:30.9363353Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:30.9364389Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:30.9365351Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:05:30.9366325Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:31.3488000Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:05:31.3488977Z Configure a credential helper to remove this warning. See 2024-06-26T07:05:31.3489866Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:05:31.3490377Z 2024-06-26T07:05:31.3490508Z Login Succeeded 2024-06-26T07:05:31.3500777Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:31.5714808Z { 2024-06-26T07:05:31.5715843Z "schemaVersion": 2, 2024-06-26T07:05:31.5716617Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-06-26T07:05:31.5717247Z "config": { 2024-06-26T07:05:31.5717691Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-06-26T07:05:31.5718229Z "size": 48681, 2024-06-26T07:05:31.5718761Z "digest": "sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd" 2024-06-26T07:05:31.5719374Z }, 2024-06-26T07:05:31.5719641Z "layers": [ 2024-06-26T07:05:31.5719931Z { 2024-06-26T07:05:31.5720437Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5720987Z "size": 28580681, 2024-06-26T07:05:31.5721530Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-06-26T07:05:31.5722317Z }, 2024-06-26T07:05:31.5722651Z { 2024-06-26T07:05:31.5723125Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5724112Z "size": 7943451, 2024-06-26T07:05:31.5724842Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-06-26T07:05:31.5725697Z + exit 0 2024-06-26T07:05:31.5726066Z }, 2024-06-26T07:05:31.5726395Z { 2024-06-26T07:05:31.5727060Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5727808Z "size": 55728572, 2024-06-26T07:05:31.5728523Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-06-26T07:05:31.5729340Z }, 2024-06-26T07:05:31.5729670Z { 2024-06-26T07:05:31.5730242Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5731023Z "size": 186, 2024-06-26T07:05:31.5731751Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-06-26T07:05:31.5732545Z }, 2024-06-26T07:05:31.5732895Z { 2024-06-26T07:05:31.5733351Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5733908Z "size": 6886, 2024-06-26T07:05:31.5734439Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-06-26T07:05:31.5735283Z }, 2024-06-26T07:05:31.5735535Z { 2024-06-26T07:05:31.5735967Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5736526Z "size": 1329001756, 2024-06-26T07:05:31.5737099Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-06-26T07:05:31.5737725Z }, 2024-06-26T07:05:31.5737971Z { 2024-06-26T07:05:31.5738402Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5738946Z "size": 62501, 2024-06-26T07:05:31.5739480Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-06-26T07:05:31.5740087Z }, 2024-06-26T07:05:31.5740337Z { 2024-06-26T07:05:31.5740761Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5741303Z "size": 1684, 2024-06-26T07:05:31.5741828Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-06-26T07:05:31.5742431Z }, 2024-06-26T07:05:31.5742680Z { 2024-06-26T07:05:31.5743114Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5743666Z "size": 1523, 2024-06-26T07:05:31.5744221Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-06-26T07:05:31.5744833Z }, 2024-06-26T07:05:31.5745183Z { 2024-06-26T07:05:31.5745617Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5746159Z "size": 2528295403, 2024-06-26T07:05:31.5746720Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-06-26T07:05:31.5747336Z }, 2024-06-26T07:05:31.5747590Z { 2024-06-26T07:05:31.5748132Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5748675Z "size": 86016, 2024-06-26T07:05:31.5749230Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-06-26T07:05:31.5749840Z }, 2024-06-26T07:05:31.5750080Z { 2024-06-26T07:05:31.5750511Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5751061Z "size": 1823, 2024-06-26T07:05:31.5751593Z "digest": "sha256:309b872da805fcdf74137a887464876fa0615dc8b24199307119eb65fc2bfb49" 2024-06-26T07:05:31.5752228Z }, 2024-06-26T07:05:31.5752480Z { 2024-06-26T07:05:31.5752913Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5753448Z "size": 246737542, 2024-06-26T07:05:31.5753995Z "digest": "sha256:e82932cb002c06a782077b8e5965f8733eb8563406a23cab260d9d8c4a994f7d" 2024-06-26T07:05:31.5754594Z }, 2024-06-26T07:05:31.5754844Z { 2024-06-26T07:05:31.5755263Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5755802Z "size": 545, 2024-06-26T07:05:31.5756353Z "digest": "sha256:ca9dd73e3e9378225ebdcfd12199ef44649e34fe55003cfbc8afe2e8a4527c5b" 2024-06-26T07:05:31.5756966Z }, 2024-06-26T07:05:31.5757214Z { 2024-06-26T07:05:31.5757648Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5758195Z "size": 1283, 2024-06-26T07:05:31.5758720Z "digest": "sha256:16d76a79d5d6616e228537785ec83a869f68a402520bbc6e826201621ca6cb98" 2024-06-26T07:05:31.5759303Z }, 2024-06-26T07:05:31.5759551Z { 2024-06-26T07:05:31.5760123Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5760668Z "size": 485, 2024-06-26T07:05:31.5761204Z "digest": "sha256:4bdf41c9ecbe1651e0986ccbea4663a0556af27b7145a679cfa45c35dc96b891" 2024-06-26T07:05:31.5761822Z }, 2024-06-26T07:05:31.5762078Z { 2024-06-26T07:05:31.5762493Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5763051Z "size": 91710937, 2024-06-26T07:05:31.5763815Z "digest": "sha256:f4863801e30cdbcd1d4e6cb1cbbcde7dbcaf6ae317ca80429295741ce60ac31d" 2024-06-26T07:05:31.5764457Z }, 2024-06-26T07:05:31.5764699Z { 2024-06-26T07:05:31.5765140Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5765805Z "size": 3233, 2024-06-26T07:05:31.5766353Z "digest": "sha256:d37dd9ed8948a0d02e1198bbedadbca36477a0980dd553196385aab0c025581d" 2024-06-26T07:05:31.5766956Z }, 2024-06-26T07:05:31.5767206Z { 2024-06-26T07:05:31.5767637Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5768178Z "size": 1905, 2024-06-26T07:05:31.5768730Z "digest": "sha256:cf2f34f225dbed17c1a9a8b9ea4ad030aba797774a716dc0a82fe212324aa751" 2024-06-26T07:05:31.5769350Z }, 2024-06-26T07:05:31.5769602Z { 2024-06-26T07:05:31.5770022Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5770568Z "size": 702, 2024-06-26T07:05:31.5771108Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:05:31.5771715Z }, 2024-06-26T07:05:31.5771956Z { 2024-06-26T07:05:31.5772385Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5772933Z "size": 2784789002, 2024-06-26T07:05:31.5773491Z "digest": "sha256:3d914950f8addfc4d657a2365c76e1143eb80b710cd3f6e9ae05640217fc6347" 2024-06-26T07:05:31.5774089Z }, 2024-06-26T07:05:31.5774345Z { 2024-06-26T07:05:31.5774774Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5775368Z "size": 381, 2024-06-26T07:05:31.5775898Z "digest": "sha256:96d0e2032184c57e62a652aaffcf0241f0d0c7a23e894ca0ad192c4a64a36583" 2024-06-26T07:05:31.5776500Z }, 2024-06-26T07:05:31.5776758Z { 2024-06-26T07:05:31.5777187Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5777744Z "size": 11879, 2024-06-26T07:05:31.5778285Z "digest": "sha256:06e12ce59790cb8b091e5d619c2a62a28329008ee184a8b9a562e3db12471f30" 2024-06-26T07:05:31.5778884Z }, 2024-06-26T07:05:31.5779137Z { 2024-06-26T07:05:31.5779584Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5780136Z "size": 803, 2024-06-26T07:05:31.5780687Z "digest": "sha256:ed712f63cde7070bdec2a6179c4c35e752c6991ddbc69fe9f2724a9903250ceb" 2024-06-26T07:05:31.5781305Z }, 2024-06-26T07:05:31.5781565Z { 2024-06-26T07:05:31.5782006Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5782564Z "size": 106, 2024-06-26T07:05:31.5783142Z "digest": "sha256:a97a71ef8a4e66234dfc2098834f8395ac5630240e60164d1a251fdb0115ba85" 2024-06-26T07:05:31.5783743Z }, 2024-06-26T07:05:31.5784000Z { 2024-06-26T07:05:31.5784425Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5784977Z "size": 503, 2024-06-26T07:05:31.5785526Z "digest": "sha256:25cd0c482ad612f936b35b4f220c580e16be982226aac8801a6a849a6bfdc221" 2024-06-26T07:05:31.5786136Z }, 2024-06-26T07:05:31.5786388Z { 2024-06-26T07:05:31.5786824Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5787383Z "size": 121477177, 2024-06-26T07:05:31.5788010Z "digest": "sha256:9bf94bc456584365fda21d0ed83b7cc690b08a71727b71c4c1e667e15028a386" 2024-06-26T07:05:31.5788627Z }, 2024-06-26T07:05:31.5788883Z { 2024-06-26T07:05:31.5789320Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5789859Z "size": 109, 2024-06-26T07:05:31.5790396Z "digest": "sha256:de45a14c9b227e4b0c9d45d42223b50a71c03522ddd9e2ca0d9517835206c823" 2024-06-26T07:05:31.5791002Z }, 2024-06-26T07:05:31.5791264Z { 2024-06-26T07:05:31.5791693Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5792283Z "size": 491, 2024-06-26T07:05:31.5792850Z "digest": "sha256:e99241e77209b0b35a0fb59a61c0821fc161190178af7e4af45e6eeaa043375e" 2024-06-26T07:05:31.5793443Z }, 2024-06-26T07:05:31.5793703Z { 2024-06-26T07:05:31.5794150Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5794705Z "size": 291, 2024-06-26T07:05:31.5795257Z "digest": "sha256:35e8f4e2a42f9df0c969dedd2199cf71719cf5ecfa676dfa014bbdeb4fdb295f" 2024-06-26T07:05:31.5795959Z }, 2024-06-26T07:05:31.5796217Z { 2024-06-26T07:05:31.5796662Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5797210Z "size": 101, 2024-06-26T07:05:31.5797751Z "digest": "sha256:a823ffa4c541394b52953391dc0a6867e26c37fe4a71fa022a2cc3f07f0ba70d" 2024-06-26T07:05:31.5798375Z }, 2024-06-26T07:05:31.5798623Z { 2024-06-26T07:05:31.5799061Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5799618Z "size": 1473, 2024-06-26T07:05:31.5800152Z "digest": "sha256:44ba272eae2064603605a6238b3f35cc6578859d660068e702a047a9cd8ab2fd" 2024-06-26T07:05:31.5800743Z }, 2024-06-26T07:05:31.5801002Z { 2024-06-26T07:05:31.5801445Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5802013Z "size": 446710781, 2024-06-26T07:05:31.5802569Z "digest": "sha256:da02d278e5240c285bc855ce4e42cd6121d5b903787f5c14f03e575ad1268650" 2024-06-26T07:05:31.5803193Z }, 2024-06-26T07:05:31.5803456Z { 2024-06-26T07:05:31.5804059Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5804609Z "size": 161, 2024-06-26T07:05:31.5805146Z "digest": "sha256:73a0ce681c295135456e190578733273276b38b4dd5babbb39298f384d43b1c6" 2024-06-26T07:05:31.5805738Z }, 2024-06-26T07:05:31.5806092Z { 2024-06-26T07:05:31.5806540Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5807104Z "size": 564, 2024-06-26T07:05:31.5807660Z "digest": "sha256:49737bd5ae709dbf7956e21fc8b05ba009e9e6c9b130b0126a73bd3d3f5efc78" 2024-06-26T07:05:31.5808257Z }, 2024-06-26T07:05:31.5808518Z { 2024-06-26T07:05:31.5808963Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5809524Z "size": 35874600, 2024-06-26T07:05:31.5810082Z "digest": "sha256:9cd4c745f8e4f9d5c9bb8050923575afad21f5b7852975a4a1981bc9eff282c5" 2024-06-26T07:05:31.5810702Z }, 2024-06-26T07:05:31.5810966Z { 2024-06-26T07:05:31.5811407Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5811964Z "size": 104, 2024-06-26T07:05:31.5812498Z "digest": "sha256:b2950a0365345d4c17a48627c905fd8930b3b3154c727f0526eb9ca547102bb7" 2024-06-26T07:05:31.5813095Z }, 2024-06-26T07:05:31.5813342Z { 2024-06-26T07:05:31.5813786Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5814341Z "size": 425, 2024-06-26T07:05:31.5814869Z "digest": "sha256:25e19d81d9fc081962ff4cbc09e098f4035419822712c19a410f5282300dfb9e" 2024-06-26T07:05:31.5815454Z }, 2024-06-26T07:05:31.5815718Z { 2024-06-26T07:05:31.5816149Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5816706Z "size": 20262084, 2024-06-26T07:05:31.5817262Z "digest": "sha256:60156b40fcbec2c25f0170b0a82fa8cd9303e0daae8bb1811fd0fc6562e88e8e" 2024-06-26T07:05:31.5817881Z }, 2024-06-26T07:05:31.5818138Z { 2024-06-26T07:05:31.5818582Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5819141Z "size": 439, 2024-06-26T07:05:31.5819666Z "digest": "sha256:7a868239227fd19b30a19927bbc8f058d247403fe5b343c55f4b4b8987918911" 2024-06-26T07:05:31.5820272Z }, 2024-06-26T07:05:31.5820520Z { 2024-06-26T07:05:31.5820958Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5821517Z "size": 702, 2024-06-26T07:05:31.5822065Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:05:31.5822718Z }, 2024-06-26T07:05:31.5822977Z { 2024-06-26T07:05:31.5823418Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5823957Z "size": 143, 2024-06-26T07:05:31.5824493Z "digest": "sha256:83911d7c7ecf11b5932bb7449975c52e7f737a5b867691935e1ccb85ceb25b52" 2024-06-26T07:05:31.5825099Z }, 2024-06-26T07:05:31.5825362Z { 2024-06-26T07:05:31.5825795Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5826434Z "size": 135, 2024-06-26T07:05:31.5826980Z "digest": "sha256:b3c5d66c679f58c71febdd502889bd8cd3d36e5001754cdf1e30eabe21135cce" 2024-06-26T07:05:31.5827594Z }, 2024-06-26T07:05:31.5827920Z { 2024-06-26T07:05:31.5828364Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5828927Z "size": 4911495247, 2024-06-26T07:05:31.5829493Z "digest": "sha256:84b2c7a899820f6acdb22148c5e8346d8b26c33269fd01842c9573515251e1e1" 2024-06-26T07:05:31.5830082Z }, 2024-06-26T07:05:31.5830334Z { 2024-06-26T07:05:31.5830766Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5831313Z "size": 190, 2024-06-26T07:05:31.5831861Z "digest": "sha256:c1068d03a811d8dead5da9e82fdc8869205fc4ee9cb92d4f7a4acfbb5ba83950" 2024-06-26T07:05:31.5832475Z }, 2024-06-26T07:05:31.5832766Z { 2024-06-26T07:05:31.5833195Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5833737Z "size": 565, 2024-06-26T07:05:31.5834269Z "digest": "sha256:671b4a301d81074fca17d36851f81801552c4ecb2f90ff3412aa9ab6a1a91371" 2024-06-26T07:05:31.5834876Z }, 2024-06-26T07:05:31.5835118Z { 2024-06-26T07:05:31.5835549Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5836105Z "size": 43163095, 2024-06-26T07:05:31.5836715Z "digest": "sha256:67760cf5ec46ab58d3ca98637fd9b26cb47a67446bfd0db176fe5e023606aa0d" 2024-06-26T07:05:31.5837332Z }, 2024-06-26T07:05:31.5837587Z { 2024-06-26T07:05:31.5838019Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5838561Z "size": 106, 2024-06-26T07:05:31.5839099Z "digest": "sha256:ddc9740bcea2b301e99b50525059aea8d9c7121a1c8eab566561a8968438a8c2" 2024-06-26T07:05:31.5839709Z }, 2024-06-26T07:05:31.5839961Z { 2024-06-26T07:05:31.5840397Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5840971Z "size": 1213, 2024-06-26T07:05:31.5841509Z "digest": "sha256:fb2e977685609fcc01de170ba7f50a573e73e6520e47253717e92a28952c5ee8" 2024-06-26T07:05:31.5842108Z }, 2024-06-26T07:05:31.5842357Z { 2024-06-26T07:05:31.5842797Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5843358Z "size": 702, 2024-06-26T07:05:31.5844049Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:05:31.5844676Z }, 2024-06-26T07:05:31.5844935Z { 2024-06-26T07:05:31.5845368Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5845909Z "size": 138, 2024-06-26T07:05:31.5846454Z "digest": "sha256:b599c5a8c7cb461cbd7b3c98b99960b8eb04506e05e79db1cc0cfadb14a47e91" 2024-06-26T07:05:31.5847074Z }, 2024-06-26T07:05:31.5847337Z { 2024-06-26T07:05:31.5847760Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5848313Z "size": 120, 2024-06-26T07:05:31.5848859Z "digest": "sha256:e08e88c1366827455ee7ddbf92b6ae615ceb3cbd3fb11cd03ddd31bd3fa94a9c" 2024-06-26T07:05:31.5849467Z }, 2024-06-26T07:05:31.5849709Z { 2024-06-26T07:05:31.5850139Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5850682Z "size": 1904157495, 2024-06-26T07:05:31.5851292Z "digest": "sha256:12ba7431ff38de8cf6af3052a2a52e79f4e712b8c99f5bafde3c20e63b1f570f" 2024-06-26T07:05:31.5852087Z }, 2024-06-26T07:05:31.5852354Z { 2024-06-26T07:05:31.5852794Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5853340Z "size": 172, 2024-06-26T07:05:31.5853878Z "digest": "sha256:cf7c2cb896cba6a594028cc0fd246e127a2f4481d48905a3aa12c5f5872332d4" 2024-06-26T07:05:31.5854659Z }, 2024-06-26T07:05:31.5854949Z { 2024-06-26T07:05:31.5855389Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5855928Z "size": 908, 2024-06-26T07:05:31.5856471Z "digest": "sha256:3a778ddfac2af5b3a11f49ddf2f3f1ad1731f74ba691f434ab69966d1e2a189f" 2024-06-26T07:05:31.5857186Z }, 2024-06-26T07:05:31.5857437Z { 2024-06-26T07:05:31.5857864Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5858399Z "size": 702, 2024-06-26T07:05:31.5858931Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:05:31.5859546Z }, 2024-06-26T07:05:31.5859791Z { 2024-06-26T07:05:31.5860203Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5860744Z "size": 134, 2024-06-26T07:05:31.5861273Z "digest": "sha256:236c8f0245f4b40616a4322eaf1e1b3e178d5990db1fcaba39622ea4e79c06a1" 2024-06-26T07:05:31.5861873Z }, 2024-06-26T07:05:31.5862113Z { 2024-06-26T07:05:31.5862537Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5863079Z "size": 32, 2024-06-26T07:05:31.5863609Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:05:31.5864210Z }, 2024-06-26T07:05:31.5864479Z { 2024-06-26T07:05:31.5864905Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5865437Z "size": 156, 2024-06-26T07:05:31.5865974Z "digest": "sha256:dda1ce302b9a9df0c8f1145d2e3eaa03c96909bb45a792451f1fcb555b06738d" 2024-06-26T07:05:31.5866587Z }, 2024-06-26T07:05:31.5866917Z { 2024-06-26T07:05:31.5867332Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5867936Z "size": 1841, 2024-06-26T07:05:31.5868478Z "digest": "sha256:e797fe86caef19319e03f8dca92cf96838ab3bb3644f4447c6dfc04230118afa" 2024-06-26T07:05:31.5869089Z }, 2024-06-26T07:05:31.5869329Z { 2024-06-26T07:05:31.5869768Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5870309Z "size": 7529784, 2024-06-26T07:05:31.5870847Z "digest": "sha256:967190076fc0ab2bb2fc0d74e58edd3d662a537d702ff3f8b6e876230154a16f" 2024-06-26T07:05:31.5871451Z }, 2024-06-26T07:05:31.5871705Z { 2024-06-26T07:05:31.5872143Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5872735Z "size": 165, 2024-06-26T07:05:31.5873266Z "digest": "sha256:4fb0a83f06855534ef82b80dcf7c685303edb2bcbd446196066dc2ad2dd71281" 2024-06-26T07:05:31.5873872Z }, 2024-06-26T07:05:31.5874119Z { 2024-06-26T07:05:31.5874546Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5875094Z "size": 7944, 2024-06-26T07:05:31.5875640Z "digest": "sha256:68af2ec4e30a33f26947ece9f3a31b7d3b04c2bca5865d4b48618457832a1da2" 2024-06-26T07:05:31.5876249Z }, 2024-06-26T07:05:31.5876488Z { 2024-06-26T07:05:31.5876918Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5877464Z "size": 8070, 2024-06-26T07:05:31.5877993Z "digest": "sha256:cc91933e985a044c6df53caa15488d2285896f5e991f38ec068c253ceec53f14" 2024-06-26T07:05:31.5878592Z }, 2024-06-26T07:05:31.5878844Z { 2024-06-26T07:05:31.5879282Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5879814Z "size": 303, 2024-06-26T07:05:31.5880357Z "digest": "sha256:9f3187916e7dcad33cc572f8f0e949d2f507b887f99cfd6acefd14826b44960d" 2024-06-26T07:05:31.5880964Z }, 2024-06-26T07:05:31.5881218Z { 2024-06-26T07:05:31.5881635Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5882184Z "size": 7629071, 2024-06-26T07:05:31.5882765Z "digest": "sha256:52608399485738d10d779254946c9e1d536010dd45593c69bf1eb9f62f85dc24" 2024-06-26T07:05:31.5883348Z }, 2024-06-26T07:05:31.5883758Z { 2024-06-26T07:05:31.5884214Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5884762Z "size": 108, 2024-06-26T07:05:31.5885291Z "digest": "sha256:05a6f6dcc1c00bac6660923a22adc7d7c34416718f8c89daf9b3351d9365f694" 2024-06-26T07:05:31.5885895Z }, 2024-06-26T07:05:31.5886148Z { 2024-06-26T07:05:31.5886578Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5887207Z "size": 54145775, 2024-06-26T07:05:31.5887810Z "digest": "sha256:6bc9dc8315b09b27c976bfba9700767ba412d72fd0528b18bbe8f8e62005ec22" 2024-06-26T07:05:31.5888486Z }, 2024-06-26T07:05:31.5888745Z { 2024-06-26T07:05:31.5889195Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5889800Z "size": 474, 2024-06-26T07:05:31.5890389Z "digest": "sha256:1b0e27c35f07b3fbf0aeb548d1949bf1d60a420136da21789d94d196c891d981" 2024-06-26T07:05:31.5891057Z }, 2024-06-26T07:05:31.5891320Z { 2024-06-26T07:05:31.5891765Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5892327Z "size": 1374858747, 2024-06-26T07:05:31.5892919Z "digest": "sha256:95237e7311a1c5ce6937db7196c866632f44e8788514a5c0eb4a3f04da4017e6" 2024-06-26T07:05:31.5893523Z }, 2024-06-26T07:05:31.5893785Z { 2024-06-26T07:05:31.5894222Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5894755Z "size": 106, 2024-06-26T07:05:31.5895314Z "digest": "sha256:789eff6c51aada70bf75eb1acafed63c0d839b768638001500616ff6e53b854a" 2024-06-26T07:05:31.5895941Z }, 2024-06-26T07:05:31.5896191Z { 2024-06-26T07:05:31.5896622Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5897180Z "size": 558, 2024-06-26T07:05:31.5897802Z "digest": "sha256:de0f8a8c45c6077b3c5ba3b84d7f07de98353fd618dad9a1e2f368ffe00493c7" 2024-06-26T07:05:31.5898413Z }, 2024-06-26T07:05:31.5898670Z { 2024-06-26T07:05:31.5899103Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5899661Z "size": 46248513, 2024-06-26T07:05:31.5900211Z "digest": "sha256:2060028f74888d218bebdd10d723f66c33ee1c708935c4ffd4654d34912ac4dc" 2024-06-26T07:05:31.5900828Z }, 2024-06-26T07:05:31.5901086Z { 2024-06-26T07:05:31.5901512Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5902046Z "size": 111, 2024-06-26T07:05:31.5902621Z "digest": "sha256:e412c7f3b2ded440992955bd92512345cbece8a8894cd3592f714ebc758ad1a0" 2024-06-26T07:05:31.5903249Z }, 2024-06-26T07:05:31.5903488Z { 2024-06-26T07:05:31.5903921Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5904460Z "size": 32, 2024-06-26T07:05:31.5904993Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:05:31.5905602Z }, 2024-06-26T07:05:31.5905851Z { 2024-06-26T07:05:31.5906273Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5906816Z "size": 32, 2024-06-26T07:05:31.5907337Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:05:31.5908026Z }, 2024-06-26T07:05:31.5908276Z { 2024-06-26T07:05:31.5908692Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5909228Z "size": 32, 2024-06-26T07:05:31.5909759Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:05:31.5910363Z }, 2024-06-26T07:05:31.5910601Z { 2024-06-26T07:05:31.5911026Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:05:31.5911568Z "size": 32, 2024-06-26T07:05:31.5912135Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:05:31.5912754Z } 2024-06-26T07:05:31.5913005Z ] 2024-06-26T07:05:31.5913253Z } 2024-06-26T07:05:31.6029004Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:05:31.6029562Z tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:05:31.6030212Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-06-26T07:05:31.6038360Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:31.6038852Z env: 2024-06-26T07:05:31.6039135Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:31.6040267Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:31.6041411Z ##[endgroup] 2024-06-26T07:05:31.6064038Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:31.6111271Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-06-26T07:05:31.6111917Z with: 2024-06-26T07:05:31.6113101Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:31.6114386Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:31.6114905Z env: 2024-06-26T07:05:31.6115191Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:31.6115547Z ##[endgroup] 2024-06-26T07:05:31.6131070Z ##[group]Run set -x 2024-06-26T07:05:31.6131417Z set -x 2024-06-26T07:05:31.6131723Z set +e 2024-06-26T07:05:31.6132035Z  2024-06-26T07:05:31.6132310Z login() { 2024-06-26T07:05:31.6132970Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:05:31.6133793Z } 2024-06-26T07:05:31.6134083Z  2024-06-26T07:05:31.6134395Z retry () { 2024-06-26T07:05:31.6134900Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:05:31.6135343Z } 2024-06-26T07:05:31.6135628Z  2024-06-26T07:05:31.6135950Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:05:31.6136362Z  2024-06-26T07:05:31.6136644Z set -e 2024-06-26T07:05:31.6137124Z # ignore output since only exit code is used for conditional 2024-06-26T07:05:31.6137828Z # only pull docker image if it's not available locally 2024-06-26T07:05:31.6138586Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-06-26T07:05:31.6139279Z  retry docker pull "${DOCKER_IMAGE}" 2024-06-26T07:05:31.6139737Z fi 2024-06-26T07:05:31.6146469Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:05:31.6146969Z env: 2024-06-26T07:05:31.6147264Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:05:31.6148472Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:31.6149763Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:31.6150288Z ##[endgroup] 2024-06-26T07:05:31.6168502Z + set +e 2024-06-26T07:05:31.6169174Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:31.6169836Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:31.6171434Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:05:31.6172378Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:05:32.0305304Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:05:32.0306190Z Configure a credential helper to remove this warning. See 2024-06-26T07:05:32.0307200Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:05:32.0307733Z 2024-06-26T07:05:32.0307952Z Login Succeeded 2024-06-26T07:05:32.0315517Z + set -e 2024-06-26T07:05:32.0316799Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:32.0438040Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:32.0440116Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:05:32.2466388Z 91382da70d5719cd7007b6b80b71d2f48398f6b7: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-06-26T07:05:32.2474202Z 7a2c55901189: Pulling fs layer 2024-06-26T07:05:32.2475086Z 224fe954d725: Pulling fs layer 2024-06-26T07:05:32.2476405Z 75722010b82e: Pulling fs layer 2024-06-26T07:05:32.2477267Z d527cbbb87e3: Pulling fs layer 2024-06-26T07:05:32.2477884Z b57676e46aee: Pulling fs layer 2024-06-26T07:05:32.2478660Z a8c1e85b5e14: Pulling fs layer 2024-06-26T07:05:32.2479212Z a41a8d1c11c8: Pulling fs layer 2024-06-26T07:05:32.2479742Z 0c1227890755: Pulling fs layer 2024-06-26T07:05:32.2480340Z d8d1234baab3: Pulling fs layer 2024-06-26T07:05:32.2480876Z 7ed32bc8e469: Pulling fs layer 2024-06-26T07:05:32.2481430Z ec1e7978c1fe: Pulling fs layer 2024-06-26T07:05:32.2481967Z 309b872da805: Pulling fs layer 2024-06-26T07:05:32.2482534Z e82932cb002c: Pulling fs layer 2024-06-26T07:05:32.2483044Z ca9dd73e3e93: Pulling fs layer 2024-06-26T07:05:32.2483421Z 16d76a79d5d6: Pulling fs layer 2024-06-26T07:05:32.2484119Z 4bdf41c9ecbe: Pulling fs layer 2024-06-26T07:05:32.2484621Z a8c1e85b5e14: Waiting 2024-06-26T07:05:32.2485072Z 0c1227890755: Waiting 2024-06-26T07:05:32.2485484Z e82932cb002c: Waiting 2024-06-26T07:05:32.2485805Z a41a8d1c11c8: Waiting 2024-06-26T07:05:32.2486139Z f4863801e30c: Pulling fs layer 2024-06-26T07:05:32.2486638Z ec1e7978c1fe: Waiting 2024-06-26T07:05:32.2486975Z d37dd9ed8948: Pulling fs layer 2024-06-26T07:05:32.2487344Z cf2f34f225db: Pulling fs layer 2024-06-26T07:05:32.2487720Z ca9dd73e3e93: Waiting 2024-06-26T07:05:32.2488041Z d8d1234baab3: Waiting 2024-06-26T07:05:32.2488383Z df546d7b3e34: Pulling fs layer 2024-06-26T07:05:32.2488752Z 3d914950f8ad: Pulling fs layer 2024-06-26T07:05:32.2489114Z 309b872da805: Waiting 2024-06-26T07:05:32.2489437Z 4bdf41c9ecbe: Waiting 2024-06-26T07:05:32.2489752Z f4863801e30c: Waiting 2024-06-26T07:05:32.2490056Z 7ed32bc8e469: Waiting 2024-06-26T07:05:32.2490382Z 96d0e2032184: Pulling fs layer 2024-06-26T07:05:32.2490744Z d527cbbb87e3: Waiting 2024-06-26T07:05:32.2491067Z 06e12ce59790: Pulling fs layer 2024-06-26T07:05:32.2491428Z d37dd9ed8948: Waiting 2024-06-26T07:05:32.2491753Z ed712f63cde7: Pulling fs layer 2024-06-26T07:05:32.2492117Z b57676e46aee: Waiting 2024-06-26T07:05:32.2492445Z a97a71ef8a4e: Pulling fs layer 2024-06-26T07:05:32.2492818Z 25cd0c482ad6: Pulling fs layer 2024-06-26T07:05:32.2493184Z ed712f63cde7: Waiting 2024-06-26T07:05:32.2493487Z df546d7b3e34: Waiting 2024-06-26T07:05:32.2493812Z 9bf94bc45658: Pulling fs layer 2024-06-26T07:05:32.2494164Z 06e12ce59790: Waiting 2024-06-26T07:05:32.2494469Z 96d0e2032184: Waiting 2024-06-26T07:05:32.2494786Z de45a14c9b22: Pulling fs layer 2024-06-26T07:05:32.2495155Z e99241e77209: Pulling fs layer 2024-06-26T07:05:32.2495529Z 35e8f4e2a42f: Pulling fs layer 2024-06-26T07:05:32.2495879Z 16d76a79d5d6: Waiting 2024-06-26T07:05:32.2496210Z a823ffa4c541: Pulling fs layer 2024-06-26T07:05:32.2496612Z 9bf94bc45658: Waiting 2024-06-26T07:05:32.2496943Z 44ba272eae20: Pulling fs layer 2024-06-26T07:05:32.2497318Z da02d278e524: Pulling fs layer 2024-06-26T07:05:32.2497661Z e99241e77209: Waiting 2024-06-26T07:05:32.2497969Z de45a14c9b22: Waiting 2024-06-26T07:05:32.2498293Z 73a0ce681c29: Pulling fs layer 2024-06-26T07:05:32.2498668Z 49737bd5ae70: Pulling fs layer 2024-06-26T07:05:32.2499026Z 35e8f4e2a42f: Waiting 2024-06-26T07:05:32.2499341Z cf2f34f225db: Waiting 2024-06-26T07:05:32.2499668Z 9cd4c745f8e4: Pulling fs layer 2024-06-26T07:05:32.2500035Z b2950a036534: Pulling fs layer 2024-06-26T07:05:32.2500395Z 25e19d81d9fc: Pulling fs layer 2024-06-26T07:05:32.2500766Z 60156b40fcbe: Pulling fs layer 2024-06-26T07:05:32.2501119Z a97a71ef8a4e: Waiting 2024-06-26T07:05:32.2501433Z 7a868239227f: Pulling fs layer 2024-06-26T07:05:32.2501787Z b2950a036534: Waiting 2024-06-26T07:05:32.2502100Z 9cd4c745f8e4: Waiting 2024-06-26T07:05:32.2502416Z a823ffa4c541: Waiting 2024-06-26T07:05:32.2502719Z 60156b40fcbe: Waiting 2024-06-26T07:05:32.2503026Z 7a868239227f: Waiting 2024-06-26T07:05:32.2503338Z 73a0ce681c29: Waiting 2024-06-26T07:05:32.2503648Z 44ba272eae20: Waiting 2024-06-26T07:05:32.2503951Z 25cd0c482ad6: Waiting 2024-06-26T07:05:32.2504274Z 83911d7c7ecf: Pulling fs layer 2024-06-26T07:05:32.2504631Z da02d278e524: Waiting 2024-06-26T07:05:32.2505047Z b3c5d66c679f: Pulling fs layer 2024-06-26T07:05:32.2505423Z 84b2c7a89982: Pulling fs layer 2024-06-26T07:05:32.2505798Z c1068d03a811: Pulling fs layer 2024-06-26T07:05:32.2506163Z 671b4a301d81: Pulling fs layer 2024-06-26T07:05:32.2506507Z 84b2c7a89982: Waiting 2024-06-26T07:05:32.2506825Z b3c5d66c679f: Waiting 2024-06-26T07:05:32.2507153Z 67760cf5ec46: Pulling fs layer 2024-06-26T07:05:32.2507506Z 83911d7c7ecf: Waiting 2024-06-26T07:05:32.2507957Z c1068d03a811: Waiting 2024-06-26T07:05:32.2508285Z ddc9740bcea2: Pulling fs layer 2024-06-26T07:05:32.2508658Z fb2e97768560: Pulling fs layer 2024-06-26T07:05:32.2509019Z b599c5a8c7cb: Pulling fs layer 2024-06-26T07:05:32.2509390Z e08e88c13668: Pulling fs layer 2024-06-26T07:05:32.2509737Z fb2e97768560: Waiting 2024-06-26T07:05:32.2510065Z 12ba7431ff38: Pulling fs layer 2024-06-26T07:05:32.2510426Z cf7c2cb896cb: Pulling fs layer 2024-06-26T07:05:32.2510783Z 67760cf5ec46: Waiting 2024-06-26T07:05:32.2511094Z e08e88c13668: Waiting 2024-06-26T07:05:32.2511417Z 3a778ddfac2a: Pulling fs layer 2024-06-26T07:05:32.2511776Z 12ba7431ff38: Waiting 2024-06-26T07:05:32.2512151Z b599c5a8c7cb: Waiting 2024-06-26T07:05:32.2512467Z cf7c2cb896cb: Waiting 2024-06-26T07:05:32.2512788Z 236c8f0245f4: Pulling fs layer 2024-06-26T07:05:32.2513166Z 4f4fb700ef54: Pulling fs layer 2024-06-26T07:05:32.2513541Z dda1ce302b9a: Pulling fs layer 2024-06-26T07:05:32.2513915Z e797fe86caef: Pulling fs layer 2024-06-26T07:05:32.2514263Z 4f4fb700ef54: Waiting 2024-06-26T07:05:32.2514572Z 236c8f0245f4: Waiting 2024-06-26T07:05:32.2514886Z dda1ce302b9a: Waiting 2024-06-26T07:05:32.2515199Z e797fe86caef: Waiting 2024-06-26T07:05:32.2515523Z 967190076fc0: Pulling fs layer 2024-06-26T07:05:32.2515877Z ddc9740bcea2: Waiting 2024-06-26T07:05:32.2516208Z 4fb0a83f0685: Pulling fs layer 2024-06-26T07:05:32.2516575Z 68af2ec4e30a: Pulling fs layer 2024-06-26T07:05:32.2516946Z cc91933e985a: Pulling fs layer 2024-06-26T07:05:32.2517301Z 3a778ddfac2a: Waiting 2024-06-26T07:05:32.2517631Z 9f3187916e7d: Pulling fs layer 2024-06-26T07:05:32.2518019Z 526083994857: Pulling fs layer 2024-06-26T07:05:32.2518390Z 68af2ec4e30a: Waiting 2024-06-26T07:05:32.2518709Z 4fb0a83f0685: Waiting 2024-06-26T07:05:32.2519017Z 967190076fc0: Waiting 2024-06-26T07:05:32.2519343Z 05a6f6dcc1c0: Pulling fs layer 2024-06-26T07:05:32.2519705Z cc91933e985a: Waiting 2024-06-26T07:05:32.2520033Z 6bc9dc8315b0: Pulling fs layer 2024-06-26T07:05:32.2520377Z 526083994857: Waiting 2024-06-26T07:05:32.2520705Z 1b0e27c35f07: Pulling fs layer 2024-06-26T07:05:32.2521064Z 9f3187916e7d: Waiting 2024-06-26T07:05:32.2521367Z 6bc9dc8315b0: Waiting 2024-06-26T07:05:32.2521698Z 95237e7311a1: Pulling fs layer 2024-06-26T07:05:32.2522065Z 1b0e27c35f07: Waiting 2024-06-26T07:05:32.2522393Z 789eff6c51aa: Pulling fs layer 2024-06-26T07:05:32.2522765Z de0f8a8c45c6: Pulling fs layer 2024-06-26T07:05:32.2523138Z 2060028f7488: Pulling fs layer 2024-06-26T07:05:32.2523506Z e412c7f3b2de: Pulling fs layer 2024-06-26T07:05:32.2524103Z 789eff6c51aa: Waiting 2024-06-26T07:05:32.2524419Z e412c7f3b2de: Waiting 2024-06-26T07:05:32.2524733Z de0f8a8c45c6: Waiting 2024-06-26T07:05:32.2525046Z 2060028f7488: Waiting 2024-06-26T07:05:32.4112979Z 224fe954d725: Verifying Checksum 2024-06-26T07:05:32.4113392Z 224fe954d725: Download complete 2024-06-26T07:05:32.4928141Z d527cbbb87e3: Verifying Checksum 2024-06-26T07:05:32.4928548Z d527cbbb87e3: Download complete 2024-06-26T07:05:32.5706174Z b57676e46aee: Verifying Checksum 2024-06-26T07:05:32.5706588Z b57676e46aee: Download complete 2024-06-26T07:05:32.6020529Z 7a2c55901189: Verifying Checksum 2024-06-26T07:05:32.6021021Z 7a2c55901189: Download complete 2024-06-26T07:05:32.6760216Z a41a8d1c11c8: Download complete 2024-06-26T07:05:32.7487580Z 0c1227890755: Verifying Checksum 2024-06-26T07:05:32.7488007Z 0c1227890755: Download complete 2024-06-26T07:05:32.8643452Z d8d1234baab3: Verifying Checksum 2024-06-26T07:05:32.8644060Z d8d1234baab3: Download complete 2024-06-26T07:05:32.8844996Z 75722010b82e: Verifying Checksum 2024-06-26T07:05:32.8845693Z 75722010b82e: Download complete 2024-06-26T07:05:32.9722039Z ec1e7978c1fe: Verifying Checksum 2024-06-26T07:05:32.9722689Z ec1e7978c1fe: Download complete 2024-06-26T07:05:33.0606763Z 309b872da805: Verifying Checksum 2024-06-26T07:05:33.0607361Z 309b872da805: Download complete 2024-06-26T07:05:33.6361834Z 7a2c55901189: Pull complete 2024-06-26T07:05:33.9187899Z 224fe954d725: Pull complete 2024-06-26T07:05:34.8557056Z 75722010b82e: Pull complete 2024-06-26T07:05:34.9007602Z d527cbbb87e3: Pull complete 2024-06-26T07:05:34.9525039Z b57676e46aee: Pull complete 2024-06-26T07:05:35.5920564Z e82932cb002c: Verifying Checksum 2024-06-26T07:05:35.5921046Z e82932cb002c: Download complete 2024-06-26T07:05:35.6626920Z ca9dd73e3e93: Verifying Checksum 2024-06-26T07:05:35.6627455Z ca9dd73e3e93: Download complete 2024-06-26T07:05:35.7352725Z 16d76a79d5d6: Verifying Checksum 2024-06-26T07:05:35.7353318Z 16d76a79d5d6: Download complete 2024-06-26T07:05:35.8104026Z 4bdf41c9ecbe: Verifying Checksum 2024-06-26T07:05:35.8104713Z 4bdf41c9ecbe: Download complete 2024-06-26T07:05:36.7904097Z f4863801e30c: Verifying Checksum 2024-06-26T07:05:36.7905687Z f4863801e30c: Download complete 2024-06-26T07:05:36.8724097Z d37dd9ed8948: Verifying Checksum 2024-06-26T07:05:36.8724615Z d37dd9ed8948: Download complete 2024-06-26T07:05:36.9439924Z cf2f34f225db: Verifying Checksum 2024-06-26T07:05:36.9440438Z cf2f34f225db: Download complete 2024-06-26T07:05:37.0176659Z df546d7b3e34: Download complete 2024-06-26T07:05:45.9202155Z a8c1e85b5e14: Verifying Checksum 2024-06-26T07:05:45.9202624Z a8c1e85b5e14: Download complete 2024-06-26T07:05:45.9856794Z 96d0e2032184: Download complete 2024-06-26T07:05:46.0686070Z 06e12ce59790: Verifying Checksum 2024-06-26T07:05:46.0686643Z 06e12ce59790: Download complete 2024-06-26T07:05:46.1455079Z ed712f63cde7: Verifying Checksum 2024-06-26T07:05:46.1455608Z ed712f63cde7: Download complete 2024-06-26T07:05:46.2388234Z a97a71ef8a4e: Verifying Checksum 2024-06-26T07:05:46.2388828Z a97a71ef8a4e: Download complete 2024-06-26T07:05:46.3428988Z 25cd0c482ad6: Verifying Checksum 2024-06-26T07:05:46.3429421Z 25cd0c482ad6: Download complete 2024-06-26T07:05:47.6074259Z 9bf94bc45658: Verifying Checksum 2024-06-26T07:05:47.6074723Z 9bf94bc45658: Download complete 2024-06-26T07:05:47.6750700Z de45a14c9b22: Verifying Checksum 2024-06-26T07:05:47.7487099Z e99241e77209: Verifying Checksum 2024-06-26T07:05:47.7487666Z e99241e77209: Download complete 2024-06-26T07:05:47.8255731Z 35e8f4e2a42f: Verifying Checksum 2024-06-26T07:05:47.8256237Z 35e8f4e2a42f: Download complete 2024-06-26T07:05:47.8972901Z a823ffa4c541: Download complete 2024-06-26T07:05:47.9805375Z 44ba272eae20: Verifying Checksum 2024-06-26T07:05:47.9805873Z 44ba272eae20: Download complete 2024-06-26T07:05:52.5059634Z da02d278e524: Verifying Checksum 2024-06-26T07:05:52.5060326Z da02d278e524: Download complete 2024-06-26T07:05:52.5843012Z 73a0ce681c29: Verifying Checksum 2024-06-26T07:05:52.5843795Z 73a0ce681c29: Download complete 2024-06-26T07:05:52.6732464Z 49737bd5ae70: Verifying Checksum 2024-06-26T07:05:52.6733165Z 49737bd5ae70: Download complete 2024-06-26T07:05:53.0983813Z 9cd4c745f8e4: Verifying Checksum 2024-06-26T07:05:53.0984276Z 9cd4c745f8e4: Download complete 2024-06-26T07:05:53.1782635Z b2950a036534: Verifying Checksum 2024-06-26T07:05:53.1783187Z b2950a036534: Download complete 2024-06-26T07:05:53.2706266Z 25e19d81d9fc: Verifying Checksum 2024-06-26T07:05:53.2706805Z 25e19d81d9fc: Download complete 2024-06-26T07:05:53.5370589Z 60156b40fcbe: Verifying Checksum 2024-06-26T07:05:53.5371204Z 60156b40fcbe: Download complete 2024-06-26T07:05:53.6179497Z 7a868239227f: Verifying Checksum 2024-06-26T07:05:53.6179960Z 7a868239227f: Download complete 2024-06-26T07:05:53.7363922Z 83911d7c7ecf: Verifying Checksum 2024-06-26T07:05:53.7364542Z 83911d7c7ecf: Download complete 2024-06-26T07:05:53.8075206Z b3c5d66c679f: Verifying Checksum 2024-06-26T07:05:53.8075823Z b3c5d66c679f: Download complete 2024-06-26T07:05:58.2021826Z 7ed32bc8e469: Verifying Checksum 2024-06-26T07:05:58.2022560Z 7ed32bc8e469: Download complete 2024-06-26T07:05:58.2817109Z c1068d03a811: Verifying Checksum 2024-06-26T07:05:58.2817758Z c1068d03a811: Download complete 2024-06-26T07:05:58.3579730Z 671b4a301d81: Verifying Checksum 2024-06-26T07:05:58.3580172Z 671b4a301d81: Download complete 2024-06-26T07:05:58.7337058Z a8c1e85b5e14: Pull complete 2024-06-26T07:05:58.8483387Z 67760cf5ec46: Verifying Checksum 2024-06-26T07:05:58.8484068Z 67760cf5ec46: Download complete 2024-06-26T07:05:58.9134210Z ddc9740bcea2: Verifying Checksum 2024-06-26T07:05:58.9134643Z ddc9740bcea2: Download complete 2024-06-26T07:05:58.9274860Z a41a8d1c11c8: Pull complete 2024-06-26T07:05:58.9976969Z fb2e97768560: Verifying Checksum 2024-06-26T07:05:58.9977414Z fb2e97768560: Download complete 2024-06-26T07:05:59.0727883Z b599c5a8c7cb: Verifying Checksum 2024-06-26T07:05:59.0728342Z b599c5a8c7cb: Download complete 2024-06-26T07:05:59.1342882Z 0c1227890755: Pull complete 2024-06-26T07:05:59.1480457Z e08e88c13668: Download complete 2024-06-26T07:05:59.3488882Z d8d1234baab3: Pull complete 2024-06-26T07:06:04.9213723Z 3d914950f8ad: Verifying Checksum 2024-06-26T07:06:04.9214404Z 3d914950f8ad: Download complete 2024-06-26T07:06:04.9958410Z cf7c2cb896cb: Download complete 2024-06-26T07:06:05.0847337Z 3a778ddfac2a: Verifying Checksum 2024-06-26T07:06:05.0847923Z 3a778ddfac2a: Download complete 2024-06-26T07:06:05.1662485Z 236c8f0245f4: Download complete 2024-06-26T07:06:05.1741251Z 4f4fb700ef54: Download complete 2024-06-26T07:06:05.2538730Z dda1ce302b9a: Verifying Checksum 2024-06-26T07:06:05.2539396Z dda1ce302b9a: Download complete 2024-06-26T07:06:05.3251964Z e797fe86caef: Download complete 2024-06-26T07:06:05.4619769Z 967190076fc0: Download complete 2024-06-26T07:06:05.5370965Z 4fb0a83f0685: Verifying Checksum 2024-06-26T07:06:05.5371527Z 4fb0a83f0685: Download complete 2024-06-26T07:06:05.6116551Z 68af2ec4e30a: Download complete 2024-06-26T07:06:05.6954512Z cc91933e985a: Verifying Checksum 2024-06-26T07:06:05.6955021Z cc91933e985a: Download complete 2024-06-26T07:06:05.7625965Z 9f3187916e7d: Verifying Checksum 2024-06-26T07:06:05.7626634Z 9f3187916e7d: Download complete 2024-06-26T07:06:05.9069513Z 526083994857: Download complete 2024-06-26T07:06:05.9867378Z 05a6f6dcc1c0: Verifying Checksum 2024-06-26T07:06:05.9868196Z 05a6f6dcc1c0: Download complete 2024-06-26T07:06:06.5765739Z 6bc9dc8315b0: Verifying Checksum 2024-06-26T07:06:06.5766289Z 6bc9dc8315b0: Download complete 2024-06-26T07:06:06.6622793Z 1b0e27c35f07: Verifying Checksum 2024-06-26T07:06:06.6623277Z 1b0e27c35f07: Download complete 2024-06-26T07:06:28.8089500Z 12ba7431ff38: Verifying Checksum 2024-06-26T07:06:28.8089973Z 12ba7431ff38: Download complete 2024-06-26T07:06:28.8992632Z 789eff6c51aa: Download complete 2024-06-26T07:06:28.9808006Z de0f8a8c45c6: Download complete 2024-06-26T07:06:30.2490013Z 2060028f7488: Verifying Checksum 2024-06-26T07:06:30.2490491Z 2060028f7488: Download complete 2024-06-26T07:06:30.3273238Z e412c7f3b2de: Verifying Checksum 2024-06-26T07:06:30.3273837Z e412c7f3b2de: Download complete 2024-06-26T07:06:34.2332533Z 95237e7311a1: Verifying Checksum 2024-06-26T07:06:34.2333012Z 95237e7311a1: Download complete 2024-06-26T07:06:57.6140038Z 84b2c7a89982: Verifying Checksum 2024-06-26T07:06:57.6140552Z 84b2c7a89982: Download complete 2024-06-26T07:07:14.9832827Z 7ed32bc8e469: Pull complete 2024-06-26T07:07:15.1852525Z ec1e7978c1fe: Pull complete 2024-06-26T07:07:15.3938400Z 309b872da805: Pull complete 2024-06-26T07:07:22.7432003Z e82932cb002c: Pull complete 2024-06-26T07:07:22.8879590Z ca9dd73e3e93: Pull complete 2024-06-26T07:07:22.9595520Z 16d76a79d5d6: Pull complete 2024-06-26T07:07:23.0231711Z 4bdf41c9ecbe: Pull complete 2024-06-26T07:07:25.3679630Z f4863801e30c: Pull complete 2024-06-26T07:07:25.5685990Z d37dd9ed8948: Pull complete 2024-06-26T07:07:25.7691668Z cf2f34f225db: Pull complete 2024-06-26T07:07:26.0018880Z df546d7b3e34: Pull complete 2024-06-26T07:08:15.6145102Z 3d914950f8ad: Pull complete 2024-06-26T07:08:15.8372925Z 96d0e2032184: Pull complete 2024-06-26T07:08:16.0370486Z 06e12ce59790: Pull complete 2024-06-26T07:08:16.1652988Z ed712f63cde7: Pull complete 2024-06-26T07:08:16.3341448Z a97a71ef8a4e: Pull complete 2024-06-26T07:08:16.4567893Z 25cd0c482ad6: Pull complete 2024-06-26T07:08:19.9750219Z 9bf94bc45658: Pull complete 2024-06-26T07:08:20.1212404Z de45a14c9b22: Pull complete 2024-06-26T07:08:20.2017004Z e99241e77209: Pull complete 2024-06-26T07:08:20.2515031Z 35e8f4e2a42f: Pull complete 2024-06-26T07:08:20.3750853Z a823ffa4c541: Pull complete 2024-06-26T07:08:20.5754950Z 44ba272eae20: Pull complete 2024-06-26T07:08:28.9308563Z da02d278e524: Pull complete 2024-06-26T07:08:29.0530369Z 73a0ce681c29: Pull complete 2024-06-26T07:08:29.2043738Z 49737bd5ae70: Pull complete 2024-06-26T07:08:30.1471592Z 9cd4c745f8e4: Pull complete 2024-06-26T07:08:30.3057052Z b2950a036534: Pull complete 2024-06-26T07:08:30.4622026Z 25e19d81d9fc: Pull complete 2024-06-26T07:08:30.9435086Z 60156b40fcbe: Pull complete 2024-06-26T07:08:31.1544263Z 7a868239227f: Pull complete 2024-06-26T07:08:31.5101300Z 83911d7c7ecf: Pull complete 2024-06-26T07:08:31.7046387Z b3c5d66c679f: Pull complete 2024-06-26T07:09:10.4314008Z 84b2c7a89982: Pull complete 2024-06-26T07:09:10.5758068Z c1068d03a811: Pull complete 2024-06-26T07:09:10.6251200Z 671b4a301d81: Pull complete 2024-06-26T07:09:12.5905020Z 67760cf5ec46: Pull complete 2024-06-26T07:09:12.6409931Z ddc9740bcea2: Pull complete 2024-06-26T07:09:12.6894673Z fb2e97768560: Pull complete 2024-06-26T07:09:12.8143682Z b599c5a8c7cb: Pull complete 2024-06-26T07:09:12.8544054Z e08e88c13668: Pull complete 2024-06-26T07:09:46.6041032Z 12ba7431ff38: Pull complete 2024-06-26T07:09:46.7523359Z cf7c2cb896cb: Pull complete 2024-06-26T07:09:46.8448788Z 3a778ddfac2a: Pull complete 2024-06-26T07:09:47.1596120Z 236c8f0245f4: Pull complete 2024-06-26T07:09:47.3583588Z 4f4fb700ef54: Pull complete 2024-06-26T07:09:47.5909931Z dda1ce302b9a: Pull complete 2024-06-26T07:09:47.7735783Z e797fe86caef: Pull complete 2024-06-26T07:09:48.1078241Z 967190076fc0: Pull complete 2024-06-26T07:09:48.3495164Z 4fb0a83f0685: Pull complete 2024-06-26T07:09:48.5453206Z 68af2ec4e30a: Pull complete 2024-06-26T07:09:48.7483164Z cc91933e985a: Pull complete 2024-06-26T07:09:48.9444958Z 9f3187916e7d: Pull complete 2024-06-26T07:09:50.0593696Z 526083994857: Pull complete 2024-06-26T07:09:50.1899778Z 05a6f6dcc1c0: Pull complete 2024-06-26T07:09:52.2087849Z 6bc9dc8315b0: Pull complete 2024-06-26T07:09:52.4306483Z 1b0e27c35f07: Pull complete 2024-06-26T07:10:06.0398013Z 95237e7311a1: Pull complete 2024-06-26T07:10:06.0971135Z 789eff6c51aa: Pull complete 2024-06-26T07:10:06.1522836Z de0f8a8c45c6: Pull complete 2024-06-26T07:10:06.8692811Z 2060028f7488: Pull complete 2024-06-26T07:10:06.9361759Z e412c7f3b2de: Pull complete 2024-06-26T07:10:07.9074841Z Digest: sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T07:10:07.9535430Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:10:07.9758661Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:10:07.9819235Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:10:07.9820133Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:10:07.9828203Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:07.9828725Z env: 2024-06-26T07:10:07.9829003Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:07.9829334Z ##[endgroup] 2024-06-26T07:10:07.9981541Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-06-26T07:10:07.9982078Z with: 2024-06-26T07:10:07.9982369Z driver-version: 550.54.15 2024-06-26T07:10:07.9982715Z env: 2024-06-26T07:10:07.9982986Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:07.9983341Z ##[endgroup] 2024-06-26T07:10:08.0067015Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:10:08.0067580Z with: 2024-06-26T07:10:08.0067959Z timeout_minutes: 10 2024-06-26T07:10:08.0068289Z max_attempts: 3 2024-06-26T07:10:08.0098993Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-06-26T07:10:08.0129962Z retry_wait_seconds: 10 2024-06-26T07:10:08.0130332Z polling_interval_seconds: 1 2024-06-26T07:10:08.0130703Z warning_on_retry: true 2024-06-26T07:10:08.0131051Z continue_on_error: false 2024-06-26T07:10:08.0131383Z env: 2024-06-26T07:10:08.0131653Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:08.0132006Z DRIVER_VERSION: 550.54.15 2024-06-26T07:10:08.0132359Z ##[endgroup] 2024-06-26T07:10:08.0643984Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-06-26T07:10:08.0645667Z + pre_install_nvidia_driver_amzn2 2024-06-26T07:10:08.0646289Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-06-26T07:10:08.3792098Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:10:08.4177621Z No Match for argument: nvidia-driver-latest-dkms 2024-06-26T07:10:08.4418574Z No Packages marked for removal 2024-06-26T07:10:08.4532499Z + install_nvidia_driver_common 2024-06-26T07:10:08.4535035Z + echo 'Before installing NVIDIA driver' 2024-06-26T07:10:08.4535465Z + lspci 2024-06-26T07:10:08.4535870Z Before installing NVIDIA driver 2024-06-26T07:10:08.4614984Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:10:08.4615737Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:10:08.4616645Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:10:08.4617561Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:10:08.4620135Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:10:08.4621030Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:10:08.4621754Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:10:08.4622554Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:10:08.4623144Z + lsmod 2024-06-26T07:10:08.4628326Z Module Size Used by 2024-06-26T07:10:08.4628763Z xt_nat 16384 0 2024-06-26T07:10:08.4629226Z ib_core 266240 0 2024-06-26T07:10:08.4629687Z nvidia_modeset 1339392 0 2024-06-26T07:10:08.4630062Z veth 16384 0 2024-06-26T07:10:08.4630428Z nvidia_uvm 4599808 0 2024-06-26T07:10:08.4631158Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:10:08.4631657Z drm 421888 1 nvidia 2024-06-26T07:10:08.4632080Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:10:08.4632536Z backlight 16384 1 nvidia_modeset 2024-06-26T07:10:08.4633062Z xt_conntrack 16384 1 2024-06-26T07:10:08.4633527Z ipt_MASQUERADE 16384 1 2024-06-26T07:10:08.4634145Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:10:08.4634817Z nf_conntrack_netlink 49152 0 2024-06-26T07:10:08.4635440Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:10:08.4636072Z xfrm_user 45056 1 2024-06-26T07:10:08.4636471Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:10:08.4636897Z iptable_nat 16384 1 2024-06-26T07:10:08.4637311Z nf_conntrack_ipv4 16384 3 2024-06-26T07:10:08.4637818Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:10:08.4638413Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:10:08.4639301Z nf_nat 32768 3 nf_nat_masquerade_ipv4,nf_nat_ipv4,xt_nat 2024-06-26T07:10:08.4640666Z nf_conntrack 155648 8 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink 2024-06-26T07:10:08.4642002Z xt_addrtype 16384 2 2024-06-26T07:10:08.4642539Z iptable_filter 16384 1 2024-06-26T07:10:08.4643266Z br_netfilter 24576 0 2024-06-26T07:10:08.4644038Z bridge 172032 1 br_netfilter 2024-06-26T07:10:08.4644489Z stp 16384 1 bridge 2024-06-26T07:10:08.4644910Z llc 16384 2 bridge,stp 2024-06-26T07:10:08.4645311Z overlay 86016 0 2024-06-26T07:10:08.4645676Z sunrpc 393216 1 2024-06-26T07:10:08.4646044Z dm_mirror 28672 0 2024-06-26T07:10:08.4646471Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:10:08.4646952Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:10:08.4647453Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:10:08.4647899Z dax 69632 1 dm_mod 2024-06-26T07:10:08.4648290Z crc32_pclmul 16384 0 2024-06-26T07:10:08.4648720Z ghash_clmulni_intel 16384 0 2024-06-26T07:10:08.4649089Z pcbc 16384 0 2024-06-26T07:10:08.4649530Z aesni_intel 188416 0 2024-06-26T07:10:08.4649921Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:10:08.4650365Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:10:08.4650808Z glue_helper 16384 1 aesni_intel 2024-06-26T07:10:08.4651348Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:10:08.4651885Z mousedev 24576 0 2024-06-26T07:10:08.4652245Z evdev 20480 3 2024-06-26T07:10:08.4652608Z psmouse 32768 0 2024-06-26T07:10:08.4652976Z button 16384 0 2024-06-26T07:10:08.4653331Z ena 139264 0 2024-06-26T07:10:08.4653693Z ptp 20480 1 ena 2024-06-26T07:10:08.4654083Z pps_core 20480 1 ptp 2024-06-26T07:10:08.4654473Z crc32c_intel 24576 0 2024-06-26T07:10:08.4654839Z autofs4 49152 2 2024-06-26T07:10:08.4655192Z + modinfo nvidia 2024-06-26T07:10:08.4655863Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:10:08.4656555Z alias: char-major-195-* 2024-06-26T07:10:08.4656927Z version: 550.54.15 2024-06-26T07:10:08.4657271Z supported: external 2024-06-26T07:10:08.4657612Z license: NVIDIA 2024-06-26T07:10:08.4657986Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:10:08.4658451Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:10:08.4658906Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:10:08.4659367Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:10:08.4659869Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:10:08.4660475Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:10:08.4660964Z depends: i2c-core,drm 2024-06-26T07:10:08.4661319Z retpoline: Y 2024-06-26T07:10:08.4661618Z name: nvidia 2024-06-26T07:10:08.4662162Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:10:08.4662810Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:10:08.4663454Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:10:08.4664031Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:10:08.4664474Z parm: NVreg_RmLogonRC:int 2024-06-26T07:10:08.4664905Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:10:08.4665356Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:10:08.4665779Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:10:08.4666215Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:10:08.4666739Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:10:08.4667300Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:10:08.4667769Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:10:08.4668351Z parm: NVreg_EnableMSI:int 2024-06-26T07:10:08.4668813Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:10:08.4669272Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:10:08.4669790Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:10:08.4670474Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:10:08.4671031Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:10:08.4671633Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:10:08.4672217Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:10:08.4672827Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:10:08.4673419Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:10:08.4673910Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:10:08.4674439Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:10:08.4674980Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:10:08.4675474Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:10:08.4675924Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:10:08.4676402Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:10:08.4676862Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:10:08.4677318Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:10:08.4677808Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:10:08.4678329Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:10:08.4678836Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:10:08.4679333Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:10:08.4679822Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:10:08.4680304Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:10:08.4680802Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:10:08.4681278Z parm: NVreg_RmMsg:charp 2024-06-26T07:10:08.4681676Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:10:08.4682137Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:10:08.4682597Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:10:08.4683054Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:10:08.4683521Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:10:08.4684302Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:10:08.4684749Z parm: rm_firmware_active:charp 2024-06-26T07:10:08.4685149Z + HAS_NVIDIA_DRIVER=0 2024-06-26T07:10:08.4685537Z ++ command -v nvidia-smi 2024-06-26T07:10:08.4685928Z + '[' -x /usr/bin/nvidia-smi ']' 2024-06-26T07:10:08.4686291Z + set +e 2024-06-26T07:10:08.4686778Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-06-26T07:10:10.4443132Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-06-26T07:10:10.4443729Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:10:10.4445058Z + '[' 0 -ne 0 ']' 2024-06-26T07:10:10.4445461Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-06-26T07:10:10.4445854Z + HAS_NVIDIA_DRIVER=1 2024-06-26T07:10:10.4446570Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-06-26T07:10:10.4447264Z + set -e 2024-06-26T07:10:10.4447565Z + '[' 1 -eq 0 ']' 2024-06-26T07:10:10.4447897Z + post_install_nvidia_driver_common 2024-06-26T07:10:10.4448565Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-06-26T07:10:10.4449225Z + sudo modprobe nvidia 2024-06-26T07:10:10.4545087Z + echo 'After installing NVIDIA driver' 2024-06-26T07:10:10.4545580Z + lspci 2024-06-26T07:10:10.4545872Z After installing NVIDIA driver 2024-06-26T07:10:10.4630415Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:10:10.4632192Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:10:10.4634029Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:10:10.4635595Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:10:10.4637055Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:10:10.4638524Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:10:10.4639568Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:10:10.4640610Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:10:10.4641184Z + lsmod 2024-06-26T07:10:10.4644425Z Module Size Used by 2024-06-26T07:10:10.4644999Z xt_nat 16384 0 2024-06-26T07:10:10.4645507Z ib_core 266240 0 2024-06-26T07:10:10.4646003Z nvidia_modeset 1339392 0 2024-06-26T07:10:10.4646507Z veth 16384 0 2024-06-26T07:10:10.4647011Z nvidia_uvm 4599808 0 2024-06-26T07:10:10.4647534Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:10:10.4648010Z drm 421888 1 nvidia 2024-06-26T07:10:10.4648479Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:10:10.4649158Z backlight 16384 1 nvidia_modeset 2024-06-26T07:10:10.4649797Z xt_conntrack 16384 1 2024-06-26T07:10:10.4650312Z ipt_MASQUERADE 16384 1 2024-06-26T07:10:10.4650904Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:10:10.4651561Z nf_conntrack_netlink 49152 0 2024-06-26T07:10:10.4652059Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:10:10.4652617Z xfrm_user 45056 1 2024-06-26T07:10:10.4653144Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:10:10.4653715Z iptable_nat 16384 1 2024-06-26T07:10:10.4654234Z nf_conntrack_ipv4 16384 3 2024-06-26T07:10:10.4654821Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:10:10.4655462Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:10:10.4656250Z nf_nat 32768 3 nf_nat_masquerade_ipv4,nf_nat_ipv4,xt_nat 2024-06-26T07:10:10.4657586Z nf_conntrack 155648 8 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink 2024-06-26T07:10:10.4658581Z xt_addrtype 16384 2 2024-06-26T07:10:10.4658960Z iptable_filter 16384 1 2024-06-26T07:10:10.4659342Z br_netfilter 24576 0 2024-06-26T07:10:10.4659743Z bridge 172032 1 br_netfilter 2024-06-26T07:10:10.4660192Z stp 16384 1 bridge 2024-06-26T07:10:10.4660612Z llc 16384 2 bridge,stp 2024-06-26T07:10:10.4661027Z overlay 86016 0 2024-06-26T07:10:10.4661387Z sunrpc 393216 1 2024-06-26T07:10:10.4661757Z dm_mirror 28672 0 2024-06-26T07:10:10.4662151Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:10:10.4662733Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:10:10.4663393Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:10:10.4663895Z dax 69632 1 dm_mod 2024-06-26T07:10:10.4664295Z crc32_pclmul 16384 0 2024-06-26T07:10:10.4664680Z ghash_clmulni_intel 16384 0 2024-06-26T07:10:10.4665052Z pcbc 16384 0 2024-06-26T07:10:10.4665426Z aesni_intel 188416 0 2024-06-26T07:10:10.4665826Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:10:10.4666282Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:10:10.4666735Z glue_helper 16384 1 aesni_intel 2024-06-26T07:10:10.4667294Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:10:10.4667945Z mousedev 24576 0 2024-06-26T07:10:10.4668311Z evdev 20480 3 2024-06-26T07:10:10.4668688Z psmouse 32768 0 2024-06-26T07:10:10.4669084Z button 16384 0 2024-06-26T07:10:10.4669450Z ena 139264 0 2024-06-26T07:10:10.4669826Z ptp 20480 1 ena 2024-06-26T07:10:10.4670211Z pps_core 20480 1 ptp 2024-06-26T07:10:10.4670602Z crc32c_intel 24576 0 2024-06-26T07:10:10.4670971Z autofs4 49152 2 2024-06-26T07:10:10.4671329Z + modinfo nvidia 2024-06-26T07:10:10.4671969Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:10:10.4672657Z alias: char-major-195-* 2024-06-26T07:10:10.4673137Z version: 550.54.15 2024-06-26T07:10:10.4673476Z supported: external 2024-06-26T07:10:10.4673825Z license: NVIDIA 2024-06-26T07:10:10.4674204Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:10:10.4674691Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:10:10.4675151Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:10:10.4675627Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:10:10.4676133Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:10:10.4676632Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:10:10.4677104Z depends: i2c-core,drm 2024-06-26T07:10:10.4677471Z retpoline: Y 2024-06-26T07:10:10.4677782Z name: nvidia 2024-06-26T07:10:10.4678337Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:10:10.4678982Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:10:10.4679679Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:10:10.4680279Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:10:10.4680721Z parm: NVreg_RmLogonRC:int 2024-06-26T07:10:10.4681149Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:10:10.4681610Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:10:10.4682051Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:10:10.4682487Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:10:10.4683027Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:10:10.4683808Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:10:10.4684319Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:10:10.4684744Z parm: NVreg_EnableMSI:int 2024-06-26T07:10:10.4685164Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:10:10.4685630Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:10:10.4686162Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:10:10.4686744Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:10:10.4687310Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:10:10.4687918Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:10:10.4688523Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:10:10.4689136Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:10:10.4689729Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:10:10.4690226Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:10:10.4690861Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:10:10.4691405Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:10:10.4691908Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:10:10.4692377Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:10:10.4692876Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:10:10.4693348Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:10:10.4693802Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:10:10.4694308Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:10:10.4695022Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:10:10.4695507Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:10:10.4695993Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:10:10.4696496Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:10:10.4696973Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:10:10.4697479Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:10:10.4697963Z parm: NVreg_RmMsg:charp 2024-06-26T07:10:10.4698387Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:10:10.4698849Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:10:10.4699317Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:10:10.4699771Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:10:10.4700244Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:10:10.4700791Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:10:10.4701244Z parm: rm_firmware_active:charp 2024-06-26T07:10:10.4701658Z + set +e 2024-06-26T07:10:10.4701957Z + nvidia-smi 2024-06-26T07:10:12.0361202Z Wed Jun 26 07:10:12 2024 2024-06-26T07:10:12.0362152Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:10:12.0363026Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-06-26T07:10:12.0364091Z |-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:10:12.0364878Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-06-26T07:10:12.0365742Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-06-26T07:10:12.0366429Z | | | MIG M. | 2024-06-26T07:10:12.0366983Z |=========================================+========================+======================| 2024-06-26T07:10:12.0500211Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-06-26T07:10:12.0501976Z | 0% 32C P0 58W / 300W | 0MiB / 23028MiB | 5% Default | 2024-06-26T07:10:12.0503274Z | | | N/A | 2024-06-26T07:10:12.0504873Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:10:12.0506042Z 2024-06-26T07:10:12.0507335Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:10:12.0508724Z | Processes: | 2024-06-26T07:10:12.0509765Z | GPU GI CI PID Type Process name GPU Memory | 2024-06-26T07:10:12.0510453Z | ID ID Usage | 2024-06-26T07:10:12.0511009Z |=========================================================================================| 2024-06-26T07:10:12.0511641Z | No running processes found | 2024-06-26T07:10:12.0512380Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:10:12.6502184Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-06-26T07:10:14.2042561Z NVIDIA A10G 2024-06-26T07:10:14.6529931Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:10:14.6530596Z + '[' 0 -eq 0 ']' 2024-06-26T07:10:14.6531096Z + echo 'INFO: Ignoring allowed status 0' 2024-06-26T07:10:14.6531605Z + set -e 2024-06-26T07:10:14.6531971Z INFO: Ignoring allowed status 0 2024-06-26T07:10:14.6533830Z == Installing nvidia container toolkit for amzn2 == 2024-06-26T07:10:14.6535651Z + sudo yum install -y yum-utils 2024-06-26T07:10:14.9592312Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:10:16.4393248Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-06-26T07:10:16.4393936Z Nothing to do 2024-06-26T07:10:16.5892143Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]] 2024-06-26T07:10:16.5893123Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:10:16.5894184Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:10:16.9157204Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:10:16.9460356Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:10:16.9461656Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:10:16.9462590Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:10:16.9577589Z + sudo yum install -y nvidia-docker2 2024-06-26T07:10:17.2656049Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:10:18.7303742Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-06-26T07:10:18.7304528Z Nothing to do 2024-06-26T07:10:18.8794453Z + sudo systemctl restart docker 2024-06-26T07:10:56.1111979Z Command completed after 1 attempt(s). 2024-06-26T07:10:56.1170808Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:10:56.1171575Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:10:56.1172270Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-06-26T07:10:56.1172928Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:10:56.1180636Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:10:56.1181120Z env: 2024-06-26T07:10:56.1181404Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:56.1181870Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:56.1182351Z ##[endgroup] 2024-06-26T07:10:56.3389363Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:10:56.3540550Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-06-26T07:10:56.3640909Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-06-26T07:10:56.4571521Z Prepare all required actions 2024-06-26T07:10:56.4572022Z Getting action download info 2024-06-26T07:10:56.5765464Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-06-26T07:10:56.7899509Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-06-26T07:10:56.9173880Z ##[group]Run ./.github/actions/download-build-artifacts 2024-06-26T07:10:56.9174372Z with: 2024-06-26T07:10:56.9174710Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:10:56.9175158Z s3-bucket: gha-artifacts 2024-06-26T07:10:56.9175499Z env: 2024-06-26T07:10:56.9175764Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:56.9176215Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:56.9176693Z ##[endgroup] 2024-06-26T07:10:56.9207901Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:10:56.9208342Z with: 2024-06-26T07:10:56.9208714Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:10:56.9209166Z s3-bucket: gha-artifacts 2024-06-26T07:10:56.9209522Z region: us-east-1 2024-06-26T07:10:56.9209823Z env: 2024-06-26T07:10:56.9210091Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:10:56.9210592Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:10:56.9211075Z ##[endgroup] 2024-06-26T07:10:57.3353002Z (node:26767) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:10:57.3354397Z 2024-06-26T07:10:57.3354906Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:10:57.3356320Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:10:57.3357995Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:10:57.4137168Z Found 1 objects with prefix pytorch/pytorch/9673646806/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-06-26T07:10:57.4138361Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:11:04.8995338Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:11:04.9000919Z Artifact download has finished successfully 2024-06-26T07:11:04.9148722Z ##[group]Run unzip -o artifacts.zip 2024-06-26T07:11:04.9149149Z unzip -o artifacts.zip 2024-06-26T07:11:04.9157050Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:04.9157542Z env: 2024-06-26T07:11:04.9158010Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:04.9158471Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:04.9158955Z ##[endgroup] 2024-06-26T07:11:04.9185845Z Archive: artifacts.zip 2024-06-26T07:11:04.9188159Z creating: dist/ 2024-06-26T07:11:06.9016582Z inflating: dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:11:06.9018239Z creating: build/custom_test_artifacts/ 2024-06-26T07:11:06.9019395Z creating: build/custom_test_artifacts/custom-op-build/ 2024-06-26T07:11:06.9020805Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-06-26T07:11:06.9022437Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:11:06.9023634Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:11:06.9024556Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-06-26T07:11:06.9025475Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:11:06.9026462Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:11:06.9027489Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:11:06.9028655Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:11:06.9029923Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:11:06.9030903Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:11:06.9031880Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:11:06.9033062Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:11:06.9034197Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:11:06.9035315Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:11:06.9036527Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:11:06.9037656Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:11:06.9038805Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:11:06.9039820Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:11:06.9040795Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:11:06.9075034Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:11:06.9113071Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:11:06.9114483Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:11:06.9156890Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:11:06.9158288Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:11:06.9159678Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:11:06.9161115Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:11:06.9162492Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:11:06.9164265Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:11:06.9165635Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:11:06.9166986Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:11:06.9168306Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:11:06.9169560Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:11:06.9170765Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:11:06.9171954Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:11:06.9173157Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:11:06.9174323Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:11:06.9175510Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:11:06.9222758Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:11:06.9280558Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:11:06.9281710Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:11:06.9282675Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:11:06.9283518Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:11:06.9284618Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-06-26T07:11:06.9285525Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-06-26T07:11:06.9286533Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-06-26T07:11:06.9287664Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-06-26T07:11:06.9288749Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-06-26T07:11:06.9289765Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-06-26T07:11:06.9290817Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-06-26T07:11:06.9291879Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-06-26T07:11:06.9293001Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-06-26T07:11:06.9294054Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-06-26T07:11:06.9295102Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-06-26T07:11:06.9306076Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-06-26T07:11:06.9433532Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-06-26T07:11:06.9434637Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-06-26T07:11:06.9435705Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-06-26T07:11:06.9436896Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-06-26T07:11:06.9438166Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-06-26T07:11:06.9439235Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-06-26T07:11:06.9440345Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-06-26T07:11:06.9441469Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-06-26T07:11:06.9442589Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-06-26T07:11:06.9443973Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-06-26T07:11:06.9445244Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-06-26T07:11:06.9457056Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-06-26T07:11:06.9536164Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-06-26T07:11:06.9537328Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:11:06.9538376Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:11:06.9539429Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-06-26T07:11:06.9540451Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-06-26T07:11:06.9541444Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-06-26T07:11:06.9542444Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-06-26T07:11:06.9543442Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-06-26T07:11:06.9544307Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-06-26T07:11:06.9545200Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-06-26T07:11:06.9651279Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-06-26T07:11:06.9710471Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-06-26T07:11:06.9711314Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-06-26T07:11:06.9712003Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-06-26T07:11:06.9712808Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:11:06.9717850Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:11:06.9718829Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-06-26T07:11:06.9719746Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:11:06.9720711Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:11:06.9721648Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:11:06.9722803Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:11:06.9724097Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:11:06.9725075Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:11:06.9726041Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:11:06.9727162Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:11:06.9728285Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:11:06.9729544Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:11:06.9730839Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:11:06.9731976Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:11:06.9733167Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:11:06.9734166Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:11:06.9735138Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:11:06.9769937Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:11:06.9807702Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:11:06.9809114Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:11:06.9851515Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:11:06.9852888Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:11:06.9854393Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:11:06.9855825Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:11:06.9857206Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:11:06.9858543Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:11:06.9860119Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:11:06.9861676Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:11:06.9863211Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:11:06.9864650Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:11:06.9866025Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:11:06.9867384Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:11:06.9868747Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:11:06.9869916Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:11:06.9871094Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:11:06.9917761Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:11:06.9975448Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:11:06.9976605Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:11:06.9977568Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:11:06.9978402Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:11:06.9979265Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-06-26T07:11:06.9980304Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-06-26T07:11:06.9981347Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-06-26T07:11:06.9982505Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-06-26T07:11:06.9983671Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-06-26T07:11:06.9984721Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-06-26T07:11:06.9985807Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-06-26T07:11:06.9986894Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-06-26T07:11:06.9987989Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-06-26T07:11:06.9989162Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-06-26T07:11:06.9990230Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-06-26T07:11:07.0000966Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-06-26T07:11:07.0062058Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-06-26T07:11:07.0063224Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:11:07.0064266Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:11:07.0065216Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-06-26T07:11:07.0066097Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-06-26T07:11:07.0066974Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-06-26T07:11:07.0067843Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-06-26T07:11:07.0068736Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-06-26T07:11:07.0069494Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-06-26T07:11:07.0070287Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-06-26T07:11:07.0116158Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-06-26T07:11:07.0117006Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-06-26T07:11:07.0117758Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-06-26T07:11:07.0118626Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:11:07.0123763Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:11:07.0124847Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-06-26T07:11:07.0125826Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:11:07.0126862Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:11:07.0127872Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:11:07.0129012Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:11:07.0146126Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:11:07.0147648Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:11:07.0149115Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:11:07.0150503Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:11:07.0151878Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:11:07.0153067Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:11:07.0154257Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:11:07.0155461Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:11:07.0156673Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:11:07.0157751Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:11:07.0158806Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:11:07.0174546Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:11:07.0212170Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:11:07.0213872Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:11:07.0255958Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:11:07.0257721Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:11:07.0259189Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:11:07.0260697Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:11:07.0262142Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:11:07.0277263Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:11:07.0279155Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:11:07.0281120Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:11:07.0282874Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:11:07.0285143Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:11:07.0286487Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:11:07.0287748Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:11:07.0289042Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:11:07.0290291Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:11:07.0291568Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:11:07.0342833Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:11:07.0400681Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:11:07.0401926Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:11:07.0402984Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:11:07.0404047Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:11:07.0405009Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-06-26T07:11:07.0406015Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-06-26T07:11:07.0407150Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-06-26T07:11:07.0408391Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-06-26T07:11:07.0409600Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-06-26T07:11:07.0410719Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-06-26T07:11:07.0411881Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-06-26T07:11:07.0413222Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-06-26T07:11:07.0414427Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-06-26T07:11:07.0415606Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-06-26T07:11:07.0416760Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-06-26T07:11:07.0417985Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-06-26T07:11:07.0531324Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-06-26T07:11:07.0532489Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-06-26T07:11:07.0533659Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-06-26T07:11:07.0534942Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-06-26T07:11:07.0536185Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-06-26T07:11:07.0537366Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-06-26T07:11:07.0538590Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-06-26T07:11:07.0539815Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-06-26T07:11:07.0541029Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-06-26T07:11:07.0542251Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-06-26T07:11:07.0543510Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-06-26T07:11:07.0555501Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-06-26T07:11:07.0608348Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-06-26T07:11:07.0609622Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:11:07.0610881Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:11:07.0611897Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-06-26T07:11:07.0612851Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-06-26T07:11:07.0613815Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-06-26T07:11:07.0614753Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-06-26T07:11:07.0615651Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-06-26T07:11:07.0616476Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-06-26T07:11:07.0617339Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-06-26T07:11:07.0713633Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-06-26T07:11:07.0754537Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-06-26T07:11:07.0755174Z creating: build/lib/ 2024-06-26T07:11:07.0839633Z inflating: build/lib/libprotobuf-lite.a 2024-06-26T07:11:07.1279193Z inflating: build/lib/libprotobuf.a 2024-06-26T07:11:07.1288660Z inflating: build/lib/libpthreadpool.a 2024-06-26T07:11:07.1296403Z inflating: build/lib/libcpuinfo.a 2024-06-26T07:11:07.1304102Z inflating: build/lib/libcpuinfo_internals.a 2024-06-26T07:11:07.1304597Z inflating: build/lib/libclog.a 2024-06-26T07:11:07.1322154Z inflating: build/lib/libnnpack.a 2024-06-26T07:11:07.1324021Z inflating: build/lib/libnnpack_reference_layers.a 2024-06-26T07:11:07.1386237Z inflating: build/lib/libgtest.a 2024-06-26T07:11:07.1458739Z inflating: build/lib/libbenchmark.a 2024-06-26T07:11:07.1520530Z inflating: build/lib/libasmjit.a 2024-06-26T07:11:07.1527698Z inflating: build/lib/libittnotify.a 2024-06-26T07:11:07.1554279Z inflating: build/lib/libtensorpipe_uv.a 2024-06-26T07:11:07.1676316Z inflating: build/lib/libgloo.a 2024-06-26T07:11:07.1697365Z inflating: build/lib/libfmt.a 2024-06-26T07:11:07.1785765Z inflating: build/lib/libc10.so 2024-06-26T07:11:07.1787552Z inflating: build/lib/libcaffe2_nvrtc.so 2024-06-26T07:11:07.1788626Z inflating: build/lib/libfoxi_loader.a 2024-06-26T07:11:07.1789596Z inflating: build/lib/libtorch_global_deps.so 2024-06-26T07:11:07.1807449Z inflating: build/lib/libpytorch_qnnpack.a 2024-06-26T07:11:07.1824498Z inflating: build/lib/libgmock.a 2024-06-26T07:11:07.1824950Z inflating: build/lib/libgtest_main.a 2024-06-26T07:11:07.1825425Z inflating: build/lib/libbenchmark_main.a 2024-06-26T07:11:07.2310016Z inflating: build/lib/libprotoc.a 2024-06-26T07:11:07.2678520Z inflating: build/lib/libgloo_cuda.a 2024-06-26T07:11:08.2385056Z inflating: build/lib/libdnnl.a 2024-06-26T07:11:08.2934664Z inflating: build/lib/libtensorpipe.a 2024-06-26T07:11:08.2987088Z inflating: build/lib/libc10_cuda.so 2024-06-26T07:11:08.2987990Z inflating: build/lib/libgmock_main.a 2024-06-26T07:11:08.4187474Z inflating: build/lib/libfbgemm.a 2024-06-26T07:11:08.4432273Z inflating: build/lib/libtensorpipe_cuda.a 2024-06-26T07:11:08.4929502Z inflating: build/lib/libkineto.a 2024-06-26T07:11:08.5107806Z inflating: build/lib/libXNNPACK.a 2024-06-26T07:11:08.5147193Z inflating: build/lib/libonnx_proto.a 2024-06-26T07:11:08.5811300Z inflating: build/lib/libonnx.a 2024-06-26T07:11:10.9068461Z inflating: build/lib/libtorch_cpu.so 2024-06-26T07:11:10.9072046Z inflating: build/lib/libunbox_lib.a 2024-06-26T07:11:10.9075651Z inflating: build/lib/libshm.so 2024-06-26T07:11:12.8927680Z inflating: build/lib/libtorch_cuda.so 2024-06-26T07:11:12.8928623Z inflating: build/lib/libtorch.so 2024-06-26T07:11:13.6867298Z inflating: build/lib/libtorch_cuda_linalg.so 2024-06-26T07:11:13.6868819Z inflating: build/lib/libc10d_cuda_test.so 2024-06-26T07:11:13.8764160Z inflating: build/lib/libtorch_python.so 2024-06-26T07:11:13.8830713Z inflating: build/lib/libtorchbind_test.so 2024-06-26T07:11:13.8850145Z inflating: build/lib/libjitbackend_test.so 2024-06-26T07:11:13.8874658Z inflating: build/lib/libbackend_with_compiler.so 2024-06-26T07:11:13.8908860Z inflating: build/lib/libnnapi_backend.so 2024-06-26T07:11:13.8909768Z creating: build/bin/ 2024-06-26T07:11:13.8957106Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-06-26T07:11:13.9007199Z inflating: build/bin/c10_DeviceGuard_test 2024-06-26T07:11:13.9055871Z inflating: build/bin/c10_Device_test 2024-06-26T07:11:13.9113108Z inflating: build/bin/c10_DispatchKeySet_test 2024-06-26T07:11:13.9165346Z inflating: build/bin/c10_Scalar_test 2024-06-26T07:11:13.9212502Z inflating: build/bin/c10_StreamGuard_test 2024-06-26T07:11:13.9265858Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-06-26T07:11:13.9314852Z inflating: build/bin/c10_SymInt_test 2024-06-26T07:11:13.9368482Z inflating: build/bin/c10_InlineStreamGuard_test 2024-06-26T07:11:13.9422857Z inflating: build/bin/c10_SizesAndStrides_test 2024-06-26T07:11:13.9474679Z inflating: build/bin/c10_Bitset_test 2024-06-26T07:11:13.9542981Z inflating: build/bin/c10_cow_test 2024-06-26T07:11:13.9591081Z inflating: build/bin/c10_ConstexprCrc_test 2024-06-26T07:11:13.9639065Z inflating: build/bin/c10_DeadlockDetection_test 2024-06-26T07:11:13.9689500Z inflating: build/bin/c10_Half_test 2024-06-26T07:11:13.9744330Z inflating: build/bin/c10_LeftRight_test 2024-06-26T07:11:13.9797540Z inflating: build/bin/c10_Metaprogramming_test 2024-06-26T07:11:13.9845996Z inflating: build/bin/c10_Synchronized_test 2024-06-26T07:11:13.9899941Z inflating: build/bin/c10_ThreadLocal_test 2024-06-26T07:11:13.9950781Z inflating: build/bin/c10_TypeIndex_test 2024-06-26T07:11:13.9999903Z inflating: build/bin/c10_TypeList_test 2024-06-26T07:11:14.0047342Z inflating: build/bin/c10_TypeTraits_test 2024-06-26T07:11:14.0097331Z inflating: build/bin/c10_accumulate_test 2024-06-26T07:11:14.0151605Z inflating: build/bin/c10_bfloat16_test 2024-06-26T07:11:14.0199981Z inflating: build/bin/c10_bit_cast_test 2024-06-26T07:11:14.0254518Z inflating: build/bin/c10_complex_math_test 2024-06-26T07:11:14.0308429Z inflating: build/bin/c10_complex_test 2024-06-26T07:11:14.0359341Z inflating: build/bin/c10_exception_test 2024-06-26T07:11:14.0408865Z inflating: build/bin/c10_flags_test 2024-06-26T07:11:14.0457296Z inflating: build/bin/c10_generic_math_test 2024-06-26T07:11:14.0615773Z inflating: build/bin/c10_intrusive_ptr_test 2024-06-26T07:11:14.0665279Z inflating: build/bin/c10_irange_test 2024-06-26T07:11:14.0716670Z inflating: build/bin/c10_lazy_test 2024-06-26T07:11:14.0772782Z inflating: build/bin/c10_logging_test 2024-06-26T07:11:14.0844633Z inflating: build/bin/c10_optional_test 2024-06-26T07:11:14.0905073Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-06-26T07:11:14.0957281Z inflating: build/bin/c10_registry_test 2024-06-26T07:11:14.1102685Z inflating: build/bin/c10_small_vector_test 2024-06-26T07:11:14.1152909Z inflating: build/bin/c10_ssize_test 2024-06-26T07:11:14.1203154Z inflating: build/bin/c10_string_util_test 2024-06-26T07:11:14.1259937Z inflating: build/bin/c10_string_view_test 2024-06-26T07:11:14.1308764Z inflating: build/bin/c10_tempfile_test 2024-06-26T07:11:14.1356000Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-06-26T07:11:14.1410301Z inflating: build/bin/c10_typeid_test 2024-06-26T07:11:14.1845526Z inflating: build/bin/protoc-3.13.0.0 2024-06-26T07:11:14.2277266Z inflating: build/bin/protoc 2024-06-26T07:11:14.2328955Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-06-26T07:11:14.2380237Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-06-26T07:11:14.2431149Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-06-26T07:11:14.2481415Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-06-26T07:11:14.2532792Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-06-26T07:11:14.2583937Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-06-26T07:11:14.2632190Z inflating: build/bin/c10_cuda_CUDATest 2024-06-26T07:11:14.2682543Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-06-26T07:11:14.3001565Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-06-26T07:11:14.3333868Z inflating: build/bin/vec_test_all_types_AVX512 2024-06-26T07:11:14.3680136Z inflating: build/bin/vec_test_all_types_AVX2 2024-06-26T07:11:14.3732112Z inflating: build/bin/FileStoreTest 2024-06-26T07:11:14.3786571Z inflating: build/bin/TCPStoreTest 2024-06-26T07:11:14.3838240Z inflating: build/bin/HashStoreTest 2024-06-26T07:11:14.3851345Z inflating: build/bin/ProcessGroupMPITest 2024-06-26T07:11:14.3903587Z inflating: build/bin/test_edge_op_registration 2024-06-26T07:11:14.3907439Z inflating: build/bin/torch_shm_manager 2024-06-26T07:11:14.3910176Z inflating: build/bin/example_allreduce 2024-06-26T07:11:14.3963394Z inflating: build/bin/test_dist_autograd 2024-06-26T07:11:14.4031532Z inflating: build/bin/test_cpp_rpc 2024-06-26T07:11:14.4033298Z inflating: build/bin/parallel_benchmark 2024-06-26T07:11:14.4098153Z inflating: build/bin/test_mobile_nnc 2024-06-26T07:11:14.4106613Z inflating: build/bin/aot_model_compiler_test 2024-06-26T07:11:14.4439779Z inflating: build/bin/test_lazy 2024-06-26T07:11:14.5583745Z inflating: build/bin/test_api 2024-06-26T07:11:14.5655218Z inflating: build/bin/Dict_test 2024-06-26T07:11:14.5705927Z inflating: build/bin/Dimname_test 2024-06-26T07:11:14.5768306Z inflating: build/bin/MaybeOwned_test 2024-06-26T07:11:14.5823723Z inflating: build/bin/NamedTensor_test 2024-06-26T07:11:14.5880975Z inflating: build/bin/apply_utils_test 2024-06-26T07:11:14.5938300Z inflating: build/bin/atest 2024-06-26T07:11:14.5999231Z inflating: build/bin/basic 2024-06-26T07:11:14.6052403Z inflating: build/bin/broadcast_test 2024-06-26T07:11:14.6101553Z inflating: build/bin/cpu_allocator_test 2024-06-26T07:11:14.6157783Z inflating: build/bin/cpu_generator_test 2024-06-26T07:11:14.6209274Z inflating: build/bin/cpu_profiling_allocator_test 2024-06-26T07:11:14.6299693Z inflating: build/bin/cpu_rng_test 2024-06-26T07:11:14.6348637Z inflating: build/bin/dispatch_key_set_test 2024-06-26T07:11:14.6396841Z inflating: build/bin/dlconvertor_test 2024-06-26T07:11:14.6453279Z inflating: build/bin/extension_backend_test 2024-06-26T07:11:14.6507215Z inflating: build/bin/half_test 2024-06-26T07:11:14.6599582Z inflating: build/bin/ivalue_test 2024-06-26T07:11:14.6647924Z inflating: build/bin/lazy_tensor_test 2024-06-26T07:11:14.6700653Z inflating: build/bin/math_kernel_test 2024-06-26T07:11:14.6753338Z inflating: build/bin/memory_format_test 2024-06-26T07:11:14.6804469Z inflating: build/bin/memory_overlapping_test 2024-06-26T07:11:14.6855792Z inflating: build/bin/mobile_memory_cleanup 2024-06-26T07:11:14.6910025Z inflating: build/bin/native_test 2024-06-26T07:11:14.6959093Z inflating: build/bin/operator_name_test 2024-06-26T07:11:14.7008984Z inflating: build/bin/operators_test 2024-06-26T07:11:14.7059125Z inflating: build/bin/packedtensoraccessor_test 2024-06-26T07:11:14.7124194Z inflating: build/bin/pow_test 2024-06-26T07:11:14.7180350Z inflating: build/bin/quantized_test 2024-06-26T07:11:14.7228743Z inflating: build/bin/reduce_ops_test 2024-06-26T07:11:14.7278321Z inflating: build/bin/reportMemoryUsage_test 2024-06-26T07:11:14.7334040Z inflating: build/bin/scalar_tensor_test 2024-06-26T07:11:14.7390681Z inflating: build/bin/scalar_test 2024-06-26T07:11:14.7440801Z inflating: build/bin/StorageUtils_test 2024-06-26T07:11:14.7491748Z inflating: build/bin/stride_properties_test 2024-06-26T07:11:14.7567946Z inflating: build/bin/tensor_iterator_test 2024-06-26T07:11:14.7620328Z inflating: build/bin/test_parallel 2024-06-26T07:11:14.7623007Z inflating: build/bin/thread_init_test 2024-06-26T07:11:14.7677129Z inflating: build/bin/type_ptr_test 2024-06-26T07:11:14.7735463Z inflating: build/bin/type_test 2024-06-26T07:11:14.7786552Z inflating: build/bin/undefined_tensor_test 2024-06-26T07:11:14.7787548Z inflating: build/bin/verify_api_visibility 2024-06-26T07:11:14.7855384Z inflating: build/bin/legacy_vmap_test 2024-06-26T07:11:14.7905898Z inflating: build/bin/weakref_test 2024-06-26T07:11:14.7955648Z inflating: build/bin/wrapdim_test 2024-06-26T07:11:14.8005361Z inflating: build/bin/xla_tensor_test 2024-06-26T07:11:14.8063392Z inflating: build/bin/IListRef_test 2024-06-26T07:11:14.8166308Z inflating: build/bin/List_test 2024-06-26T07:11:14.8230608Z inflating: build/bin/KernelFunction_test 2024-06-26T07:11:14.8348923Z inflating: build/bin/kernel_function_legacy_test 2024-06-26T07:11:14.8442460Z inflating: build/bin/kernel_function_test 2024-06-26T07:11:14.8566699Z inflating: build/bin/kernel_lambda_legacy_test 2024-06-26T07:11:14.8666956Z inflating: build/bin/kernel_lambda_test 2024-06-26T07:11:14.8726157Z inflating: build/bin/kernel_stackbased_test 2024-06-26T07:11:14.8818957Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-06-26T07:11:14.8868739Z inflating: build/bin/CppSignature_test 2024-06-26T07:11:14.8922240Z inflating: build/bin/backend_fallback_test 2024-06-26T07:11:14.8969587Z inflating: build/bin/op_allowlist_test 2024-06-26T07:11:14.9260934Z inflating: build/bin/op_registration_test 2024-06-26T07:11:14.9322433Z inflating: build/bin/inline_container_test 2024-06-26T07:11:14.9373584Z inflating: build/bin/cuda_apply_test 2024-06-26T07:11:14.9422714Z inflating: build/bin/cuda_allocator_test 2024-06-26T07:11:14.9479918Z inflating: build/bin/cuda_atomic_ops_test 2024-06-26T07:11:14.9532698Z inflating: build/bin/cuda_caching_host_allocator_test 2024-06-26T07:11:14.9600174Z inflating: build/bin/cuda_complex_math_test 2024-06-26T07:11:14.9657954Z inflating: build/bin/cuda_complex_test 2024-06-26T07:11:14.9705947Z inflating: build/bin/cuda_device_test 2024-06-26T07:11:14.9761160Z inflating: build/bin/cuda_cub_test 2024-06-26T07:11:14.9811385Z inflating: build/bin/cuda_dlconvertor_test 2024-06-26T07:11:14.9873958Z inflating: build/bin/cuda_distributions_test 2024-06-26T07:11:14.9928883Z inflating: build/bin/cuda_generator_test 2024-06-26T07:11:14.9977001Z inflating: build/bin/cuda_half_test 2024-06-26T07:11:15.0027495Z inflating: build/bin/cuda_integer_divider_test 2024-06-26T07:11:15.0074460Z inflating: build/bin/cuda_optional_test 2024-06-26T07:11:15.0124679Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-06-26T07:11:15.0175659Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-06-26T07:11:15.0224058Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-06-26T07:11:15.0282807Z inflating: build/bin/cuda_stream_test 2024-06-26T07:11:15.0330656Z inflating: build/bin/cuda_cudnn_test 2024-06-26T07:11:15.0380891Z inflating: build/bin/cuda_vectorized_test 2024-06-26T07:11:15.0395326Z inflating: build/bin/tutorial_tensorexpr 2024-06-26T07:11:15.0458896Z inflating: build/bin/ProcessGroupGlooTest 2024-06-26T07:11:15.0514445Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-06-26T07:11:15.0575591Z inflating: build/bin/ProcessGroupNCCLTest 2024-06-26T07:11:15.0635564Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-06-26T07:11:15.1440513Z inflating: build/bin/test_tensorexpr 2024-06-26T07:11:15.1998918Z inflating: build/bin/test_jit 2024-06-26T07:11:15.2000040Z creating: .additional_ci_files/ 2024-06-26T07:11:15.2053098Z inflating: .additional_ci_files/test-times.json 2024-06-26T07:11:15.2262192Z inflating: .additional_ci_files/test-class-times.json 2024-06-26T07:11:15.2287699Z ##[group]Run rm artifacts.zip 2024-06-26T07:11:15.2288084Z rm artifacts.zip 2024-06-26T07:11:15.2297231Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:15.2297730Z env: 2024-06-26T07:11:15.2298023Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:15.2298467Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:15.2298955Z ##[endgroup] 2024-06-26T07:11:15.3010394Z ##[group]Run df -H 2024-06-26T07:11:15.3010722Z df -H 2024-06-26T07:11:15.3018322Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:15.3018851Z env: 2024-06-26T07:11:15.3019129Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:15.3019607Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:15.3020134Z ##[endgroup] 2024-06-26T07:11:15.3046076Z Filesystem Size Used Avail Use% Mounted on 2024-06-26T07:11:15.3046736Z devtmpfs 34G 0 34G 0% /dev 2024-06-26T07:11:15.3047664Z tmpfs 34G 599k 34G 1% /dev/shm 2024-06-26T07:11:15.3048133Z tmpfs 34G 455k 34G 1% /run 2024-06-26T07:11:15.3048617Z tmpfs 34G 0 34G 0% /sys/fs/cgroup 2024-06-26T07:11:15.3049084Z /dev/nvme0n1p1 162G 67G 95G 42% / 2024-06-26T07:11:15.3087923Z Prepare all required actions 2024-06-26T07:11:15.3088362Z Getting action download info 2024-06-26T07:11:15.4110269Z ##[group]Run ./.github/actions/download-td-artifacts 2024-06-26T07:11:15.4110737Z with: 2024-06-26T07:11:15.4110993Z env: 2024-06-26T07:11:15.4111270Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:15.4111716Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:15.4112200Z ##[endgroup] 2024-06-26T07:11:15.4142483Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:11:15.4142944Z with: 2024-06-26T07:11:15.4143212Z name: td_results 2024-06-26T07:11:15.4143535Z s3-bucket: gha-artifacts 2024-06-26T07:11:15.4143887Z region: us-east-1 2024-06-26T07:11:15.4144195Z env: 2024-06-26T07:11:15.4144466Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:15.4144920Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:15.4145404Z ##[endgroup] 2024-06-26T07:11:15.8193785Z (node:26788) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:11:15.8194611Z 2024-06-26T07:11:15.8194868Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:11:15.8195580Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:11:15.8196395Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:11:15.8861300Z Found 0 objects with prefix pytorch/pytorch/9673646806/td_results/ 2024-06-26T07:11:15.8868307Z Artifact download has finished successfully 2024-06-26T07:11:15.9001530Z ##[group]Run mkdir -p .additional_ci_files 2024-06-26T07:11:15.9002007Z mkdir -p .additional_ci_files 2024-06-26T07:11:15.9002556Z mv td_results.json .additional_ci_files/td_results.json 2024-06-26T07:11:15.9010685Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:15.9011169Z env: 2024-06-26T07:11:15.9011450Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:15.9011913Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:15.9012392Z ##[endgroup] 2024-06-26T07:11:15.9046317Z mv: cannot stat 'td_results.json': No such file or directory 2024-06-26T07:11:15.9058512Z ##[error]Process completed with exit code 1. 2024-06-26T07:11:15.9104828Z ##[group]Run .github/scripts/parse_ref.py 2024-06-26T07:11:15.9105298Z .github/scripts/parse_ref.py 2024-06-26T07:11:15.9112392Z shell: /usr/bin/bash -e {0} 2024-06-26T07:11:15.9112731Z env: 2024-06-26T07:11:15.9113025Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:15.9113612Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:15.9114091Z ##[endgroup] 2024-06-26T07:11:15.9341260Z Prepare all required actions 2024-06-26T07:11:15.9377401Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-06-26T07:11:15.9377847Z with: 2024-06-26T07:11:15.9378385Z github-token: *** 2024-06-26T07:11:15.9378690Z env: 2024-06-26T07:11:15.9378957Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:15.9379408Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:15.9379889Z ##[endgroup] 2024-06-26T07:11:15.9394849Z ##[group]Run set -eux 2024-06-26T07:11:15.9395191Z set -eux 2024-06-26T07:11:15.9395821Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-06-26T07:11:15.9403373Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:15.9404007Z env: 2024-06-26T07:11:15.9404299Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:15.9404757Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:15.9405465Z GITHUB_TOKEN: *** 2024-06-26T07:11:15.9405777Z ##[endgroup] 2024-06-26T07:11:15.9425387Z + python3 .github/scripts/get_workflow_job_id.py 9673646806 i-0a96abc6f4fd8966c 2024-06-26T07:11:16.5336543Z setting job-id=26688916472 2024-06-26T07:11:16.5337563Z setting job-name=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:11:16.5530052Z Prepare all required actions 2024-06-26T07:11:16.5530500Z Getting action download info 2024-06-26T07:11:16.6596464Z ##[group]Run ./.github/actions/filter-test-configs 2024-06-26T07:11:16.6596915Z with: 2024-06-26T07:11:16.6597357Z github-token: *** 2024-06-26T07:11:16.6604029Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:11:16.6611391Z job-name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:11:16.6612271Z env: 2024-06-26T07:11:16.6612543Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:16.6612996Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:16.6613476Z ##[endgroup] 2024-06-26T07:11:16.6653385Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:11:16.6653935Z with: 2024-06-26T07:11:16.6654212Z shell: bash 2024-06-26T07:11:16.6654526Z timeout_minutes: 10 2024-06-26T07:11:16.6654877Z max_attempts: 5 2024-06-26T07:11:16.6655337Z retry_wait_seconds: 30 2024-06-26T07:11:16.6656503Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:11:16.6657657Z polling_interval_seconds: 1 2024-06-26T07:11:16.6658048Z warning_on_retry: true 2024-06-26T07:11:16.6658403Z continue_on_error: false 2024-06-26T07:11:16.6658758Z env: 2024-06-26T07:11:16.6659041Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:16.6659501Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:16.6660131Z GITHUB_TOKEN: *** 2024-06-26T07:11:16.6660460Z ##[endgroup] 2024-06-26T07:11:16.7144210Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:11:16.9304305Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:11:16.9458732Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1) 2024-06-26T07:11:16.9603873Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-06-26T07:11:16.9612953Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12) 2024-06-26T07:11:16.9633954Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19) 2024-06-26T07:11:16.9834085Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.6.2) 2024-06-26T07:11:16.9840715Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7) 2024-06-26T07:11:17.7149514Z Command completed after 1 attempt(s). 2024-06-26T07:11:17.7189973Z ##[group]Run set -x 2024-06-26T07:11:17.7190304Z set -x 2024-06-26T07:11:17.7190609Z  2024-06-26T07:11:17.7191358Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:11:17.7192020Z # in runner workspace 2024-06-26T07:11:17.7192540Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-06-26T07:11:17.7200393Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:17.7200883Z env: 2024-06-26T07:11:17.7201172Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:17.7201623Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:17.7202108Z ##[endgroup] 2024-06-26T07:11:17.7223285Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-06-26T07:11:17.7420902Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:11:17.7421449Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:11:17.7421922Z echo "Job name: ${JOB_NAME}" 2024-06-26T07:11:17.7422344Z  2024-06-26T07:11:17.7422899Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:11:17.7423573Z # in runner workspace 2024-06-26T07:11:17.7424164Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-06-26T07:11:17.7424809Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-06-26T07:11:17.7425295Z  --job-name "${JOB_NAME}" \ 2024-06-26T07:11:17.7431956Z  --test-matrix "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-06-26T07:11:17.7438937Z  --selected-test-configs "" \ 2024-06-26T07:11:17.7439398Z  --pr-number "${PR_NUMBER}" \ 2024-06-26T07:11:17.7439840Z  --tag "${TAG}" \ 2024-06-26T07:11:17.7440252Z  --event-name "${EVENT_NAME}" \ 2024-06-26T07:11:17.7440707Z  --schedule "${SCHEDULE}" \ 2024-06-26T07:11:17.7458050Z  --branch "${HEAD_BRANCH}" 2024-06-26T07:11:17.7465935Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:17.7466438Z env: 2024-06-26T07:11:17.7466729Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:17.7467204Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:17.7467960Z GITHUB_TOKEN: *** 2024-06-26T07:11:17.7468788Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:11:17.7469678Z PR_NUMBER: 2024-06-26T07:11:17.7470005Z TAG: ciflow/inductor/129470 2024-06-26T07:11:17.7470366Z EVENT_NAME: push 2024-06-26T07:11:17.7470678Z SCHEDULE: 2024-06-26T07:11:17.7470973Z HEAD_BRANCH: 2024-06-26T07:11:17.7471276Z ##[endgroup] 2024-06-26T07:11:17.7490828Z Workflow: inductor-periodic 2024-06-26T07:11:17.7491902Z Job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:11:17.9468479Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-06-26T07:11:18.0897848Z ##[group]Run echo "Filtered matrix:" 2024-06-26T07:11:18.0898299Z echo "Filtered matrix:" 2024-06-26T07:11:18.0904776Z echo "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-06-26T07:11:18.0911783Z  2024-06-26T07:11:18.0912058Z echo 2024-06-26T07:11:18.0912441Z echo "Is the current job unstable? False" 2024-06-26T07:11:18.0912898Z  2024-06-26T07:11:18.0913165Z echo 2024-06-26T07:11:18.0913531Z echo "Is keep-going label set? False" 2024-06-26T07:11:18.0913974Z  2024-06-26T07:11:18.0914241Z echo 2024-06-26T07:11:18.0914565Z echo "Renabled issues? " 2024-06-26T07:11:18.0922259Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:18.0922752Z env: 2024-06-26T07:11:18.0923034Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:18.0923490Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:18.0924286Z ##[endgroup] 2024-06-26T07:11:18.0942969Z Filtered matrix: 2024-06-26T07:11:18.0951101Z {include: [{config: dynamo_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-06-26T07:11:18.0957742Z 2024-06-26T07:11:18.0957903Z Is the current job unstable? False 2024-06-26T07:11:18.0958188Z 2024-06-26T07:11:18.0958473Z Is keep-going label set? False 2024-06-26T07:11:18.0958725Z 2024-06-26T07:11:18.0958855Z Renabled issues? 2024-06-26T07:11:18.0997152Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:11:18.0997846Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:11:18.1004877Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:11:18.1005369Z env: 2024-06-26T07:11:18.1005698Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:18.1006154Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:18.1006637Z JOB_TIMEOUT: 240 2024-06-26T07:11:18.1006942Z ##[endgroup] 2024-06-26T07:11:18.1076105Z ##[group]Run set -x 2024-06-26T07:11:18.1076596Z set -x 2024-06-26T07:11:18.1076898Z  2024-06-26T07:11:18.1077251Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-06-26T07:11:18.1077786Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-06-26T07:11:18.1078345Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-06-26T07:11:18.1078858Z  TEST_COMMAND=.ci/onnx/test.sh 2024-06-26T07:11:18.1079269Z else 2024-06-26T07:11:18.1079606Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:11:18.1080020Z fi 2024-06-26T07:11:18.1080299Z  2024-06-26T07:11:18.1080906Z # detached container should get cleaned up by teardown_ec2_linux 2024-06-26T07:11:18.1081651Z # TODO: Stop building test binaries as part of the build phase 2024-06-26T07:11:18.1082335Z # Used for GPU_FLAG since that doesn't play nice 2024-06-26T07:11:18.1082932Z # shellcheck disable=SC2086,SC2090 2024-06-26T07:11:18.1083395Z container_name=$(docker run \ 2024-06-26T07:11:18.1083957Z  ${GPU_FLAG:-} \ 2024-06-26T07:11:18.1084333Z  -e BUILD_ENVIRONMENT \ 2024-06-26T07:11:18.1084731Z  -e PR_NUMBER \ 2024-06-26T07:11:18.1085088Z  -e GITHUB_ACTIONS \ 2024-06-26T07:11:18.1085491Z  -e GITHUB_REPOSITORY \ 2024-06-26T07:11:18.1085899Z  -e GITHUB_WORKFLOW \ 2024-06-26T07:11:18.1086285Z  -e GITHUB_JOB \ 2024-06-26T07:11:18.1086635Z  -e GITHUB_RUN_ID \ 2024-06-26T07:11:18.1087025Z  -e GITHUB_RUN_NUMBER \ 2024-06-26T07:11:18.1087446Z  -e GITHUB_RUN_ATTEMPT \ 2024-06-26T07:11:18.1087832Z  -e JOB_ID \ 2024-06-26T07:11:18.1088180Z  -e JOB_NAME \ 2024-06-26T07:11:18.1088529Z  -e BASE_SHA \ 2024-06-26T07:11:18.1088879Z  -e BRANCH \ 2024-06-26T07:11:18.1089215Z  -e SHA1 \ 2024-06-26T07:11:18.1089565Z  -e AWS_DEFAULT_REGION \ 2024-06-26T07:11:18.1089974Z  -e IN_WHEEL_TEST \ 2024-06-26T07:11:18.1090352Z  -e SHARD_NUMBER \ 2024-06-26T07:11:18.1090711Z  -e TEST_CONFIG \ 2024-06-26T07:11:18.1091081Z  -e NUM_TEST_SHARDS \ 2024-06-26T07:11:18.1091481Z  -e REENABLED_ISSUES \ 2024-06-26T07:11:18.1091898Z  -e CONTINUE_THROUGH_ERROR \ 2024-06-26T07:11:18.1092317Z  -e VERBOSE_TEST_LOGS \ 2024-06-26T07:11:18.1092715Z  -e NO_TEST_TIMEOUT \ 2024-06-26T07:11:18.1093098Z  -e NO_TD \ 2024-06-26T07:11:18.1093439Z  -e TD_DISTRIBUTED \ 2024-06-26T07:11:18.1093829Z  -e PR_LABELS \ 2024-06-26T07:11:18.1094241Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-06-26T07:11:18.1094698Z  -e SCCACHE_BUCKET \ 2024-06-26T07:11:18.1095087Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-06-26T07:11:18.1095495Z  -e XLA_CUDA \ 2024-06-26T07:11:18.1095925Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-06-26T07:11:18.1096458Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-06-26T07:11:18.1096973Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-06-26T07:11:18.1097481Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-06-26T07:11:18.1097954Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-06-26T07:11:18.1098387Z  -e DASHBOARD_TAG \ 2024-06-26T07:11:18.1098865Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-06-26T07:11:18.1099432Z  --security-opt seccomp=unconfined \ 2024-06-26T07:11:18.1099900Z  --cap-add=SYS_PTRACE \ 2024-06-26T07:11:18.1100298Z  --ipc=host \ 2024-06-26T07:11:18.1100656Z  --shm-size="${SHM_SIZE}" \ 2024-06-26T07:11:18.1101055Z  --tty \ 2024-06-26T07:11:18.1101381Z  --detach \ 2024-06-26T07:11:18.1101749Z  --name="${container_name}" \ 2024-06-26T07:11:18.1102160Z  --user jenkins \ 2024-06-26T07:11:18.1102738Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-06-26T07:11:18.1103312Z  -w /var/lib/jenkins/workspace \ 2024-06-26T07:11:18.1103751Z  "${DOCKER_IMAGE}" 2024-06-26T07:11:18.1104100Z ) 2024-06-26T07:11:18.1104521Z # Propagate download.pytorch.org IP to container 2024-06-26T07:11:18.1105440Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-06-26T07:11:18.1106404Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-06-26T07:11:18.1107315Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-06-26T07:11:18.1115194Z shell: /usr/bin/bash -e {0} 2024-06-26T07:11:18.1115557Z env: 2024-06-26T07:11:18.1115885Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:11:18.1116335Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:18.1116957Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:11:18.1117466Z PR_NUMBER: 2024-06-26T07:11:18.1117795Z GITHUB_REPOSITORY: pytorch/pytorch 2024-06-26T07:11:18.1118228Z GITHUB_WORKFLOW: inductor-periodic 2024-06-26T07:11:18.1118636Z GITHUB_JOB: test 2024-06-26T07:11:18.1118968Z GITHUB_RUN_ID: 9673646806 2024-06-26T07:11:18.1119329Z GITHUB_RUN_NUMBER: 45567 2024-06-26T07:11:18.1119675Z GITHUB_RUN_ATTEMPT: 1 2024-06-26T07:11:18.1120011Z JOB_ID: 26688916472 2024-06-26T07:11:18.1120846Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:11:18.1121723Z BRANCH: 2024-06-26T07:11:18.1122067Z SHA1: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:18.1122577Z BASE_SHA: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:18.1123056Z TEST_CONFIG: aot_eager_huggingface 2024-06-26T07:11:18.1123453Z SHARD_NUMBER: 1 2024-06-26T07:11:18.1123878Z NUM_TEST_SHARDS: 1 2024-06-26T07:11:18.1124212Z REENABLED_ISSUES: 2024-06-26T07:11:18.1124566Z CONTINUE_THROUGH_ERROR: False 2024-06-26T07:11:18.1124948Z VERBOSE_TEST_LOGS: False 2024-06-26T07:11:18.1125315Z NO_TEST_TIMEOUT: False 2024-06-26T07:11:18.1125654Z NO_TD: False 2024-06-26T07:11:18.1125981Z TD_DISTRIBUTED: False 2024-06-26T07:11:18.1126416Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-06-26T07:11:18.1126929Z SCCACHE_S3_KEY_PREFIX: inductor-periodic 2024-06-26T07:11:18.1127355Z SHM_SIZE: 2g 2024-06-26T07:11:18.1128428Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:11:18.1129563Z XLA_CUDA: 2024-06-26T07:11:18.1130041Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:11:18.1130644Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-06-26T07:11:18.1131081Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-06-26T07:11:18.1131482Z DASHBOARD_TAG: 2024-06-26T07:11:18.1131992Z HUGGING_FACE_HUB_TOKEN: *** 2024-06-26T07:11:18.1132357Z ##[endgroup] 2024-06-26T07:11:18.1151600Z + [[ aot_eager_huggingface == \m\u\l\t\i\g\p\u ]] 2024-06-26T07:11:18.1152302Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-06-26T07:11:18.1152813Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:11:18.1158957Z +++ nproc --ignore=2 2024-06-26T07:11:18.1171774Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9673646806 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:11:29.3216034Z + container_name=b752ec5bddbf78c4c938e2d957d66195460cd43fb409c9c551e6c40eb7d6a5db 2024-06-26T07:11:29.3217192Z + grep download.pytorch.org /etc/hosts 2024-06-26T07:11:29.3218336Z + docker exec -i b752ec5bddbf78c4c938e2d957d66195460cd43fb409c9c551e6c40eb7d6a5db sudo bash -c '/bin/cat >> /etc/hosts' 2024-06-26T07:11:29.3927321Z + echo DOCKER_CONTAINER_ID=b752ec5bddbf78c4c938e2d957d66195460cd43fb409c9c551e6c40eb7d6a5db 2024-06-26T07:11:29.3930074Z ++ echo dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:11:29.3931986Z + docker exec -t b752ec5bddbf78c4c938e2d957d66195460cd43fb409c9c551e6c40eb7d6a5db sh -c 'pip install dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-06-26T07:11:29.7719080Z Processing ./dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+gitb8c4c54) 2024-06-26T07:11:30.0876007Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.13.1) 2024-06-26T07:11:30.0879872Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (4.12.2) 2024-06-26T07:11:30.0883186Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.12.1) 2024-06-26T07:11:30.0886824Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.8.8) 2024-06-26T07:11:30.0890893Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.1.4) 2024-06-26T07:11:30.0893460Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2024.2.0) 2024-06-26T07:11:30.0920807Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.3.0) 2024-06-26T07:11:30.0979689Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.21.2) 2024-06-26T07:11:30.1419084Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.1.5) 2024-06-26T07:11:30.1578877Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.3.0) 2024-06-26T07:11:31.4668422Z Installing collected packages: torch 2024-06-26T07:11:40.1572336Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-06-26T07:11:40.1573997Z timm 0.9.7 requires torchvision, which is not installed. 2024-06-26T07:11:40.1575667Z Successfully installed torch-2.5.0a0+gitb8c4c54 2024-06-26T07:11:40.2483115Z ++ dirname .ci/pytorch/test.sh 2024-06-26T07:11:40.2487724Z + source .ci/pytorch/common.sh 2024-06-26T07:11:40.2490239Z +++ dirname .ci/pytorch/common.sh 2024-06-26T07:11:40.2496946Z ++ source .ci/pytorch/common_utils.sh 2024-06-26T07:11:40.2498638Z +++ declare -f -t trap_add 2024-06-26T07:11:40.2504968Z ++ set -ex 2024-06-26T07:11:40.2505575Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:11:40.2506093Z ++ BUILD_TEST_LIBTORCH=0 2024-06-26T07:11:40.2506581Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-06-26T07:11:40.2508391Z ++ stat -c %u /var/lib/jenkins/workspace 2024-06-26T07:11:40.2516337Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-06-26T07:11:40.2516887Z + trap_add cleanup_workspace EXIT 2024-06-26T07:11:40.2517303Z + trap_add_cmd=cleanup_workspace 2024-06-26T07:11:40.2517670Z + shift 2024-06-26T07:11:40.2518136Z + for trap_add_name in "$@" 2024-06-26T07:11:40.2521719Z +++ trap -p EXIT 2024-06-26T07:11:40.2523356Z ++ eval 'extract_trap_cmd ' 2024-06-26T07:11:40.2524069Z +++ extract_trap_cmd 2024-06-26T07:11:40.2524606Z +++ printf '%s\n' '' 2024-06-26T07:11:40.2525174Z ++ printf '%s\n' cleanup_workspace 2024-06-26T07:11:40.2525682Z + trap -- ' 2024-06-26T07:11:40.2526120Z cleanup_workspace' EXIT 2024-06-26T07:11:40.2526771Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-06-26T07:11:40.7439809Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-06-26T07:11:40.7450435Z + echo 'Environment variables:' 2024-06-26T07:11:40.7450990Z Environment variables: 2024-06-26T07:11:40.7453635Z + env 2024-06-26T07:11:40.7455811Z INSTALLED_DB=yes 2024-06-26T07:11:40.7456384Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:11:40.7456936Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:11:40.7457457Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:11:40.7458247Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:11:40.7459015Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:11:40.7459744Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:11:40.7460328Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:11:40.7460983Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:11:40.7461622Z HOSTNAME=b752ec5bddbf 2024-06-26T07:11:40.7462780Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_f895147a-a9a7-4a06-a280-cdf3a4586b2a 2024-06-26T07:11:40.7463809Z GITHUB_ACTION=__self 2024-06-26T07:11:40.7464156Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:11:40.7468549Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:11:40.7472382Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:11:40.7472881Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:11:40.7473264Z GITHUB_RUN_NUMBER=45567 2024-06-26T07:11:40.7473611Z TEST_CONFIG=aot_eager_huggingface 2024-06-26T07:11:40.7474013Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:11:40.7474476Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:11:40.7474922Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:11:40.7475363Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:11:40.7475779Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:11:40.7476220Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:11:40.7476703Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:11:40.7477159Z GITHUB_REF_TYPE=tag 2024-06-26T07:11:40.7477480Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:11:40.7477850Z NCCL_VERSION=2.17.1-1 2024-06-26T07:11:40.7478600Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:40.7479049Z XLA_CUDA= 2024-06-26T07:11:40.7479546Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:11:40.7480175Z *** 2024-06-26T07:11:40.7480456Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:11:40.7480849Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:11:40.7481213Z GITHUB_ACTIONS=true 2024-06-26T07:11:40.7481543Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:40.7482027Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:11:40.7482556Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:11:40.7483013Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:40.7483500Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:11:40.7484694Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:40.7485739Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:11:40.7486557Z UCC_HOME=/usr 2024-06-26T07:11:40.7486893Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:11:40.7487289Z VERBOSE_TEST_LOGS=False 2024-06-26T07:11:40.7487625Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:11:40.7488067Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:11:40.7488548Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:11:40.7488999Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:11:40.7489368Z SHARD_NUMBER=1 2024-06-26T07:11:40.7489671Z GITHUB_REF_PROTECTED=false 2024-06-26T07:11:40.7490080Z HOME=/var/lib/jenkins 2024-06-26T07:11:40.7490443Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:11:40.7490877Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:11:40.7491342Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:11:40.7491880Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-06-26T07:11:40.7492288Z CUDA_VERSION=12.1.1 2024-06-26T07:11:40.7492698Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:11:40.7493123Z NUM_TEST_SHARDS=1 2024-06-26T07:11:40.7493417Z UCX_HOME=/usr 2024-06-26T07:11:40.7493921Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:11:40.7494972Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_f895147a-a9a7-4a06-a280-cdf3a4586b2a 2024-06-26T07:11:40.7496347Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:11:40.7497767Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_f895147a-a9a7-4a06-a280-cdf3a4586b2a 2024-06-26T07:11:40.7498855Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:11:40.7499502Z GITHUB_EVENT_NAME=push 2024-06-26T07:11:40.7499836Z DASHBOARD_TAG= 2024-06-26T07:11:40.7500127Z GITHUB_RUN_ID=9673646806 2024-06-26T07:11:40.7500573Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:11:40.7501098Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:11:40.7502083Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_f895147a-a9a7-4a06-a280-cdf3a4586b2a 2024-06-26T07:11:40.7502982Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:11:40.7503380Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:11:40.7503753Z PR_NUMBER= 2024-06-26T07:11:40.7504025Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:11:40.7504355Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:11:40.7504789Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:11:40.7505239Z TERM=xterm 2024-06-26T07:11:40.7505558Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:11:40.7505963Z INSTALLED_VISION=yes 2024-06-26T07:11:40.7506266Z BRANCH= 2024-06-26T07:11:40.7506540Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:11:40.7506935Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:11:40.7507354Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:11:40.7508208Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:11:40.7508941Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:11:40.7509579Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:11:40.7510043Z REENABLED_ISSUES= 2024-06-26T07:11:40.7510352Z SHLVL=1 2024-06-26T07:11:40.7510662Z MAX_JOBS=14 2024-06-26T07:11:40.7510980Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:11:40.7511325Z NVARCH=x86_64 2024-06-26T07:11:40.7511613Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:11:40.7512056Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:40.7512574Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:11:40.7513052Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:11:40.7513692Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:11:40.7514255Z GITHUB_JOB=test 2024-06-26T07:11:40.7514750Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:11:40.7515292Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:11:40.7515787Z NO_TEST_TIMEOUT=False 2024-06-26T07:11:40.7516107Z TD_DISTRIBUTED=False 2024-06-26T07:11:40.7516473Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:11:40.7516915Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:11:40.7517319Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:11:40.7517674Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:11:40.7518018Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:11:40.7518355Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:11:40.7519374Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:40.7520410Z GITHUB_BASE_REF= 2024-06-26T07:11:40.7520728Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:11:40.7521113Z CI=true 2024-06-26T07:11:40.7521441Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:11:40.7521850Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:11:40.7522215Z JOB_ID=26688916472 2024-06-26T07:11:40.7522517Z INSTALLED_PROTOBUF=yes 2024-06-26T07:11:40.7522838Z GITHUB_HEAD_REF= 2024-06-26T07:11:40.7523133Z GITHUB_ACTION_REF= 2024-06-26T07:11:40.7523545Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:11:40.7524409Z GITHUB_WORKFLOW=inductor-periodic 2024-06-26T07:11:40.7524811Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:11:40.7525695Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_f895147a-a9a7-4a06-a280-cdf3a4586b2a 2024-06-26T07:11:40.7526482Z NO_TD=False 2024-06-26T07:11:40.7526783Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:11:40.7527135Z _=/usr/bin/env 2024-06-26T07:11:40.7527621Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-06-26T07:11:40.7599920Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-06-26T07:11:40.7601052Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T07:11:40.7602100Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-06-26T07:11:40.7602960Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-06-26T07:11:40.7603557Z + BUILD_DIR=build 2024-06-26T07:11:40.7604053Z + BUILD_RENAMED_DIR=build_renamed 2024-06-26T07:11:40.7604459Z + BUILD_BIN_DIR=build/bin 2024-06-26T07:11:40.7604797Z + SHARD_NUMBER=1 2024-06-26T07:11:40.7605094Z + NUM_TEST_SHARDS=1 2024-06-26T07:11:40.7605411Z + export VALGRIND=ON 2024-06-26T07:11:40.7605727Z + VALGRIND=ON 2024-06-26T07:11:40.7606179Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-06-26T07:11:40.7606657Z + [[ 0 == \1 ]] 2024-06-26T07:11:40.7606947Z + [[ False == \1 ]] 2024-06-26T07:11:40.7607532Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-06-26T07:11:40.7608114Z ++ realpath build/custom_test_artifacts 2024-06-26T07:11:40.7609728Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-06-26T07:11:40.7610470Z + [[ -n '' ]] 2024-06-26T07:11:40.7610825Z + echo 'Environment variables' 2024-06-26T07:11:40.7611272Z Environment variables 2024-06-26T07:11:40.7611591Z + env 2024-06-26T07:11:40.7615433Z INSTALLED_DB=yes 2024-06-26T07:11:40.7615947Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:11:40.7616963Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:11:40.7617385Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:11:40.7618067Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:11:40.7618895Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:11:40.7619551Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:11:40.7620212Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:11:40.7620980Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:11:40.7621518Z HOSTNAME=b752ec5bddbf 2024-06-26T07:11:40.7622419Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_f895147a-a9a7-4a06-a280-cdf3a4586b2a 2024-06-26T07:11:40.7623422Z GITHUB_ACTION=__self 2024-06-26T07:11:40.7623759Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:11:40.7627395Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:11:40.7631263Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:11:40.7631772Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:11:40.7632116Z GITHUB_RUN_NUMBER=45567 2024-06-26T07:11:40.7632460Z TEST_CONFIG=aot_eager_huggingface 2024-06-26T07:11:40.7632862Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:11:40.7633316Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:11:40.7633769Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:11:40.7634190Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:11:40.7634599Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:11:40.7635028Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:11:40.7635506Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:11:40.7635951Z GITHUB_REF_TYPE=tag 2024-06-26T07:11:40.7636270Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:11:40.7636636Z NCCL_VERSION=2.17.1-1 2024-06-26T07:11:40.7637008Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:40.7637436Z XLA_CUDA= 2024-06-26T07:11:40.7637927Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:11:40.7638312Z *** 2024-06-26T07:11:40.7638592Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:11:40.7638974Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:11:40.7639329Z GITHUB_ACTIONS=true 2024-06-26T07:11:40.7639653Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:11:40.7640127Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:11:40.7640644Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:11:40.7641101Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:40.7641587Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:11:40.7642049Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:40.7642945Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:11:40.7643869Z UCC_HOME=/usr 2024-06-26T07:11:40.7644202Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:11:40.7644588Z VERBOSE_TEST_LOGS=False 2024-06-26T07:11:40.7644923Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:11:40.7645365Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:11:40.7645859Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:11:40.7646312Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:11:40.7646681Z SHARD_NUMBER=1 2024-06-26T07:11:40.7646986Z GITHUB_REF_PROTECTED=false 2024-06-26T07:11:40.7647323Z HOME=/var/lib/jenkins 2024-06-26T07:11:40.7647682Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:11:40.7648264Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:11:40.7648736Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:11:40.7649257Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-06-26T07:11:40.7649665Z CUDA_VERSION=12.1.1 2024-06-26T07:11:40.7650075Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:11:40.7650502Z NUM_TEST_SHARDS=1 2024-06-26T07:11:40.7650795Z UCX_HOME=/usr 2024-06-26T07:11:40.7651296Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:11:40.7652343Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_f895147a-a9a7-4a06-a280-cdf3a4586b2a 2024-06-26T07:11:40.7653797Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:11:40.7655137Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_f895147a-a9a7-4a06-a280-cdf3a4586b2a 2024-06-26T07:11:40.7656223Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:11:40.7656864Z GITHUB_EVENT_NAME=push 2024-06-26T07:11:40.7657185Z DASHBOARD_TAG= 2024-06-26T07:11:40.7657473Z GITHUB_RUN_ID=9673646806 2024-06-26T07:11:40.7657913Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:11:40.7658434Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:11:40.7659404Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_f895147a-a9a7-4a06-a280-cdf3a4586b2a 2024-06-26T07:11:40.7660282Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:11:40.7660682Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:11:40.7661065Z PR_NUMBER= 2024-06-26T07:11:40.7661379Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:11:40.7661681Z VALGRIND=ON 2024-06-26T07:11:40.7661977Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:11:40.7662412Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:11:40.7662854Z TERM=xterm 2024-06-26T07:11:40.7663184Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:11:40.7663588Z INSTALLED_VISION=yes 2024-06-26T07:11:40.7663891Z BRANCH= 2024-06-26T07:11:40.7664165Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:11:40.7664561Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:11:40.7664979Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:11:40.7665710Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:11:40.7666439Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:11:40.7666915Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:11:40.7667370Z REENABLED_ISSUES= 2024-06-26T07:11:40.7667740Z SHLVL=1 2024-06-26T07:11:40.7684144Z MAX_JOBS=14 2024-06-26T07:11:40.7684485Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:11:40.7684829Z NVARCH=x86_64 2024-06-26T07:11:40.7685126Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:11:40.7685728Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:11:40.7686257Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:11:40.7686740Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:11:40.7687399Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:11:40.7687962Z GITHUB_JOB=test 2024-06-26T07:11:40.7688347Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:11:40.7688896Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:11:40.7689400Z NO_TEST_TIMEOUT=False 2024-06-26T07:11:40.7689730Z TD_DISTRIBUTED=False 2024-06-26T07:11:40.7690096Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:11:40.7690532Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:11:40.7690945Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:11:40.7691309Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:11:40.7691698Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:11:40.7692047Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:11:40.7693083Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:40.7694252Z GITHUB_BASE_REF= 2024-06-26T07:11:40.7694574Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:11:40.7694940Z CI=true 2024-06-26T07:11:40.7695250Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:11:40.7695649Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:11:40.7696017Z JOB_ID=26688916472 2024-06-26T07:11:40.7696324Z INSTALLED_PROTOBUF=yes 2024-06-26T07:11:40.7696645Z GITHUB_HEAD_REF= 2024-06-26T07:11:40.7696936Z GITHUB_ACTION_REF= 2024-06-26T07:11:40.7697355Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:11:40.7697866Z GITHUB_WORKFLOW=inductor-periodic 2024-06-26T07:11:40.7698268Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:11:40.7699223Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_f895147a-a9a7-4a06-a280-cdf3a4586b2a 2024-06-26T07:11:40.7700023Z NO_TD=False 2024-06-26T07:11:40.7700329Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:11:40.7700679Z _=/usr/bin/env 2024-06-26T07:11:40.7701010Z + echo 'Testing pytorch' 2024-06-26T07:11:40.7701355Z Testing pytorch 2024-06-26T07:11:40.7701669Z + export LANG=C.UTF-8 2024-06-26T07:11:40.7701995Z + LANG=C.UTF-8 2024-06-26T07:11:40.7702298Z + PR_NUMBER= 2024-06-26T07:11:40.7702641Z + [[ aot_eager_huggingface == \d\e\f\a\u\l\t ]] 2024-06-26T07:11:40.7703159Z + [[ aot_eager_huggingface == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-06-26T07:11:40.7703645Z + [[ aot_eager_huggingface == \s\l\o\w ]] 2024-06-26T07:11:40.7704269Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-06-26T07:11:40.7704938Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-06-26T07:11:40.7705482Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:11:40.7705946Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:11:40.7706391Z + [[ aot_eager_huggingface == *crossref* ]] 2024-06-26T07:11:40.7706953Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:11:40.7707642Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-06-26T07:11:40.7708271Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:11:40.7708812Z + pip_install --user ninja==1.10.2 2024-06-26T07:11:40.7709331Z + pip install --progress-bar off --user ninja==1.10.2 2024-06-26T07:11:41.1237063Z Collecting ninja==1.10.2 2024-06-26T07:11:41.1384840Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-06-26T07:11:41.1516824Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-06-26T07:11:42.4324770Z Installing collected packages: ninja 2024-06-26T07:11:42.4392709Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-06-26T07:11:42.4393991Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-06-26T07:11:42.4442375Z Successfully installed ninja-1.10.2 2024-06-26T07:11:42.5380626Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:42.5382714Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:42.5384207Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:11:42.5384705Z + install_tlparse 2024-06-26T07:11:42.5385078Z + pip_install --user tlparse==0.3.7 2024-06-26T07:11:42.5385612Z + pip install --progress-bar off --user tlparse==0.3.7 2024-06-26T07:11:42.8915916Z Collecting tlparse==0.3.7 2024-06-26T07:11:42.9086242Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-06-26T07:11:42.9189128Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-06-26T07:11:44.2267661Z Installing collected packages: tlparse 2024-06-26T07:11:44.2635198Z Successfully installed tlparse-0.3.7 2024-06-26T07:11:44.3517383Z ++ python -m site --user-base 2024-06-26T07:11:44.3673947Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:11:44.3675791Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-06-26T07:11:44.3676531Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-06-26T07:11:44.3677363Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:11:44.3678285Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-06-26T07:11:44.3679611Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-06-26T07:11:44.3680296Z + cd test 2024-06-26T07:11:44.3680809Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-06-26T07:11:45.8435828Z + [[ aot_eager_huggingface == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-06-26T07:11:45.8436464Z + [[ aot_eager_huggingface == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-06-26T07:11:45.8437345Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:11:45.8438014Z + pushd test 2024-06-26T07:11:45.8438382Z ~/workspace/test ~/workspace 2024-06-26T07:11:45.8438903Z ++ python -c 'import torch; print(torch.version.cuda)' 2024-06-26T07:11:47.3216563Z + CUDA_VERSION=12.1 2024-06-26T07:11:47.3217155Z + '[' 12.1 == 12.4 ']' 2024-06-26T07:11:47.3217525Z + ISCUDA124= 2024-06-26T07:11:47.3217809Z + popd 2024-06-26T07:11:47.3218076Z ~/workspace 2024-06-26T07:11:47.3221094Z + DYNAMO_BENCHMARK_FLAGS=() 2024-06-26T07:11:47.3221741Z + [[ aot_eager_huggingface == *dynamo_eager* ]] 2024-06-26T07:11:47.3222318Z + [[ aot_eager_huggingface == *aot_eager* ]] 2024-06-26T07:11:47.3222882Z + DYNAMO_BENCHMARK_FLAGS+=(--backend aot_eager) 2024-06-26T07:11:47.3223359Z + [[ aot_eager_huggingface == *dynamic* ]] 2024-06-26T07:11:47.3223834Z + [[ aot_eager_huggingface == *cpu_inductor* ]] 2024-06-26T07:11:47.3224347Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-06-26T07:11:47.3249375Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:11:47.3250060Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-06-26T07:11:47.3251789Z + cd test 2024-06-26T07:11:47.3252318Z + python -c 'import torch; print(torch.__config__.show())' 2024-06-26T07:11:48.7036587Z PyTorch built with: 2024-06-26T07:11:48.7037398Z - GCC 9.4 2024-06-26T07:11:48.7037863Z - C++ Version: 201703 2024-06-26T07:11:48.7039035Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:11:48.7040157Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:11:48.7040814Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:11:48.7041329Z - LAPACK is enabled (usually provided by MKL) 2024-06-26T07:11:48.7041807Z - NNPACK is enabled 2024-06-26T07:11:48.7042195Z - CPU capability usage: AVX2 2024-06-26T07:11:48.7042602Z - CUDA Runtime 12.1 2024-06-26T07:11:48.7043117Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-06-26T07:11:48.7043880Z - CuDNN 90.1 (built against CUDA 12.4) 2024-06-26T07:11:48.7044324Z - Magma 2.6.1 2024-06-26T07:11:48.7051277Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-06-26T07:11:48.7057640Z 2024-06-26T07:11:48.9865095Z + cd test 2024-06-26T07:11:48.9866139Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-06-26T07:11:50.1663809Z ATen/Parallel: 2024-06-26T07:11:50.1664374Z at::get_num_threads() : 8 2024-06-26T07:11:50.1664903Z at::get_num_interop_threads() : 16 2024-06-26T07:11:50.1665440Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:11:50.1665948Z omp_get_max_threads() : 8 2024-06-26T07:11:50.1666913Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:11:50.1667775Z mkl_get_max_threads() : 8 2024-06-26T07:11:50.1668381Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:11:50.1669002Z std::thread::hardware_concurrency() : 16 2024-06-26T07:11:50.1669422Z Environment variables: 2024-06-26T07:11:50.1669763Z OMP_NUM_THREADS : [not set] 2024-06-26T07:11:50.1670128Z MKL_NUM_THREADS : [not set] 2024-06-26T07:11:50.1670498Z ATen parallel backend: OpenMP 2024-06-26T07:11:50.1670743Z 2024-06-26T07:11:50.4079938Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:11:50.4080707Z + [[ aot_eager_huggingface == *backward* ]] 2024-06-26T07:11:50.4081325Z + [[ aot_eager_huggingface == *xla* ]] 2024-06-26T07:11:50.4081926Z + [[ aot_eager_huggingface == *executorch* ]] 2024-06-26T07:11:50.4082612Z + [[ aot_eager_huggingface == \j\i\t\_\l\e\g\a\c\y ]] 2024-06-26T07:11:50.4083508Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:11:50.4084505Z + [[ aot_eager_huggingface == distributed ]] 2024-06-26T07:11:50.4085017Z + [[ aot_eager_huggingface == *inductor_distributed* ]] 2024-06-26T07:11:50.4085604Z + [[ aot_eager_huggingface == *inductor-halide* ]] 2024-06-26T07:11:50.4086216Z + [[ aot_eager_huggingface == *inductor-micro-benchmark* ]] 2024-06-26T07:11:50.4086761Z + [[ aot_eager_huggingface == *huggingface* ]] 2024-06-26T07:11:50.4087193Z + install_torchvision 2024-06-26T07:11:50.4087522Z + local orig_preload 2024-06-26T07:11:50.4087837Z + local commit 2024-06-26T07:11:50.4088152Z ++ get_pinned_commit vision 2024-06-26T07:11:50.4088543Z ++ cat .github/ci_commit_pins/vision.txt 2024-06-26T07:11:50.4099486Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:50.4100077Z + orig_preload= 2024-06-26T07:11:50.4100427Z + '[' -n '' ']' 2024-06-26T07:11:50.4101256Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:50.4102656Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:50.7217913Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:50.7222282Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-mxhf4dgw 2024-06-26T07:11:50.7238732Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-mxhf4dgw 2024-06-26T07:11:52.2648414Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-06-26T07:11:52.2662561Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:53.4813618Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:53.7328704Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:11:56.0478729Z Preparing metadata (setup.py) ... [?25l- \ done 2024-06-26T07:11:56.0526299Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2) 2024-06-26T07:11:56.0529836Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+gitb8c4c54) 2024-06-26T07:11:56.0535768Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-06-26T07:11:56.0739093Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-06-26T07:11:56.0745543Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-06-26T07:11:56.0748883Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12.1) 2024-06-26T07:11:56.0752115Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-06-26T07:11:56.0755099Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-06-26T07:11:56.0758703Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.2.0) 2024-06-26T07:11:56.1181693Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-06-26T07:11:56.1332960Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-06-26T07:11:56.1406169Z Building wheels for collected packages: torchvision 2024-06-26T07:13:09.2196221Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2024-06-26T07:13:09.2222997Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2112853 sha256=002d21a3599560fc59ca59dee03ab0e8b4eb04f2618c9aaf74531385af2bdfe8 2024-06-26T07:13:09.2225222Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-06-26T07:13:09.2258992Z Successfully built torchvision 2024-06-26T07:13:10.4073553Z Installing collected packages: torchvision 2024-06-26T07:13:10.7896518Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-06-26T07:13:10.9039161Z + '[' -n '' ']' 2024-06-26T07:13:10.9039548Z + id=0 2024-06-26T07:13:10.9039923Z + test_dynamo_benchmark huggingface 0 2024-06-26T07:13:10.9042051Z ++ pwd 2024-06-26T07:13:10.9044576Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:13:10.9045259Z + local suite=huggingface 2024-06-26T07:13:10.9045734Z + shift 2024-06-26T07:13:10.9046007Z + local shard_id=0 2024-06-26T07:13:10.9046392Z + shift 2024-06-26T07:13:10.9046796Z + [[ aot_eager_huggingface == *perf_compare* ]] 2024-06-26T07:13:10.9048427Z + [[ aot_eager_huggingface == *perf* ]] 2024-06-26T07:13:10.9048994Z + [[ aot_eager_huggingface == *cpu_inductor* ]] 2024-06-26T07:13:10.9051608Z + [[ aot_eager_huggingface == *aot_inductor* ]] 2024-06-26T07:13:10.9052355Z + test_single_dynamo_benchmark inference huggingface 0 --inference --bfloat16 2024-06-26T07:13:10.9052933Z ++ pwd 2024-06-26T07:13:10.9053639Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:13:10.9054298Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:13:10.9062229Z + local name=inference 2024-06-26T07:13:10.9062618Z + shift 2024-06-26T07:13:10.9063030Z + local suite=huggingface 2024-06-26T07:13:10.9063389Z + shift 2024-06-26T07:13:10.9063658Z + local shard_id=0 2024-06-26T07:13:10.9063955Z + shift 2024-06-26T07:13:10.9065825Z + partition_flags=() 2024-06-26T07:13:10.9066646Z + local partition_flags 2024-06-26T07:13:10.9067144Z + [[ -n 1 ]] 2024-06-26T07:13:10.9067443Z + [[ -n 0 ]] 2024-06-26T07:13:10.9068016Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-06-26T07:13:10.9069105Z + [[ aot_eager_huggingface == *perf_compare* ]] 2024-06-26T07:13:10.9069575Z + [[ aot_eager_huggingface == *perf* ]] 2024-06-26T07:13:10.9070028Z + [[ aot_eager_huggingface == *aot_inductor* ]] 2024-06-26T07:13:10.9071662Z + python benchmarks/dynamo/huggingface.py --ci --accuracy --timing --explain --backend aot_eager --device cuda --inference --bfloat16 --total-partitions 1 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv 2024-06-26T07:13:13.5655305Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:13:13.5656928Z warnings.warn( 2024-06-26T07:13:13.6061453Z 2024-06-26T07:13:13.6061861Z config.json: 0% 0.00/694 [00:00 will be ignored 2024-06-26T07:29:39.8611917Z pass 2024-06-26T07:29:39.9135190Z TIMING: entire_frame_compile:10.73268 backend_compile:7.15012 2024-06-26T07:29:39.9137298Z STATS: call_* op count: 534 | FakeTensorMode.__torch_dispatch__:35947 | FakeTensor.__torch_dispatch__:1758 | ProxyTorchDispatchMode.__torch_dispatch__:14646 2024-06-26T07:29:39.9138546Z Dynamo produced 2 graphs covering 534 ops with 5 graph breaks (4 unique) 2024-06-26T07:29:43.4388375Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:29:43.4389999Z warnings.warn( 2024-06-26T07:29:43.6673732Z 2024-06-26T07:29:46.3329083Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:29:46.3329661Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:29:46.3330208Z cuda train AllenaiLongformerBase 2024-06-26T07:29:53.7879704Z W0626 07:29:53.787000 140548695777920 torch/_dynamo/variables/tensor.py:714] [3/0] Graph break from `Tensor.item()`, consider setting: 2024-06-26T07:29:53.7881201Z W0626 07:29:53.787000 140548695777920 torch/_dynamo/variables/tensor.py:714] [3/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-26T07:29:53.7882565Z W0626 07:29:53.787000 140548695777920 torch/_dynamo/variables/tensor.py:714] [3/0] or: 2024-06-26T07:29:53.7883844Z W0626 07:29:53.787000 140548695777920 torch/_dynamo/variables/tensor.py:714] [3/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-26T07:29:53.7885245Z W0626 07:29:53.787000 140548695777920 torch/_dynamo/variables/tensor.py:714] [3/0] to include these operations in the captured graph. 2024-06-26T07:29:53.7886372Z W0626 07:29:53.787000 140548695777920 torch/_dynamo/variables/tensor.py:714] [3/0] 2024-06-26T07:30:23.9522931Z W0626 07:30:23.951000 140548695777920 torch/_logging/_internal.py:1040] [16/0] Profiler function will be ignored 2024-06-26T07:30:44.9297321Z pass 2024-06-26T07:30:44.9920213Z TIMING: entire_frame_compile:51.56141 backend_compile:34.324 2024-06-26T07:30:44.9922463Z STATS: call_* op count: 2367 | FakeTensorMode.__torch_dispatch__:116429 | FakeTensor.__torch_dispatch__:4625 | ProxyTorchDispatchMode.__torch_dispatch__:48592 2024-06-26T07:30:44.9923829Z Dynamo produced 7 graphs covering 2367 ops with 9 graph breaks (5 unique) 2024-06-26T07:30:49.5325296Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:30:49.5327614Z warnings.warn( 2024-06-26T07:30:49.7631059Z 2024-06-26T07:30:53.3920346Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:30:53.3920877Z loading model: 0it [00:03, ?it/s] 2024-06-26T07:30:53.3923327Z cuda train BartForCausalLM 2024-06-26T07:31:08.6090705Z pass 2024-06-26T07:31:08.7527126Z TIMING: entire_frame_compile:7.43643 backend_compile:5.72646 2024-06-26T07:31:08.7529263Z STATS: call_* op count: 325 | FakeTensorMode.__torch_dispatch__:26169 | FakeTensor.__torch_dispatch__:847 | ProxyTorchDispatchMode.__torch_dispatch__:11407 2024-06-26T07:31:08.7531469Z Dynamo produced 16 graphs covering 325 ops with 6 graph breaks (5 unique) 2024-06-26T07:31:12.2356697Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:31:12.2358468Z warnings.warn( 2024-06-26T07:31:12.4408248Z 2024-06-26T07:31:18.9383780Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:31:18.9384265Z loading model: 0it [00:06, ?it/s] 2024-06-26T07:31:18.9384733Z cuda train BartForConditionalGeneration 2024-06-26T07:31:54.1165594Z pass 2024-06-26T07:31:54.3637394Z TIMING: entire_frame_compile:18.90397 backend_compile:14.64355 2024-06-26T07:31:54.3639560Z STATS: call_* op count: 878 | FakeTensorMode.__torch_dispatch__:67193 | FakeTensor.__torch_dispatch__:2172 | ProxyTorchDispatchMode.__torch_dispatch__:29836 2024-06-26T07:31:54.3640704Z Dynamo produced 30 graphs covering 878 ops with 8 graph breaks (5 unique) 2024-06-26T07:31:58.1321463Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:31:58.1324634Z warnings.warn( 2024-06-26T07:31:58.3279277Z 2024-06-26T07:32:00.1269425Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:32:00.1269905Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:32:00.1270338Z cuda train BertForMaskedLM 2024-06-26T07:32:14.2561709Z W0626 07:32:14.255000 139763394900608 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:32:33.6738437Z pass 2024-06-26T07:32:33.7942302Z TIMING: entire_frame_compile:30.37638 backend_compile:18.35701 2024-06-26T07:32:33.7945611Z STATS: call_* op count: 996 | FakeTensorMode.__torch_dispatch__:62011 | FakeTensor.__torch_dispatch__:5113 | ProxyTorchDispatchMode.__torch_dispatch__:21485 2024-06-26T07:32:33.7946805Z Dynamo produced 2 graphs covering 996 ops with 5 graph breaks (4 unique) 2024-06-26T07:32:37.7778557Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:32:37.7780194Z warnings.warn( 2024-06-26T07:32:37.9908759Z 2024-06-26T07:32:39.5720861Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:32:39.5721802Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:32:39.5722379Z cuda train BertForQuestionAnswering 2024-06-26T07:32:53.2952107Z W0626 07:32:53.294000 140686599459456 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:33:12.5738234Z pass 2024-06-26T07:33:12.6901653Z TIMING: entire_frame_compile:30.17049 backend_compile:17.97166 2024-06-26T07:33:12.6903966Z STATS: call_* op count: 994 | FakeTensorMode.__torch_dispatch__:61510 | FakeTensor.__torch_dispatch__:5037 | ProxyTorchDispatchMode.__torch_dispatch__:21302 2024-06-26T07:33:12.6905129Z Dynamo produced 2 graphs covering 994 ops with 5 graph breaks (4 unique) 2024-06-26T07:33:16.6904441Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:33:16.6906172Z warnings.warn( 2024-06-26T07:33:16.8920990Z 2024-06-26T07:33:44.0202948Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:33:44.0203425Z loading model: 0it [00:27, ?it/s] 2024-06-26T07:33:44.0206666Z cuda train BlenderbotForCausalLM 2024-06-26T07:33:44.0211271Z Traceback (most recent call last): 2024-06-26T07:33:44.0212038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T07:33:44.0212882Z self.model_iter_fn(model, example_inputs) 2024-06-26T07:33:44.0213723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in forward_and_backward_pass 2024-06-26T07:33:44.0214562Z pred = mod(**cloned_inputs) 2024-06-26T07:33:44.0215617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:33:44.0216496Z return self._call_impl(*args, **kwargs) 2024-06-26T07:33:44.0217419Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:33:44.0218252Z return forward_call(*args, **kwargs) 2024-06-26T07:33:44.0219300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 1531, in forward 2024-06-26T07:33:44.0224870Z outputs = self.model.decoder( 2024-06-26T07:33:44.0226364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:33:44.0227423Z return self._call_impl(*args, **kwargs) 2024-06-26T07:33:44.0228784Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:33:44.0229643Z return forward_call(*args, **kwargs) 2024-06-26T07:33:44.0230946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 997, in forward 2024-06-26T07:33:44.0231923Z layer_outputs = decoder_layer( 2024-06-26T07:33:44.0232858Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:33:44.0233753Z return self._call_impl(*args, **kwargs) 2024-06-26T07:33:44.0234929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:33:44.0235989Z return forward_call(*args, **kwargs) 2024-06-26T07:33:44.0237239Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 397, in forward 2024-06-26T07:33:44.0238327Z hidden_states, self_attn_weights, present_key_value = self.self_attn( 2024-06-26T07:33:44.0239428Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:33:44.0240297Z return self._call_impl(*args, **kwargs) 2024-06-26T07:33:44.0241405Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:33:44.0242226Z return forward_call(*args, **kwargs) 2024-06-26T07:33:44.0243264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 152, in forward 2024-06-26T07:33:44.0244616Z query_states = self.q_proj(hidden_states) * self.scaling 2024-06-26T07:33:44.0245659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:33:44.0260902Z return self._call_impl(*args, **kwargs) 2024-06-26T07:33:44.0261859Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:33:44.0262843Z return forward_call(*args, **kwargs) 2024-06-26T07:33:44.0263749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 125, in forward 2024-06-26T07:33:44.0264596Z return F.linear(input, self.weight, self.bias) 2024-06-26T07:33:44.0267750Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 14.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 6.44 MiB is free. Process 1937 has 21.96 GiB memory in use. Of the allocated memory 21.61 GiB is allocated by PyTorch, and 27.77 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-06-26T07:33:44.0270546Z 2024-06-26T07:33:44.0270841Z The above exception was the direct cause of the following exception: 2024-06-26T07:33:44.0271295Z 2024-06-26T07:33:44.0271436Z Traceback (most recent call last): 2024-06-26T07:33:44.0272076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T07:33:44.0272725Z ) = runner.load_model( 2024-06-26T07:33:44.0273381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 522, in load_model 2024-06-26T07:33:44.0274135Z self.validate_model(model, example_inputs) 2024-06-26T07:33:44.0274877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T07:33:44.0275636Z raise RuntimeError("Eager run failed") from e 2024-06-26T07:33:44.0276094Z RuntimeError: Eager run failed 2024-06-26T07:33:44.0276352Z 2024-06-26T07:33:44.0276467Z eager_fail_to_run 2024-06-26T07:33:47.2831498Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:33:47.2833203Z warnings.warn( 2024-06-26T07:33:47.5239490Z 2024-06-26T07:33:48.7834872Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:33:48.7835341Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:33:48.7835832Z cuda train BlenderbotSmallForCausalLM 2024-06-26T07:33:48.8051493Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForCausalLM. Setting accuracy check to cosine 2024-06-26T07:33:55.4782841Z pass 2024-06-26T07:33:55.5135036Z TIMING: entire_frame_compile:5.87762 backend_compile:4.29564 2024-06-26T07:33:55.5136104Z STATS: call_* op count: 324 | FakeTensorMode.__torch_dispatch__:19676 | FakeTensor.__torch_dispatch__:535 | ProxyTorchDispatchMode.__torch_dispatch__:8650 2024-06-26T07:33:55.5137208Z Dynamo produced 13 graphs covering 324 ops with 6 graph breaks (5 unique) 2024-06-26T07:33:58.9941156Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:33:58.9942771Z warnings.warn( 2024-06-26T07:33:59.2433234Z 2024-06-26T07:34:00.8811446Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:34:00.8811919Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:34:00.8812419Z cuda train BlenderbotSmallForConditionalGeneration 2024-06-26T07:34:01.0219575Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T07:34:16.0383571Z pass 2024-06-26T07:34:16.0752282Z TIMING: entire_frame_compile:13.66537 backend_compile:10.18364 2024-06-26T07:34:16.0753409Z STATS: call_* op count: 830 | FakeTensorMode.__torch_dispatch__:49002 | FakeTensor.__torch_dispatch__:1303 | ProxyTorchDispatchMode.__torch_dispatch__:22166 2024-06-26T07:34:16.0754911Z Dynamo produced 22 graphs covering 830 ops with 8 graph breaks (5 unique) 2024-06-26T07:34:19.7234985Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:34:19.7236615Z warnings.warn( 2024-06-26T07:34:19.9325622Z 2024-06-26T07:34:21.7606253Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:34:21.7607428Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:34:21.7608523Z cuda train CamemBert 2024-06-26T07:34:35.7719859Z W0626 07:34:35.771000 139958019523200 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:34:55.1168709Z pass 2024-06-26T07:34:55.2345242Z TIMING: entire_frame_compile:30.17794 backend_compile:18.21524 2024-06-26T07:34:55.2347684Z STATS: call_* op count: 1004 | FakeTensorMode.__torch_dispatch__:62113 | FakeTensor.__torch_dispatch__:5113 | ProxyTorchDispatchMode.__torch_dispatch__:21521 2024-06-26T07:34:55.2350225Z Dynamo produced 2 graphs covering 1004 ops with 5 graph breaks (4 unique) 2024-06-26T07:34:59.2197562Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:34:59.2199156Z warnings.warn( 2024-06-26T07:34:59.4047808Z 2024-06-26T07:35:01.5611822Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:35:01.5612378Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:35:01.5613694Z cuda train DebertaForMaskedLM 2024-06-26T07:35:16.8552257Z W0626 07:35:16.854000 140172227805824 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:35:32.7930210Z pass 2024-06-26T07:35:32.9022353Z TIMING: entire_frame_compile:27.90476 backend_compile:16.91486 2024-06-26T07:35:32.9024787Z STATS: call_* op count: 1319 | FakeTensorMode.__torch_dispatch__:62189 | FakeTensor.__torch_dispatch__:3744 | ProxyTorchDispatchMode.__torch_dispatch__:23799 2024-06-26T07:35:32.9026172Z Dynamo produced 2 graphs covering 1319 ops with 5 graph breaks (4 unique) 2024-06-26T07:35:36.8749834Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:35:37.1443428Z warnings.warn( 2024-06-26T07:35:37.1443854Z 2024-06-26T07:35:38.9245659Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:35:38.9246131Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:35:38.9248536Z cuda train DebertaForQuestionAnswering 2024-06-26T07:35:53.6961158Z W0626 07:35:53.695000 140356333089408 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:36:09.4377060Z pass 2024-06-26T07:36:09.5612361Z TIMING: entire_frame_compile:27.71413 backend_compile:16.85308 2024-06-26T07:36:09.5614941Z STATS: call_* op count: 1317 | FakeTensorMode.__torch_dispatch__:61688 | FakeTensor.__torch_dispatch__:3673 | ProxyTorchDispatchMode.__torch_dispatch__:23616 2024-06-26T07:36:09.5616094Z Dynamo produced 2 graphs covering 1317 ops with 5 graph breaks (4 unique) 2024-06-26T07:36:13.5066951Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:36:13.5068945Z warnings.warn( 2024-06-26T07:36:13.6968574Z 2024-06-26T07:36:25.6424752Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:36:25.6425514Z loading model: 0it [00:11, ?it/s] 2024-06-26T07:36:25.6425972Z cuda train DebertaV2ForMaskedLM 2024-06-26T07:36:25.6427945Z pass_due_to_skip 2024-06-26T07:36:25.8631606Z TIMING: 2024-06-26T07:36:25.8632272Z STATS: call_* op count: 0 2024-06-26T07:36:25.8633382Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T07:36:29.1434018Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:36:29.1435770Z warnings.warn( 2024-06-26T07:36:29.4034143Z 2024-06-26T07:36:39.3365547Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:36:39.3366030Z loading model: 0it [00:09, ?it/s] 2024-06-26T07:36:39.3366492Z cuda train DebertaV2ForQuestionAnswering 2024-06-26T07:36:49.9691090Z ERROR:common: 2024-06-26T07:36:49.9691657Z Traceback (most recent call last): 2024-06-26T07:36:49.9692494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2543, in check_accuracy 2024-06-26T07:36:49.9693464Z correct_result = self.run_n_iterations( 2024-06-26T07:36:49.9694399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2359, in run_n_iterations 2024-06-26T07:36:49.9695490Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-06-26T07:36:49.9696673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 582, in forward_and_backward_pass 2024-06-26T07:36:49.9697624Z pred = mod(**cloned_inputs) 2024-06-26T07:36:49.9698991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:36:49.9699968Z return self._call_impl(*args, **kwargs) 2024-06-26T07:36:49.9701225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:36:49.9702190Z return forward_call(*args, **kwargs) 2024-06-26T07:36:49.9703402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1486, in forward 2024-06-26T07:36:49.9704599Z outputs = self.deberta( 2024-06-26T07:36:49.9705531Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:36:49.9706407Z return self._call_impl(*args, **kwargs) 2024-06-26T07:36:49.9707330Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:36:49.9708294Z return forward_call(*args, **kwargs) 2024-06-26T07:36:49.9709332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1070, in forward 2024-06-26T07:36:49.9710277Z encoder_outputs = self.encoder( 2024-06-26T07:36:49.9711218Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:36:49.9712101Z return self._call_impl(*args, **kwargs) 2024-06-26T07:36:49.9713015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:36:49.9714136Z return forward_call(*args, **kwargs) 2024-06-26T07:36:49.9715184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 514, in forward 2024-06-26T07:36:49.9716123Z output_states = layer_module( 2024-06-26T07:36:49.9717060Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:36:49.9717936Z return self._call_impl(*args, **kwargs) 2024-06-26T07:36:49.9718862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:36:49.9719810Z return forward_call(*args, **kwargs) 2024-06-26T07:36:49.9720843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 362, in forward 2024-06-26T07:36:49.9721791Z attention_output = self.attention( 2024-06-26T07:36:49.9722758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:36:49.9724013Z return self._call_impl(*args, **kwargs) 2024-06-26T07:36:49.9725259Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:36:49.9726091Z return forward_call(*args, **kwargs) 2024-06-26T07:36:49.9727126Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 293, in forward 2024-06-26T07:36:49.9728040Z self_output = self.self( 2024-06-26T07:36:49.9728950Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:36:49.9729851Z return self._call_impl(*args, **kwargs) 2024-06-26T07:36:49.9730771Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:36:49.9731586Z return forward_call(*args, **kwargs) 2024-06-26T07:36:49.9732623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 733, in forward 2024-06-26T07:36:49.9733767Z attention_probs = XSoftmax.apply(attention_scores, attention_mask, -1) 2024-06-26T07:36:49.9734799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/function.py", line 574, in apply 2024-06-26T07:36:49.9735678Z return super().apply(*args, **kwargs) # type: ignore[misc] 2024-06-26T07:36:49.9736807Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 113, in forward 2024-06-26T07:36:49.9737784Z output = torch.softmax(output, self.dim) 2024-06-26T07:36:49.9740939Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 24.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 10.44 MiB is free. Process 2638 has 21.96 GiB memory in use. Of the allocated memory 21.29 GiB is allocated by PyTorch, and 318.25 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-06-26T07:36:50.0008803Z eager_1st_run_OOM 2024-06-26T07:36:50.0009232Z TIMING: 2024-06-26T07:36:50.0009611Z STATS: call_* op count: 0 2024-06-26T07:36:50.0010145Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T07:36:53.3060905Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:36:53.3062534Z warnings.warn( 2024-06-26T07:36:53.7314618Z 2024-06-26T07:36:55.0397174Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:36:55.0397994Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:36:55.0398458Z cuda train DistilBertForMaskedLM 2024-06-26T07:37:00.5542737Z W0626 07:37:00.553000 140451637723776 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:37:10.6608235Z pass 2024-06-26T07:37:10.7240787Z TIMING: entire_frame_compile:14.49171 backend_compile:9.21449 2024-06-26T07:37:10.7242036Z STATS: call_* op count: 541 | FakeTensor.__torch_dispatch__:2683 | FakeTensorMode.__torch_dispatch__:32380 | ProxyTorchDispatchMode.__torch_dispatch__:11147 2024-06-26T07:37:10.7243149Z Dynamo produced 2 graphs covering 541 ops with 5 graph breaks (4 unique) 2024-06-26T07:37:14.3567430Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:37:14.3569130Z warnings.warn( 2024-06-26T07:37:14.5691342Z 2024-06-26T07:37:15.6270395Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:37:15.6270980Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:37:15.6271540Z cuda train DistilBertForQuestionAnswering 2024-06-26T07:37:20.9847092Z W0626 07:37:20.983000 140322796520064 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:37:30.8173300Z pass 2024-06-26T07:37:30.8805524Z TIMING: entire_frame_compile:14.15136 backend_compile:9.00853 2024-06-26T07:37:30.8806714Z STATS: call_* op count: 540 | FakeTensor.__torch_dispatch__:2612 | FakeTensorMode.__torch_dispatch__:31927 | ProxyTorchDispatchMode.__torch_dispatch__:10978 2024-06-26T07:37:30.8807852Z Dynamo produced 2 graphs covering 540 ops with 5 graph breaks (4 unique) 2024-06-26T07:37:34.5143192Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:37:34.5144801Z warnings.warn( 2024-06-26T07:37:34.6873221Z 2024-06-26T07:37:36.4457232Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:37:36.4457697Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:37:36.4458140Z cuda train DistillGPT2 2024-06-26T07:37:43.4316221Z W0626 07:37:43.430000 140647572673152 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:37:50.8286530Z pass 2024-06-26T07:37:50.8861555Z TIMING: entire_frame_compile:12.30333 backend_compile:7.39847 2024-06-26T07:37:50.8862964Z STATS: call_* op count: 572 | FakeTensorMode.__torch_dispatch__:26704 | FakeTensor.__torch_dispatch__:1681 | ProxyTorchDispatchMode.__torch_dispatch__:9304 2024-06-26T07:37:50.8865335Z Dynamo produced 2 graphs covering 572 ops with 5 graph breaks (4 unique) 2024-06-26T07:37:54.4608476Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:37:54.4610185Z warnings.warn( 2024-06-26T07:37:54.6760396Z 2024-06-26T07:37:54.6768959Z loading model: 0it [00:00, ?it/s]If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2024-06-26T07:37:55.2748573Z 2024-06-26T07:37:55.2749182Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:37:55.2749695Z cuda train ElectraForCausalLM 2024-06-26T07:38:07.9278911Z pass 2024-06-26T07:38:07.9492677Z TIMING: entire_frame_compile:11.42593 backend_compile:6.75187 2024-06-26T07:38:07.9493775Z STATS: call_* op count: 377 | FakeTensorMode.__torch_dispatch__:32441 | FakeTensor.__torch_dispatch__:1252 | ProxyTorchDispatchMode.__torch_dispatch__:12488 2024-06-26T07:38:07.9495154Z Dynamo produced 1 graphs covering 377 ops with 4 graph breaks (4 unique) 2024-06-26T07:38:11.5372859Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:38:11.5374471Z warnings.warn( 2024-06-26T07:38:11.7960472Z 2024-06-26T07:38:12.3531516Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:38:12.3532005Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:38:12.3532462Z cuda train ElectraForQuestionAnswering 2024-06-26T07:38:24.9758185Z W0626 07:38:24.975000 140061096125056 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:38:44.4687279Z pass 2024-06-26T07:38:44.4868040Z TIMING: entire_frame_compile:30.66489 backend_compile:18.44261 2024-06-26T07:38:44.4870326Z STATS: call_* op count: 1001 | FakeTensorMode.__torch_dispatch__:62040 | FakeTensor.__torch_dispatch__:5102 | ProxyTorchDispatchMode.__torch_dispatch__:21467 2024-06-26T07:38:44.4872545Z Dynamo produced 2 graphs covering 1001 ops with 5 graph breaks (4 unique) 2024-06-26T07:38:48.5273251Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:38:48.5276226Z warnings.warn( 2024-06-26T07:38:48.7172353Z 2024-06-26T07:38:50.7990580Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:38:50.7991730Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:38:50.7992655Z cuda train GPT2ForSequenceClassification 2024-06-26T07:39:06.5331489Z W0626 07:39:06.532000 140240039993984 torch/_logging/_internal.py:1040] [7/0] Profiler function will be ignored 2024-06-26T07:39:20.7952441Z pass 2024-06-26T07:39:20.9095459Z TIMING: entire_frame_compile:25.43195 backend_compile:14.469 2024-06-26T07:39:20.9096532Z STATS: call_* op count: 1102 | FakeTensorMode.__torch_dispatch__:55081 | FakeTensor.__torch_dispatch__:3413 | ProxyTorchDispatchMode.__torch_dispatch__:18351 2024-06-26T07:39:20.9097646Z Dynamo produced 5 graphs covering 1102 ops with 7 graph breaks (5 unique) 2024-06-26T07:39:24.7950681Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:39:24.7952314Z warnings.warn( 2024-06-26T07:39:25.0643164Z 2024-06-26T07:39:26.3572246Z loading model: 0it [00:00, ?it/s]WARNING:common:Model GoogleFnet supports float32 only 2024-06-26T07:39:26.5125978Z 2024-06-26T07:39:26.5126429Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:39:26.5127033Z WARNING:common:Model GoogleFnet supports float32 only 2024-06-26T07:39:26.5137979Z cuda train GoogleFnet 2024-06-26T07:39:27.8420235Z WARNING:common:Model GoogleFnet supports float32 only 2024-06-26T07:39:33.0897901Z W0626 07:39:33.088000 139862632878720 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:39:43.4512556Z pass 2024-06-26T07:39:43.5133040Z TIMING: entire_frame_compile:14.86592 backend_compile:8.7345 2024-06-26T07:39:43.5134123Z STATS: call_* op count: 574 | FakeTensorMode.__torch_dispatch__:29842 | FakeTensor.__torch_dispatch__:2775 | ProxyTorchDispatchMode.__torch_dispatch__:10146 2024-06-26T07:39:43.5135243Z Dynamo produced 2 graphs covering 574 ops with 5 graph breaks (4 unique) 2024-06-26T07:39:47.1546851Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:39:47.1550319Z warnings.warn( 2024-06-26T07:39:47.3760208Z 2024-06-26T07:39:49.2255513Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:39:49.2256917Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:39:49.2257779Z cuda train LayoutLMForMaskedLM 2024-06-26T07:40:03.6399440Z W0626 07:40:03.638000 139677584433792 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:40:23.5760978Z pass 2024-06-26T07:40:23.7031453Z TIMING: entire_frame_compile:31.14267 backend_compile:18.76341 2024-06-26T07:40:23.7033617Z STATS: call_* op count: 1034 | FakeTensorMode.__torch_dispatch__:63598 | FakeTensor.__torch_dispatch__:5242 | ProxyTorchDispatchMode.__torch_dispatch__:21936 2024-06-26T07:40:23.7034831Z Dynamo produced 2 graphs covering 1034 ops with 5 graph breaks (4 unique) 2024-06-26T07:40:27.7324843Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:40:27.7326537Z warnings.warn( 2024-06-26T07:40:28.1849366Z 2024-06-26T07:40:29.7978270Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:40:29.7978750Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:40:29.7979218Z cuda train LayoutLMForSequenceClassification 2024-06-26T07:40:46.0517195Z W0626 07:40:46.050000 140064555819648 torch/_logging/_internal.py:1040] [7/0] Profiler function will be ignored 2024-06-26T07:41:05.9609276Z pass 2024-06-26T07:41:06.0816516Z TIMING: entire_frame_compile:33.27564 backend_compile:18.49971 2024-06-26T07:41:06.0820620Z STATS: call_* op count: 1031 | FakeTensorMode.__torch_dispatch__:68019 | FakeTensor.__torch_dispatch__:5850 | ProxyTorchDispatchMode.__torch_dispatch__:22181 2024-06-26T07:41:06.0821768Z Dynamo produced 5 graphs covering 1031 ops with 7 graph breaks (5 unique) 2024-06-26T07:41:10.1274737Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:41:10.1277230Z warnings.warn( 2024-06-26T07:41:10.3671627Z 2024-06-26T07:41:20.2602097Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:41:20.2602709Z loading model: 0it [00:09, ?it/s] 2024-06-26T07:41:20.2605691Z cuda train M2M100ForConditionalGeneration 2024-06-26T07:41:20.3381613Z WARNING:common:fp64 golden ref were not generated for M2M100ForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T07:41:43.4961593Z pass 2024-06-26T07:41:43.7107611Z TIMING: entire_frame_compile:20.5504 backend_compile:15.03841 2024-06-26T07:41:43.7109600Z STATS: call_* op count: 1296 | FakeTensorMode.__torch_dispatch__:81067 | FakeTensor.__torch_dispatch__:3135 | ProxyTorchDispatchMode.__torch_dispatch__:32498 2024-06-26T07:41:43.7110713Z Dynamo produced 1 graphs covering 1296 ops with 4 graph breaks (4 unique) 2024-06-26T07:41:47.4278693Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:41:47.4280360Z warnings.warn( 2024-06-26T07:41:47.6075825Z 2024-06-26T07:41:51.2076357Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:41:51.2076827Z loading model: 0it [00:03, ?it/s] 2024-06-26T07:41:51.2080445Z cuda train MBartForCausalLM 2024-06-26T07:41:51.2382282Z WARNING:common:fp64 golden ref were not generated for MBartForCausalLM. Setting accuracy check to cosine 2024-06-26T07:42:00.8504681Z pass 2024-06-26T07:42:00.9597303Z TIMING: entire_frame_compile:8.2899 backend_compile:6.15834 2024-06-26T07:42:00.9599423Z STATS: call_* op count: 478 | FakeTensorMode.__torch_dispatch__:28954 | FakeTensor.__torch_dispatch__:763 | ProxyTorchDispatchMode.__torch_dispatch__:12867 2024-06-26T07:42:00.9601631Z Dynamo produced 17 graphs covering 478 ops with 6 graph breaks (5 unique) 2024-06-26T07:42:04.4775706Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:42:04.4777887Z warnings.warn( 2024-06-26T07:42:04.7157444Z 2024-06-26T07:42:11.2872845Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:42:11.2873874Z loading model: 0it [00:06, ?it/s] 2024-06-26T07:42:11.2874923Z cuda train MBartForConditionalGeneration 2024-06-26T07:42:11.3874245Z WARNING:common:fp64 golden ref were not generated for MBartForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T07:42:34.7632163Z pass 2024-06-26T07:42:34.9859135Z TIMING: entire_frame_compile:19.92622 backend_compile:15.28771 2024-06-26T07:42:34.9861398Z STATS: call_* op count: 1251 | FakeTensorMode.__torch_dispatch__:73174 | FakeTensor.__torch_dispatch__:1873 | ProxyTorchDispatchMode.__torch_dispatch__:33239 2024-06-26T07:42:34.9862572Z Dynamo produced 31 graphs covering 1251 ops with 8 graph breaks (5 unique) 2024-06-26T07:42:38.7679624Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:42:38.7681276Z warnings.warn( 2024-06-26T07:42:38.9554786Z 2024-06-26T07:42:42.7589988Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:42:42.7590490Z loading model: 0it [00:03, ?it/s] 2024-06-26T07:42:42.7590961Z cuda train MT5ForConditionalGeneration 2024-06-26T07:43:05.7227058Z W0626 07:43:05.721000 139858187027072 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:43:26.0565400Z pass 2024-06-26T07:43:26.3686863Z TIMING: entire_frame_compile:40.0741 backend_compile:24.88555 2024-06-26T07:43:26.3687940Z STATS: call_* op count: 1764 | FakeTensorMode.__torch_dispatch__:92032 | FakeTensor.__torch_dispatch__:4589 | ProxyTorchDispatchMode.__torch_dispatch__:36442 2024-06-26T07:43:26.3691472Z Dynamo produced 2 graphs covering 1764 ops with 5 graph breaks (4 unique) 2024-06-26T07:43:30.5889916Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:43:30.5891560Z warnings.warn( 2024-06-26T07:43:30.8036425Z 2024-06-26T07:43:30.8081490Z loading model: 0it [00:00, ?it/s]If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2024-06-26T07:43:35.2225326Z 2024-06-26T07:43:35.2226143Z loading model: 0it [00:04, ?it/s] 2024-06-26T07:43:35.2226668Z cuda train MegatronBertForCausalLM 2024-06-26T07:44:04.9826645Z W0626 07:44:04.981000 140282343092864 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:44:43.5233458Z pass 2024-06-26T07:44:43.8717104Z TIMING: entire_frame_compile:60.30837 backend_compile:36.33864 2024-06-26T07:44:43.8720728Z STATS: call_* op count: 1923 | FakeTensorMode.__torch_dispatch__:120693 | FakeTensor.__torch_dispatch__:9913 | ProxyTorchDispatchMode.__torch_dispatch__:42045 2024-06-26T07:44:43.8722776Z Dynamo produced 2 graphs covering 1923 ops with 5 graph breaks (4 unique) 2024-06-26T07:44:48.6488270Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:44:48.6489892Z warnings.warn( 2024-06-26T07:44:48.8434497Z 2024-06-26T07:44:52.9596806Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:44:52.9597333Z loading model: 0it [00:04, ?it/s] 2024-06-26T07:44:52.9597798Z cuda train MegatronBertForQuestionAnswering 2024-06-26T07:45:22.1113498Z W0626 07:45:22.110000 140114431890048 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:46:00.1936930Z pass 2024-06-26T07:46:00.5329407Z TIMING: entire_frame_compile:59.69006 backend_compile:35.75371 2024-06-26T07:46:00.5331969Z STATS: call_* op count: 1917 | FakeTensorMode.__torch_dispatch__:120076 | FakeTensor.__torch_dispatch__:9837 | ProxyTorchDispatchMode.__torch_dispatch__:41827 2024-06-26T07:46:00.5333119Z Dynamo produced 2 graphs covering 1917 ops with 5 graph breaks (4 unique) 2024-06-26T07:46:05.2511500Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:46:05.2513184Z warnings.warn( 2024-06-26T07:46:05.4431339Z 2024-06-26T07:46:06.7494717Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:46:06.7495187Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:46:06.7495635Z cuda train MobileBertForMaskedLM 2024-06-26T07:47:04.6271575Z pass 2024-06-26T07:47:04.6601706Z TIMING: entire_frame_compile:54.09745 backend_compile:30.3937 2024-06-26T07:47:04.6604524Z STATS: call_* op count: 1449 | FakeTensorMode.__torch_dispatch__:130119 | FakeTensor.__torch_dispatch__:4395 | ProxyTorchDispatchMode.__torch_dispatch__:55808 2024-06-26T07:47:04.6605960Z Dynamo produced 1 graphs covering 1449 ops with 3 graph breaks (3 unique) 2024-06-26T07:47:09.2952856Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:47:09.2954511Z warnings.warn( 2024-06-26T07:47:09.5165839Z 2024-06-26T07:47:10.6757007Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:47:10.6757875Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:47:10.6758414Z cuda train MobileBertForQuestionAnswering 2024-06-26T07:48:07.7263794Z pass 2024-06-26T07:48:07.7484456Z TIMING: entire_frame_compile:53.32545 backend_compile:29.63838 2024-06-26T07:48:07.7487202Z STATS: call_* op count: 1453 | FakeTensorMode.__torch_dispatch__:129979 | FakeTensor.__torch_dispatch__:4387 | ProxyTorchDispatchMode.__torch_dispatch__:55709 2024-06-26T07:48:07.7488361Z Dynamo produced 1 graphs covering 1453 ops with 3 graph breaks (3 unique) 2024-06-26T07:48:12.3518753Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:48:12.3520518Z warnings.warn( 2024-06-26T07:48:12.5742216Z 2024-06-26T07:48:14.8651562Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:48:14.8652444Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:48:14.8653304Z cuda train OPTForCausalLM 2024-06-26T07:48:14.8854672Z WARNING:common:fp64 golden ref were not generated for OPTForCausalLM. Setting accuracy check to cosine 2024-06-26T07:48:25.2814377Z pass 2024-06-26T07:48:25.3333528Z TIMING: entire_frame_compile:8.29239 backend_compile:6.48076 2024-06-26T07:48:25.3334908Z STATS: call_* op count: 533 | FakeTensorMode.__torch_dispatch__:29523 | FakeTensor.__torch_dispatch__:721 | ProxyTorchDispatchMode.__torch_dispatch__:13208 2024-06-26T07:48:25.3336025Z Dynamo produced 17 graphs covering 533 ops with 6 graph breaks (5 unique) 2024-06-26T07:48:28.8334255Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:48:28.8335877Z warnings.warn( 2024-06-26T07:48:29.1169600Z 2024-06-26T07:48:30.8805766Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:48:30.8806634Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:48:30.8807081Z cuda train PLBartForCausalLM 2024-06-26T07:48:30.8976423Z WARNING:common:fp64 golden ref were not generated for PLBartForCausalLM. Setting accuracy check to cosine 2024-06-26T07:48:36.2793185Z pass 2024-06-26T07:48:36.3220569Z TIMING: entire_frame_compile:4.61001 backend_compile:3.26854 2024-06-26T07:48:36.3221686Z STATS: call_* op count: 250 | FakeTensorMode.__torch_dispatch__:15285 | FakeTensor.__torch_dispatch__:421 | ProxyTorchDispatchMode.__torch_dispatch__:6609 2024-06-26T07:48:36.3222789Z Dynamo produced 11 graphs covering 250 ops with 6 graph breaks (5 unique) 2024-06-26T07:48:39.7760881Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:48:39.7762653Z warnings.warn( 2024-06-26T07:48:40.1112903Z 2024-06-26T07:48:43.2109511Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:48:43.2110411Z loading model: 0it [00:03, ?it/s] 2024-06-26T07:48:43.2111066Z cuda train PLBartForConditionalGeneration 2024-06-26T07:48:43.2641550Z WARNING:common:fp64 golden ref were not generated for PLBartForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T07:48:55.7618288Z pass 2024-06-26T07:48:55.8320520Z TIMING: entire_frame_compile:10.87474 backend_compile:8.3077 2024-06-26T07:48:55.8321723Z STATS: call_* op count: 646 | FakeTensorMode.__torch_dispatch__:38771 | FakeTensor.__torch_dispatch__:1045 | ProxyTorchDispatchMode.__torch_dispatch__:17249 2024-06-26T07:48:55.8322843Z Dynamo produced 19 graphs covering 646 ops with 8 graph breaks (5 unique) 2024-06-26T07:48:59.4680759Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:48:59.4682434Z warnings.warn( 2024-06-26T07:48:59.6838370Z 2024-06-26T07:49:05.7215036Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:49:05.7215516Z loading model: 0it [00:06, ?it/s] 2024-06-26T07:49:05.7215977Z cuda train PegasusForCausalLM 2024-06-26T07:49:05.8447031Z WARNING:common:fp64 golden ref were not generated for PegasusForCausalLM. Setting accuracy check to cosine 2024-06-26T07:49:15.2599674Z pass 2024-06-26T07:49:15.3419560Z TIMING: entire_frame_compile:8.38037 backend_compile:6.39856 2024-06-26T07:49:15.3421731Z STATS: call_* op count: 476 | FakeTensorMode.__torch_dispatch__:28829 | FakeTensor.__torch_dispatch__:744 | ProxyTorchDispatchMode.__torch_dispatch__:12848 2024-06-26T07:49:15.3423989Z Dynamo produced 17 graphs covering 476 ops with 6 graph breaks (5 unique) 2024-06-26T07:49:18.8665570Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:49:18.8667437Z warnings.warn( 2024-06-26T07:49:19.0519781Z 2024-06-26T07:49:29.3670817Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:49:29.3671653Z loading model: 0it [00:10, ?it/s] 2024-06-26T07:49:29.3672123Z cuda train PegasusForConditionalGeneration 2024-06-26T07:49:29.4459773Z WARNING:common:fp64 golden ref were not generated for PegasusForConditionalGeneration. Setting accuracy check to cosine 2024-06-26T07:49:51.5975106Z pass 2024-06-26T07:49:51.7080115Z TIMING: entire_frame_compile:19.91252 backend_compile:15.4851 2024-06-26T07:49:51.7082142Z STATS: call_* op count: 1234 | FakeTensorMode.__torch_dispatch__:72609 | FakeTensor.__torch_dispatch__:1833 | ProxyTorchDispatchMode.__torch_dispatch__:33098 2024-06-26T07:49:51.7083765Z Dynamo produced 30 graphs covering 1234 ops with 7 graph breaks (4 unique) 2024-06-26T07:49:55.5050848Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:49:55.5052466Z warnings.warn( 2024-06-26T07:49:55.6996424Z 2024-06-26T07:49:55.7031792Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2024-06-26T07:49:57.5916779Z We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2024-06-26T07:49:57.5918672Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2024-06-26T07:49:57.8039198Z 2024-06-26T07:49:57.8039604Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:49:57.8042501Z cuda train RobertaForCausalLM 2024-06-26T07:50:12.0661659Z W0626 07:50:12.065000 139793772642944 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:50:31.5528930Z pass 2024-06-26T07:50:31.6664096Z TIMING: entire_frame_compile:30.32731 backend_compile:18.26311 2024-06-26T07:50:31.6665212Z STATS: call_* op count: 1008 | FakeTensorMode.__torch_dispatch__:62215 | FakeTensor.__torch_dispatch__:5113 | ProxyTorchDispatchMode.__torch_dispatch__:21552 2024-06-26T07:50:31.6669446Z Dynamo produced 2 graphs covering 1008 ops with 5 graph breaks (4 unique) 2024-06-26T07:50:35.6899120Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:50:35.6900846Z warnings.warn( 2024-06-26T07:50:35.9212043Z 2024-06-26T07:50:37.4267937Z loading model: 0it [00:00, ?it/s]We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2024-06-26T07:50:37.4270338Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2024-06-26T07:50:37.6558780Z 2024-06-26T07:50:37.6559383Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:50:37.6559893Z cuda train RobertaForQuestionAnswering 2024-06-26T07:50:51.3568386Z W0626 07:50:51.356000 140689257276032 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:51:10.5961937Z pass 2024-06-26T07:51:10.7217559Z TIMING: entire_frame_compile:30.07111 backend_compile:18.18318 2024-06-26T07:51:10.7220127Z STATS: call_* op count: 1001 | FakeTensorMode.__torch_dispatch__:61594 | FakeTensor.__torch_dispatch__:5037 | ProxyTorchDispatchMode.__torch_dispatch__:21334 2024-06-26T07:51:10.7221300Z Dynamo produced 2 graphs covering 1001 ops with 5 graph breaks (4 unique) 2024-06-26T07:51:14.7218082Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:51:14.7220751Z warnings.warn( 2024-06-26T07:51:14.9401545Z 2024-06-26T07:51:15.4467434Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:51:15.4468255Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:51:15.4468829Z cuda train Speech2Text2ForCausalLM 2024-06-26T07:51:15.4623351Z WARNING:common:fp64 golden ref were not generated for Speech2Text2ForCausalLM. Setting accuracy check to cosine 2024-06-26T07:51:20.6963126Z pass 2024-06-26T07:51:20.7017924Z TIMING: entire_frame_compile:4.62591 backend_compile:3.45345 2024-06-26T07:51:20.7018982Z STATS: call_* op count: 258 | FakeTensorMode.__torch_dispatch__:15309 | FakeTensor.__torch_dispatch__:404 | ProxyTorchDispatchMode.__torch_dispatch__:6652 2024-06-26T07:51:20.7020107Z Dynamo produced 11 graphs covering 258 ops with 6 graph breaks (5 unique) 2024-06-26T07:51:24.1754371Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:51:24.1755985Z warnings.warn( 2024-06-26T07:51:24.4358895Z 2024-06-26T07:51:25.6929608Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:51:25.6930341Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:51:25.6931028Z cuda train T5ForConditionalGeneration 2024-06-26T07:51:43.7845026Z W0626 07:51:43.783000 140320286544512 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:51:57.7786633Z pass 2024-06-26T07:51:57.8425819Z TIMING: entire_frame_compile:28.02264 backend_compile:17.32305 2024-06-26T07:51:57.8427948Z STATS: call_* op count: 1224 | FakeTensorMode.__torch_dispatch__:63325 | FakeTensor.__torch_dispatch__:3167 | ProxyTorchDispatchMode.__torch_dispatch__:24715 2024-06-26T07:51:57.8429137Z Dynamo produced 2 graphs covering 1224 ops with 5 graph breaks (4 unique) 2024-06-26T07:52:01.7930606Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:52:01.7932267Z warnings.warn( 2024-06-26T07:52:01.9850062Z 2024-06-26T07:52:03.2510915Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:52:03.2511550Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:52:03.2512086Z cuda train T5Small 2024-06-26T07:52:21.3312975Z W0626 07:52:21.330000 140523165393536 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:52:35.3122167Z pass 2024-06-26T07:52:35.3713986Z TIMING: entire_frame_compile:27.98252 backend_compile:17.27008 2024-06-26T07:52:35.3717569Z STATS: call_* op count: 1224 | FakeTensorMode.__torch_dispatch__:63325 | FakeTensor.__torch_dispatch__:3167 | ProxyTorchDispatchMode.__torch_dispatch__:24715 2024-06-26T07:52:35.3718710Z Dynamo produced 2 graphs covering 1224 ops with 5 graph breaks (4 unique) 2024-06-26T07:52:39.2680070Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:52:39.2681725Z warnings.warn( 2024-06-26T07:52:39.4520782Z 2024-06-26T07:52:43.0203246Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:52:43.0204084Z loading model: 0it [00:03, ?it/s] 2024-06-26T07:52:43.0204533Z cuda train TrOCRForCausalLM 2024-06-26T07:52:43.0519024Z WARNING:common:fp64 golden ref were not generated for TrOCRForCausalLM. Setting accuracy check to cosine 2024-06-26T07:52:52.5207075Z pass 2024-06-26T07:52:52.6415420Z TIMING: entire_frame_compile:8.3384 backend_compile:6.21487 2024-06-26T07:52:52.6417000Z STATS: call_* op count: 477 | FakeTensorMode.__torch_dispatch__:28948 | FakeTensor.__torch_dispatch__:763 | ProxyTorchDispatchMode.__torch_dispatch__:12867 2024-06-26T07:52:52.6418705Z Dynamo produced 17 graphs covering 477 ops with 6 graph breaks (5 unique) 2024-06-26T07:52:56.1292497Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:52:56.1294465Z warnings.warn( 2024-06-26T07:52:56.4848989Z 2024-06-26T07:53:05.6446562Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:53:05.6447039Z loading model: 0it [00:09, ?it/s] 2024-06-26T07:53:05.6450522Z cuda train XGLMForCausalLM 2024-06-26T07:53:05.7892613Z WARNING:common:fp64 golden ref were not generated for XGLMForCausalLM. Setting accuracy check to cosine 2024-06-26T07:53:24.3233491Z pass 2024-06-26T07:53:24.4857994Z TIMING: entire_frame_compile:16.47381 backend_compile:13.03906 2024-06-26T07:53:24.4859114Z STATS: call_* op count: 987 | FakeTensorMode.__torch_dispatch__:58729 | FakeTensor.__torch_dispatch__:1433 | ProxyTorchDispatchMode.__torch_dispatch__:26536 2024-06-26T07:53:24.4860215Z Dynamo produced 29 graphs covering 987 ops with 6 graph breaks (5 unique) 2024-06-26T07:53:28.1995340Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:53:28.1998529Z warnings.warn( 2024-06-26T07:53:28.3845780Z 2024-06-26T07:53:32.5019966Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:53:32.5020590Z loading model: 0it [00:04, ?it/s] 2024-06-26T07:53:32.5021768Z cuda train XLNetLMHeadModel 2024-06-26T07:54:15.2435454Z W0626 07:54:15.242000 140017642615424 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:54:56.0317136Z pass 2024-06-26T07:54:56.4008498Z TIMING: entire_frame_compile:73.71532 backend_compile:46.52639 2024-06-26T07:54:56.4009598Z STATS: call_* op count: 1874 | FakeTensorMode.__torch_dispatch__:152227 | FakeTensor.__torch_dispatch__:8098 | ProxyTorchDispatchMode.__torch_dispatch__:55611 2024-06-26T07:54:56.4010740Z Dynamo produced 2 graphs covering 1874 ops with 5 graph breaks (4 unique) 2024-06-26T07:55:01.4007937Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-06-26T07:55:01.4009720Z warnings.warn( 2024-06-26T07:55:01.7430606Z 2024-06-26T07:55:03.8057747Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:55:03.8058215Z loading model: 0it [00:02, ?it/s] 2024-06-26T07:55:03.8058655Z cuda train YituTechConvBert 2024-06-26T07:55:23.6222672Z W0626 07:55:23.621000 139814052065920 torch/_logging/_internal.py:1040] [4/0] Profiler function will be ignored 2024-06-26T07:55:52.5510009Z pass 2024-06-26T07:55:52.6479371Z TIMING: entire_frame_compile:45.33598 backend_compile:27.1406 2024-06-26T07:55:52.6481836Z STATS: call_* op count: 1512 | FakeTensorMode.__torch_dispatch__:89993 | FakeTensor.__torch_dispatch__:7175 | ProxyTorchDispatchMode.__torch_dispatch__:31640 2024-06-26T07:55:52.6482969Z Dynamo produced 2 graphs covering 1512 ops with 5 graph breaks (4 unique) 2024-06-26T07:55:54.4573504Z accuracy pass_rate=93.48% 2024-06-26T07:55:54.4582957Z calls_captured gmean=0.00x mean=920.348x 2024-06-26T07:55:54.4586006Z unique_graphs gmean=0.00x mean=7.478x 2024-06-26T07:55:54.4589457Z graph_breaks gmean=0.00x mean=5.196x 2024-06-26T07:55:54.4592978Z unique_graph_breaks gmean=0.00x mean=4.043x 2024-06-26T07:55:54.4596311Z autograd_captures gmean=0.00x mean=0.000x 2024-06-26T07:55:54.4599618Z autograd_compiles gmean=0.00x mean=0.000x 2024-06-26T07:55:54.4602760Z cudagraph_skips gmean=0.00x mean=0.000x 2024-06-26T07:55:55.0172810Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy//aot_eager_huggingface_training.csv 2024-06-26T07:55:55.2731914Z AlbertForMaskedLM PASS 2024-06-26T07:55:55.2734684Z AlbertForQuestionAnswering PASS 2024-06-26T07:55:55.2740290Z AllenaiLongformerBase PASS 2024-06-26T07:55:55.2745758Z BartForCausalLM PASS 2024-06-26T07:55:55.2751398Z BartForConditionalGeneration PASS 2024-06-26T07:55:55.2756835Z BertForMaskedLM PASS 2024-06-26T07:55:55.2762277Z BertForQuestionAnswering PASS 2024-06-26T07:55:55.2768021Z BlenderbotForCausalLM XFAIL 2024-06-26T07:55:55.2773459Z BlenderbotSmallForCausalLM PASS 2024-06-26T07:55:55.2778727Z BlenderbotSmallForConditionalGeneration PASS 2024-06-26T07:55:55.2783795Z CamemBert PASS 2024-06-26T07:55:55.2788978Z DebertaForMaskedLM PASS 2024-06-26T07:55:55.2794183Z DebertaForQuestionAnswering PASS 2024-06-26T07:55:55.2799055Z DebertaV2ForMaskedLM XFAIL 2024-06-26T07:55:55.2803984Z DebertaV2ForQuestionAnswering XFAIL 2024-06-26T07:55:55.2809117Z DistilBertForMaskedLM PASS 2024-06-26T07:55:55.2814056Z DistilBertForQuestionAnswering PASS 2024-06-26T07:55:55.2819263Z DistillGPT2 PASS 2024-06-26T07:55:55.2824148Z ElectraForCausalLM PASS 2024-06-26T07:55:55.2829276Z ElectraForQuestionAnswering PASS 2024-06-26T07:55:55.2834419Z GPT2ForSequenceClassification PASS 2024-06-26T07:55:55.2839198Z GoogleFnet PASS 2024-06-26T07:55:55.2844438Z LayoutLMForMaskedLM PASS 2024-06-26T07:55:55.2849275Z LayoutLMForSequenceClassification PASS 2024-06-26T07:55:55.2854126Z M2M100ForConditionalGeneration PASS 2024-06-26T07:55:55.2859051Z MBartForCausalLM PASS 2024-06-26T07:55:55.2864048Z MBartForConditionalGeneration PASS 2024-06-26T07:55:55.2869110Z MT5ForConditionalGeneration PASS 2024-06-26T07:55:55.2874115Z MegatronBertForCausalLM PASS 2024-06-26T07:55:55.2879031Z MegatronBertForQuestionAnswering PASS 2024-06-26T07:55:55.2884281Z MobileBertForMaskedLM PASS 2024-06-26T07:55:55.2889129Z MobileBertForQuestionAnswering PASS 2024-06-26T07:55:55.2894004Z OPTForCausalLM PASS 2024-06-26T07:55:55.2898920Z PLBartForCausalLM PASS 2024-06-26T07:55:55.2903891Z PLBartForConditionalGeneration PASS 2024-06-26T07:55:55.2908861Z PegasusForCausalLM PASS 2024-06-26T07:55:55.2913951Z PegasusForConditionalGeneration PASS 2024-06-26T07:55:55.2918674Z RobertaForCausalLM PASS 2024-06-26T07:55:55.2923427Z RobertaForQuestionAnswering PASS 2024-06-26T07:55:55.2928452Z Speech2Text2ForCausalLM PASS 2024-06-26T07:55:55.2933177Z T5ForConditionalGeneration PASS 2024-06-26T07:55:55.2937920Z T5Small PASS 2024-06-26T07:55:55.2942666Z TrOCRForCausalLM PASS 2024-06-26T07:55:55.2947397Z XGLMForCausalLM PASS 2024-06-26T07:55:55.2952371Z XLNetLMHeadModel PASS 2024-06-26T07:55:55.2957077Z YituTechConvBert PASS 2024-06-26T07:55:55.3346121Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy//aot_eager_huggingface_training.csv 2024-06-26T07:55:55.5895740Z AlbertForMaskedLM PASS 2024-06-26T07:55:55.5900188Z AlbertForQuestionAnswering PASS 2024-06-26T07:55:55.5904803Z AllenaiLongformerBase PASS 2024-06-26T07:55:55.5909311Z BartForCausalLM PASS 2024-06-26T07:55:55.5913960Z BartForConditionalGeneration PASS 2024-06-26T07:55:55.5918330Z BertForMaskedLM PASS 2024-06-26T07:55:55.5922730Z BertForQuestionAnswering PASS 2024-06-26T07:55:55.5927377Z BlenderbotForCausalLM PASS 2024-06-26T07:55:55.5931732Z BlenderbotSmallForCausalLM PASS 2024-06-26T07:55:55.5936136Z BlenderbotSmallForConditionalGeneration PASS 2024-06-26T07:55:55.5940462Z CamemBert PASS 2024-06-26T07:55:55.5944858Z DebertaForMaskedLM PASS 2024-06-26T07:55:55.5949224Z DebertaForQuestionAnswering PASS 2024-06-26T07:55:55.5953708Z DebertaV2ForMaskedLM PASS 2024-06-26T07:55:55.5958085Z DebertaV2ForQuestionAnswering PASS 2024-06-26T07:55:55.5962409Z DistilBertForMaskedLM PASS 2024-06-26T07:55:55.5967052Z DistilBertForQuestionAnswering PASS 2024-06-26T07:55:55.5971467Z DistillGPT2 PASS 2024-06-26T07:55:55.5976032Z ElectraForCausalLM PASS 2024-06-26T07:55:55.5980168Z ElectraForQuestionAnswering PASS 2024-06-26T07:55:55.5984503Z GPT2ForSequenceClassification PASS 2024-06-26T07:55:55.5988991Z GoogleFnet PASS 2024-06-26T07:55:55.5993523Z LayoutLMForMaskedLM PASS 2024-06-26T07:55:55.5997971Z LayoutLMForSequenceClassification PASS 2024-06-26T07:55:55.6002236Z M2M100ForConditionalGeneration PASS 2024-06-26T07:55:55.6006792Z MBartForCausalLM PASS 2024-06-26T07:55:55.6011212Z MBartForConditionalGeneration PASS 2024-06-26T07:55:55.6015624Z MT5ForConditionalGeneration PASS 2024-06-26T07:55:55.6019956Z MegatronBertForCausalLM PASS 2024-06-26T07:55:55.6024351Z MegatronBertForQuestionAnswering PASS 2024-06-26T07:55:55.6028904Z MobileBertForMaskedLM PASS 2024-06-26T07:55:55.6033337Z MobileBertForQuestionAnswering PASS 2024-06-26T07:55:55.6037648Z OPTForCausalLM PASS 2024-06-26T07:55:55.6042014Z PLBartForCausalLM PASS 2024-06-26T07:55:55.6046596Z PLBartForConditionalGeneration PASS 2024-06-26T07:55:55.6051079Z PegasusForCausalLM PASS 2024-06-26T07:55:55.6055396Z PegasusForConditionalGeneration PASS 2024-06-26T07:55:55.6059773Z RobertaForCausalLM PASS 2024-06-26T07:55:55.6064203Z RobertaForQuestionAnswering PASS 2024-06-26T07:55:55.6068686Z Speech2Text2ForCausalLM PASS 2024-06-26T07:55:55.6073098Z T5ForConditionalGeneration PASS 2024-06-26T07:55:55.6077444Z T5Small PASS 2024-06-26T07:55:55.6081807Z TrOCRForCausalLM PASS 2024-06-26T07:55:55.6086338Z XGLMForCausalLM PASS 2024-06-26T07:55:55.6090723Z XLNetLMHeadModel PASS 2024-06-26T07:55:55.6095118Z YituTechConvBert PASS 2024-06-26T07:55:55.6472570Z + cleanup_workspace 2024-06-26T07:55:55.6473627Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-06-26T07:55:55.6476595Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-06-26T07:55:55.6477886Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-06-26T07:55:55.6478781Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-06-26T07:55:55.6479595Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-06-26T07:55:55.6480461Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-06-26T07:55:55.6481263Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-06-26T07:55:56.1125771Z ##[group]Run cat test/**/*_toprint.log || true 2024-06-26T07:55:56.1126295Z cat test/**/*_toprint.log || true 2024-06-26T07:55:56.1134311Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:55:56.1134806Z env: 2024-06-26T07:55:56.1135076Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:56.1135540Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:55:56.1136279Z DOCKER_CONTAINER_ID: b752ec5bddbf78c4c938e2d957d66195460cd43fb409c9c551e6c40eb7d6a5db 2024-06-26T07:55:56.1136917Z ##[endgroup] 2024-06-26T07:55:56.1201871Z cat: test/**/*_toprint.log: No such file or directory 2024-06-26T07:55:56.1229025Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-06-26T07:55:56.1229500Z kill "$MONITOR_SCRIPT_PID" 2024-06-26T07:55:56.1236599Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:55:56.1237090Z env: 2024-06-26T07:55:56.1237373Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:56.1237827Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:55:56.1238563Z DOCKER_CONTAINER_ID: b752ec5bddbf78c4c938e2d957d66195460cd43fb409c9c551e6c40eb7d6a5db 2024-06-26T07:55:56.1239211Z MONITOR_SCRIPT_PID: 26737 2024-06-26T07:55:56.1239567Z ##[endgroup] 2024-06-26T07:55:56.1425110Z Prepare all required actions 2024-06-26T07:55:56.1425551Z Getting action download info 2024-06-26T07:55:56.5402563Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-06-26T07:55:56.6977931Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-06-26T07:55:56.6978397Z with: 2024-06-26T07:55:56.6978952Z file-suffix: test-aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688916472 2024-06-26T07:55:56.6979618Z s3-bucket: gha-artifacts 2024-06-26T07:55:56.6979957Z env: 2024-06-26T07:55:56.6980225Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:56.6980678Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:55:56.6981426Z DOCKER_CONTAINER_ID: b752ec5bddbf78c4c938e2d957d66195460cd43fb409c9c551e6c40eb7d6a5db 2024-06-26T07:55:56.6982052Z ##[endgroup] 2024-06-26T07:55:56.7006606Z ##[group]Run # Remove any previous test jsons if they exist 2024-06-26T07:55:56.7007256Z # Remove any previous test jsons if they exist 2024-06-26T07:55:56.7007761Z rm -f test-jsons-*.zip 2024-06-26T07:55:56.7008272Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-06-26T07:55:56.7015988Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:55:56.7016506Z env: 2024-06-26T07:55:56.7016814Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:56.7017265Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:55:56.7017994Z DOCKER_CONTAINER_ID: b752ec5bddbf78c4c938e2d957d66195460cd43fb409c9c551e6c40eb7d6a5db 2024-06-26T07:55:56.7018894Z FILE_SUFFIX: test-aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688916472 2024-06-26T07:55:56.7019537Z ##[endgroup] 2024-06-26T07:55:56.7192685Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-06-26T07:55:56.7221815Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-06-26T07:55:56.7222549Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-06-26T07:55:56.7228509Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-06-26T07:55:56.7253692Z ##[group]Run # Remove any previous test reports if they exist 2024-06-26T07:55:56.7254340Z # Remove any previous test reports if they exist 2024-06-26T07:55:56.7254868Z rm -f test-reports-*.zip 2024-06-26T07:55:56.7255452Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-06-26T07:55:56.7263164Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:55:56.7263655Z env: 2024-06-26T07:55:56.7263940Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:56.7264438Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:55:56.7265228Z DOCKER_CONTAINER_ID: b752ec5bddbf78c4c938e2d957d66195460cd43fb409c9c551e6c40eb7d6a5db 2024-06-26T07:55:56.7266125Z FILE_SUFFIX: test-aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688916472 2024-06-26T07:55:56.7266768Z ##[endgroup] 2024-06-26T07:55:56.7423835Z adding: test/test-reports/inference_huggingface.csv (deflated 76%) 2024-06-26T07:55:56.7424718Z adding: test/test-reports/inference_huggingface_graph_breaks.csv (deflated 89%) 2024-06-26T07:55:56.7425648Z adding: test/test-reports/inference_huggingface_graph_break_deduped.csv (deflated 78%) 2024-06-26T07:55:56.7426513Z adding: test/test-reports/training_huggingface.csv (deflated 73%) 2024-06-26T07:55:56.7433419Z adding: test/test-reports/training_huggingface_graph_breaks.csv (deflated 97%) 2024-06-26T07:55:56.7434871Z adding: test/test-reports/training_huggingface_graph_break_deduped.csv (deflated 77%) 2024-06-26T07:55:56.7466065Z ##[group]Run # Remove any previous usage logs if they exist 2024-06-26T07:55:56.7466689Z # Remove any previous usage logs if they exist 2024-06-26T07:55:56.7467188Z rm -f logs-*.zip 2024-06-26T07:55:56.7467844Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-06-26T07:55:56.7468856Z # so check to see if the file exists first 2024-06-26T07:55:56.7469358Z if [ -f 'usage_log.txt' ]; then 2024-06-26T07:55:56.7469884Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-06-26T07:55:56.7470366Z fi 2024-06-26T07:55:56.7470726Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-06-26T07:55:56.7471299Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-06-26T07:55:56.7471804Z fi 2024-06-26T07:55:56.7479025Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:55:56.7479555Z env: 2024-06-26T07:55:56.7479833Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:56.7480311Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:55:56.7481083Z DOCKER_CONTAINER_ID: b752ec5bddbf78c4c938e2d957d66195460cd43fb409c9c551e6c40eb7d6a5db 2024-06-26T07:55:56.7481984Z FILE_SUFFIX: test-aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688916472 2024-06-26T07:55:56.7482622Z ##[endgroup] 2024-06-26T07:55:56.7532847Z adding: usage_log.txt (deflated 92%) 2024-06-26T07:55:56.7600664Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-06-26T07:55:56.7601367Z # Remove any previous debugging artifacts if they exist 2024-06-26T07:55:56.7601908Z rm -f debug-*.zip 2024-06-26T07:55:56.7602294Z if [ -d 'test/debug' ]; then 2024-06-26T07:55:56.7602787Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-06-26T07:55:56.7603271Z fi 2024-06-26T07:55:56.7610300Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:55:56.7610793Z env: 2024-06-26T07:55:56.7611075Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:56.7611516Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:55:56.7612255Z DOCKER_CONTAINER_ID: b752ec5bddbf78c4c938e2d957d66195460cd43fb409c9c551e6c40eb7d6a5db 2024-06-26T07:55:56.7613155Z FILE_SUFFIX: test-aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688916472 2024-06-26T07:55:56.7613801Z ##[endgroup] 2024-06-26T07:55:56.7705700Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T07:55:56.7706151Z with: 2024-06-26T07:55:56.7706434Z s3-bucket: gha-artifacts 2024-06-26T07:55:56.7706869Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T07:55:56.7707342Z retention-days: 14 2024-06-26T07:55:56.7707681Z if-no-files-found: warn 2024-06-26T07:55:56.7708113Z path: test-jsons-*.zip 2024-06-26T07:55:56.7708466Z name: artifact 2024-06-26T07:55:56.7708781Z region: us-east-1 2024-06-26T07:55:56.7709078Z env: 2024-06-26T07:55:56.7709349Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:56.7709800Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:55:56.7710534Z DOCKER_CONTAINER_ID: b752ec5bddbf78c4c938e2d957d66195460cd43fb409c9c551e6c40eb7d6a5db 2024-06-26T07:55:56.7711160Z ##[endgroup] 2024-06-26T07:55:57.0724513Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T07:55:57.0725342Z With the provided path, there will be 1 file uploaded 2024-06-26T07:55:57.0725998Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T07:55:57.0751687Z Starting upload of test-jsons-test-aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688916472.zip 2024-06-26T07:55:57.6676521Z Finished upload of test-jsons-test-aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688916472.zip 2024-06-26T07:55:57.6801517Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T07:55:57.6802077Z with: 2024-06-26T07:55:57.6802364Z s3-bucket: gha-artifacts 2024-06-26T07:55:57.6802789Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T07:55:57.6803261Z retention-days: 14 2024-06-26T07:55:57.6803755Z if-no-files-found: error 2024-06-26T07:55:57.6804128Z path: test-reports-*.zip 2024-06-26T07:55:57.6804484Z name: artifact 2024-06-26T07:55:57.6804781Z region: us-east-1 2024-06-26T07:55:57.6805081Z env: 2024-06-26T07:55:57.6805358Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:57.6805813Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:55:57.6806702Z DOCKER_CONTAINER_ID: b752ec5bddbf78c4c938e2d957d66195460cd43fb409c9c551e6c40eb7d6a5db 2024-06-26T07:55:57.6807343Z ##[endgroup] 2024-06-26T07:55:57.9822927Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T07:55:57.9823817Z With the provided path, there will be 1 file uploaded 2024-06-26T07:55:57.9824647Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T07:55:57.9851672Z Starting upload of test-reports-test-aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688916472.zip 2024-06-26T07:55:58.0838088Z Finished upload of test-reports-test-aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688916472.zip 2024-06-26T07:55:58.0967885Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T07:55:58.0968318Z with: 2024-06-26T07:55:58.0968599Z s3-bucket: gha-artifacts 2024-06-26T07:55:58.0969024Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T07:55:58.0969489Z retention-days: 14 2024-06-26T07:55:58.0969814Z if-no-files-found: ignore 2024-06-26T07:55:58.0970186Z path: logs-*.zip 2024-06-26T07:55:58.0970495Z name: artifact 2024-06-26T07:55:58.0970798Z region: us-east-1 2024-06-26T07:55:58.0971102Z env: 2024-06-26T07:55:58.0971375Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:58.0971833Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:55:58.0972554Z DOCKER_CONTAINER_ID: b752ec5bddbf78c4c938e2d957d66195460cd43fb409c9c551e6c40eb7d6a5db 2024-06-26T07:55:58.0973202Z ##[endgroup] 2024-06-26T07:55:58.4052080Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T07:55:58.4052952Z With the provided path, there will be 1 file uploaded 2024-06-26T07:55:58.4053771Z Uploading to s3 prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T07:55:58.4080731Z Starting upload of logs-test-aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688916472.zip 2024-06-26T07:55:58.5144762Z Finished upload of logs-test-aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_26688916472.zip 2024-06-26T07:55:58.5270903Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T07:55:58.5271341Z with: 2024-06-26T07:55:58.5271629Z s3-bucket: gha-artifacts 2024-06-26T07:55:58.5272055Z s3-prefix: pytorch/pytorch/9673646806/1/artifact 2024-06-26T07:55:58.5272529Z retention-days: 14 2024-06-26T07:55:58.5272867Z if-no-files-found: ignore 2024-06-26T07:55:58.5273235Z path: debug-*.zip 2024-06-26T07:55:58.5273578Z name: artifact 2024-06-26T07:55:58.5273879Z region: us-east-1 2024-06-26T07:55:58.5274186Z env: 2024-06-26T07:55:58.5274469Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:58.5274923Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:55:58.5275649Z DOCKER_CONTAINER_ID: b752ec5bddbf78c4c938e2d957d66195460cd43fb409c9c551e6c40eb7d6a5db 2024-06-26T07:55:58.5276284Z ##[endgroup] 2024-06-26T07:55:58.8247373Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-06-26T07:55:58.8381185Z ##[group]Run # shellcheck disable=SC2156 2024-06-26T07:55:58.8381662Z # shellcheck disable=SC2156 2024-06-26T07:55:58.8382451Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-06-26T07:55:58.8390572Z shell: /usr/bin/bash -e {0} 2024-06-26T07:55:58.8390942Z env: 2024-06-26T07:55:58.8391232Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:58.8391710Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:55:58.8392641Z DOCKER_CONTAINER_ID: b752ec5bddbf78c4c938e2d957d66195460cd43fb409c9c551e6c40eb7d6a5db 2024-06-26T07:55:58.8393358Z ##[endgroup] 2024-06-26T07:55:59.0034327Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-06-26T07:55:59.0034881Z with: 2024-06-26T07:55:59.0035142Z env: 2024-06-26T07:55:59.0035428Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:59.0035875Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:55:59.0036656Z DOCKER_CONTAINER_ID: b752ec5bddbf78c4c938e2d957d66195460cd43fb409c9c551e6c40eb7d6a5db 2024-06-26T07:55:59.0037296Z ##[endgroup] 2024-06-26T07:55:59.0055326Z ##[group]Run set -eou pipefail 2024-06-26T07:55:59.0055886Z set -eou pipefail 2024-06-26T07:55:59.0056238Z  2024-06-26T07:55:59.0056748Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-06-26T07:55:59.0057372Z for _ in $(seq 1440); do 2024-06-26T07:55:59.0057844Z  # Break if no ssh session exists anymore 2024-06-26T07:55:59.0058353Z  if [ "$(who)" = "" ]; then 2024-06-26T07:55:59.0058761Z  break 2024-06-26T07:55:59.0059103Z  fi 2024-06-26T07:55:59.0059394Z  echo "." 2024-06-26T07:55:59.0059722Z  sleep 5 2024-06-26T07:55:59.0060042Z done 2024-06-26T07:55:59.0067510Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:55:59.0068063Z env: 2024-06-26T07:55:59.0068336Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:59.0068788Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:55:59.0069513Z DOCKER_CONTAINER_ID: b752ec5bddbf78c4c938e2d957d66195460cd43fb409c9c551e6c40eb7d6a5db 2024-06-26T07:55:59.0070147Z ##[endgroup] 2024-06-26T07:55:59.0089090Z Holding runner for 2 hours until all ssh sessions have logged out 2024-06-26T07:55:59.0126337Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:55:59.0127092Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:55:59.0127688Z # shellcheck disable=SC2046 2024-06-26T07:55:59.0128162Z docker stop $(docker ps -q) || true 2024-06-26T07:55:59.0128634Z # Prune all of the docker images 2024-06-26T07:55:59.0129085Z docker system prune -af 2024-06-26T07:55:59.0136044Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:55:59.0136542Z env: 2024-06-26T07:55:59.0136816Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:59.0137269Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:55:59.0138004Z DOCKER_CONTAINER_ID: b752ec5bddbf78c4c938e2d957d66195460cd43fb409c9c551e6c40eb7d6a5db 2024-06-26T07:55:59.0138650Z ##[endgroup] 2024-06-26T07:55:59.2891063Z b752ec5bddbf 2024-06-26T07:55:59.6738143Z Deleted Containers: 2024-06-26T07:55:59.6738650Z b752ec5bddbf78c4c938e2d957d66195460cd43fb409c9c551e6c40eb7d6a5db 2024-06-26T07:55:59.6739084Z 2024-06-26T07:56:04.6613557Z Deleted Images: 2024-06-26T07:56:04.6615103Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:56:04.6617301Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T07:56:04.6619003Z deleted: sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd 2024-06-26T07:56:04.6619833Z deleted: sha256:754ead7012f0b7691885a9660f02e80be6cee8c5665486fb995c1fb06adb6e92 2024-06-26T07:56:04.6620660Z deleted: sha256:fbe4496d8830e11be86d8f222522e7956f32f85bfc06b05ea53961f34a58af74 2024-06-26T07:56:04.6621521Z deleted: sha256:1c83814aa1a1a7e0cc998e3479bdab8e8766b975560cbf00690acf39ed899a80 2024-06-26T07:56:04.6622343Z deleted: sha256:e9ea8e61942c7f8ae4aab97b8d102b01d6bb4b6cf2a605808cd34130a17d8a0f 2024-06-26T07:56:04.6623169Z deleted: sha256:acc5fc32588cd95e082a6ce35e48d0f303ae8f96e3ab65c4e375b85e1f3c4a92 2024-06-26T07:56:04.6624167Z deleted: sha256:a93e53400904d29e4185d3be942ea34f19572ea438962788ae4e9b3d9bf4c498 2024-06-26T07:56:04.6625139Z deleted: sha256:c97092b88b5e4c77c68bbc4b51afab64e38719af1ff58013ec49a696cdd40a6e 2024-06-26T07:56:04.6625959Z deleted: sha256:b132889c90d67ffa53655846603e3fcc2372c23f04dbdbc54690a39d304eb89b 2024-06-26T07:56:04.6626768Z deleted: sha256:915d54081ed607710624abe3f2490114c745858831f4fc13396bd3b4b9a121bc 2024-06-26T07:56:04.6627566Z deleted: sha256:bfa8d446c498c87f535aef102695564de1b22c755e6c63944ae2f5073a2b31ec 2024-06-26T07:56:04.6628572Z deleted: sha256:e71460fb363c29e70bbe357d680aaa7ae694f094ef2f2713609158ad90611744 2024-06-26T07:56:04.6629384Z deleted: sha256:84079bb1b4ee1d0119ae3d3e701de460a1324912297c675425c9a36d30fcdb0f 2024-06-26T07:56:04.6630195Z deleted: sha256:d86ac3ca148d4758cb03b493fdd49109d8f1a2c7c85c4b755493c3c4529553e5 2024-06-26T07:56:04.6631083Z deleted: sha256:532b3ab97c44a673a0d5f62ee46fa6a83248a84de8877754a612355be56ed0a3 2024-06-26T07:56:04.6632036Z deleted: sha256:617e2a35f25ae24500c88c0c9b8475f0687fbcbed3d200013b4c99c6d8941aff 2024-06-26T07:56:04.6632992Z deleted: sha256:b88120bcd5a26a660c09c5df13176e0ef098fb2c8cdb57d74ab0796a3d1194d6 2024-06-26T07:56:04.6633797Z deleted: sha256:c016b2e7764735243205fb76fa9a273bb3dd4bf05532288e3a613ce022131ac3 2024-06-26T07:56:04.6634615Z deleted: sha256:bcaca462c390deddd2995de2d245a7495e7d2e1e94f0edad30a573c83d0588fd 2024-06-26T07:56:04.6635422Z deleted: sha256:9e260eba003d5c25d7b410e5c4c3093845d9e46611896840901c689d7fbf8ee6 2024-06-26T07:56:04.6636216Z deleted: sha256:3e62b86f8b911a9b59c62d55f063e71683ae6a01c22418db5d27930712adaad6 2024-06-26T07:56:04.6637055Z deleted: sha256:905920ccb1d88449b8dde686c34a4b2a8b6f59e78ac44916b189ea3593b9fbbb 2024-06-26T07:56:04.6637860Z deleted: sha256:961201237e03acd5061406238dd7a3c1343ac888d7583212cd0ab5ae1b4f0540 2024-06-26T07:56:04.6638667Z deleted: sha256:70047a2aac62159fa28fb4dbd4567ec2ffbab68fe046967e09acb0eb3e5cb9cb 2024-06-26T07:56:04.6639502Z deleted: sha256:1ec680ad781436ae65fa987f5fa1ed032ecd5c82a6e35eea6707de501f62c0f4 2024-06-26T07:56:04.6640329Z deleted: sha256:ef859623a50e85abf6e6c1dbd6188aa765a66cc7b2bd618a1cbe4825b7b2bd91 2024-06-26T07:56:04.6641154Z deleted: sha256:404ade5cacff59773b93ab438bfa64cd3a171d07f6f168bdc4a5d4bfa0b43526 2024-06-26T07:56:04.6641966Z deleted: sha256:5b27de6ee3d9f2f18217bf18dec725f7b4954c0e48c85d35752634c22d62209b 2024-06-26T07:56:04.6642782Z deleted: sha256:498e39c575b3fa069acaac7e46b91b4e54ae3a05f2ada5c4347091f346f25d4c 2024-06-26T07:56:04.6643860Z deleted: sha256:1405919d2694d65669b1f4c842282021afe89208ce86570e96464903733da2d0 2024-06-26T07:56:04.6644695Z deleted: sha256:d5b41e5d70896c0472fc3e6da0516a17d89cd6635dc75010ecddc31ce9690fad 2024-06-26T07:56:04.6645506Z deleted: sha256:18e9379b62b9d0bb40293d28a652eafaee66a499f66f0776bf14e37a513101b7 2024-06-26T07:56:04.6646317Z deleted: sha256:a781f80ed00ff2926a699fd01a309bb0b4fb7d5655eb652c205bb9c7f811035f 2024-06-26T07:56:04.6647119Z deleted: sha256:c0430973b7cf759075330ee29c79bfd1312c91e6033d9dd883b4b1c29743a171 2024-06-26T07:56:04.6647947Z deleted: sha256:e733b3d66efb9342e5a283376c0e1ddffa47a42fecc26e61ecaf0a3472885d97 2024-06-26T07:56:04.6648767Z deleted: sha256:8929c0b2047f74b84c82c39eff9077a92ae1a828cde00c9ab78fb3e0ed5e0283 2024-06-26T07:56:04.6649563Z deleted: sha256:f5b56155fb58a0a60fe6391d3581cf91610ed69d4f0a321257c19944da296a52 2024-06-26T07:56:04.6650375Z deleted: sha256:b54d2cb355238d1aa91e58df93064447b80f53706d1ca62823acc834f5ac54fe 2024-06-26T07:56:04.6651192Z deleted: sha256:e7311285af9eeddb51aa5629aa98386e89f1456ff32843b6aee6443247edf66c 2024-06-26T07:56:04.6652010Z deleted: sha256:67657ceec347da8dc8085d5731633efeac115ba677fa4b7c5966c4260f7abab0 2024-06-26T07:56:04.6652817Z deleted: sha256:b496e50409dbf414930fc9b4437aa0aa4ce27bc7840e050ae459aaffcf9aca86 2024-06-26T07:56:04.6653625Z deleted: sha256:8f5276170cc4a60d36ad7b964a502306b29546254cedcbcb101ea9123497d2c0 2024-06-26T07:56:04.6654419Z deleted: sha256:53d524d1564745e96a7e515462171b73184d9f1ff5454b32fcfbbca39d361467 2024-06-26T07:56:04.6655352Z deleted: sha256:2acbadb00d69b6aca575f04baba4ea26ae2b5c2d85f09945004ad93a78a8eb75 2024-06-26T07:56:04.6656259Z deleted: sha256:02665efeff878aeb603c1edb2e21cb700c171c025d64f5aae1b2b20b12f3964a 2024-06-26T07:56:04.6657089Z deleted: sha256:19bd23e9a51dfe883a8b8d2ff6d66ff1b3476b8912856406bb79b81762aa6d4b 2024-06-26T07:56:04.6657911Z deleted: sha256:73aa8637bf9299d6298dabd858480c274a62117388f2ca1d5e9eb15c6d1e4155 2024-06-26T07:56:04.6658715Z deleted: sha256:5f8968406305f348b1caebc0e035187d6e9965ace7b9d5f2f21114d37c9fe2e1 2024-06-26T07:56:04.6659513Z deleted: sha256:d827104914db0c503cf5c81f13e2456bf4efce42d4cf1f8d5d70838207278bee 2024-06-26T07:56:04.6660316Z deleted: sha256:867b219724f2f1aaf56f42e3e9f03a5d64bdca8a6a72407efd10328aeea5d510 2024-06-26T07:56:04.6661130Z deleted: sha256:b9ff2a7cceb9237b26bcb17728edf8a558eec5a76f63626e84f80541dfc0f1d4 2024-06-26T07:56:04.6661949Z deleted: sha256:be73f509106819007ce2b0f5a1301ac56ae583cb6c1fb69bee93efa0840368f1 2024-06-26T07:56:04.6662772Z deleted: sha256:2a11c07c915c3fdbb4e73096dea73ea950c970d50fe1a5764060bd62b7b7cf2c 2024-06-26T07:56:04.6663576Z deleted: sha256:5e9989622e2abc1c72f8184343507b45d75e4ff971edb68b96cbcd75ae484ff5 2024-06-26T07:56:04.6664389Z deleted: sha256:05ce642bd9cf5a9205bff107df28b1d6cef7d5f6ab3ea944d28c032ec61270df 2024-06-26T07:56:04.6665194Z deleted: sha256:017702617f6c07b73f2088678f4710b794cfe897a213a5cf122a0e2519eae0ac 2024-06-26T07:56:04.6665996Z deleted: sha256:44b3b37bb1ca39dc5cf356de41a23a5c12f605a1992813e408087fb2b0ad6d1c 2024-06-26T07:56:04.6666797Z deleted: sha256:8c387982e9f27db357a1d149dc149cabc0bd04668ad6956387a5072682dada67 2024-06-26T07:56:04.6667597Z deleted: sha256:10d415dca4e3fbb88a747b40317f3c6333c041ba8f73855f67fb9fe981fae5ee 2024-06-26T07:56:04.6668533Z deleted: sha256:6c65469bf0f53dcd625faefe363b8f3136c08388a6cf64d9158e8576e4ee89b1 2024-06-26T07:56:04.6669352Z deleted: sha256:027db5d8b76deeefbd8c34b485ab7ff14ed2a3e6800425c1ce0d77c2e9fdd4ab 2024-06-26T07:56:04.6670162Z deleted: sha256:24941234ad28cf5c8174d7cf18326c10b4e17f2c2137672ea0322df461597dd7 2024-06-26T07:56:04.6670974Z deleted: sha256:240ea2eebfa8519d6c2d88f4c99e66ce4e038d000f93ce8d03726c01d81180ef 2024-06-26T07:56:04.6671797Z deleted: sha256:09e823cb391a69f6a195e3ca8a41624b5f8b34eee14c6eb36277de814a91966e 2024-06-26T07:56:04.6672589Z deleted: sha256:730479d695169a6593bf0ac16b377491dce65175a6d158e012422367d04577fd 2024-06-26T07:56:04.6673374Z deleted: sha256:b0a30f9f7bd5459ff662453b9346a7ba8ea8db1191528345c1953eab4aa2f54e 2024-06-26T07:56:04.6674182Z deleted: sha256:75cefa87effaa3ad361c74642839923101f0f9982f814f3c9e2ed4ad52f2a008 2024-06-26T07:56:04.6674987Z deleted: sha256:6f97db87ba9512e615e871c6982000602982e1e5a5be334a56e7292251e84e00 2024-06-26T07:56:04.6675795Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-06-26T07:56:04.6676603Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-06-26T07:56:04.6677400Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-06-26T07:56:04.6678214Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-06-26T07:56:04.6679055Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-06-26T07:56:04.6679870Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-06-26T07:56:04.6680690Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-06-26T07:56:04.6681517Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-06-26T07:56:04.6682342Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-06-26T07:56:04.6683168Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-06-26T07:56:04.6684148Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-06-26T07:56:04.6684649Z 2024-06-26T07:56:04.6684797Z Total reclaimed space: 31.35GB 2024-06-26T07:56:04.6747966Z Post job cleanup. 2024-06-26T07:56:04.6795667Z Post job cleanup. 2024-06-26T07:56:04.7600977Z [command]/usr/bin/git version 2024-06-26T07:56:04.7644143Z git version 2.40.1 2024-06-26T07:56:04.7675584Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/58af685b-c691-45ff-96d3-2ec50d473d58/.gitconfig' 2024-06-26T07:56:04.7687003Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/58af685b-c691-45ff-96d3-2ec50d473d58' before making global git config changes 2024-06-26T07:56:04.7688268Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T07:56:04.7691589Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:56:04.7718269Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T07:56:04.7740757Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T07:56:04.7949497Z Entering 'android/libs/fbjni' 2024-06-26T07:56:04.7983961Z Entering 'third_party/FP16' 2024-06-26T07:56:04.8021063Z Entering 'third_party/FXdiv' 2024-06-26T07:56:04.8055876Z Entering 'third_party/NNPACK' 2024-06-26T07:56:04.8090307Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:56:04.8122375Z Entering 'third_party/XNNPACK' 2024-06-26T07:56:04.8169531Z Entering 'third_party/benchmark' 2024-06-26T07:56:04.8204995Z Entering 'third_party/cpp-httplib' 2024-06-26T07:56:04.8239542Z Entering 'third_party/cpuinfo' 2024-06-26T07:56:04.8272882Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:56:04.8310151Z Entering 'third_party/cutlass' 2024-06-26T07:56:04.8352019Z Entering 'third_party/eigen' 2024-06-26T07:56:04.8389651Z Entering 'third_party/fbgemm' 2024-06-26T07:56:04.8424316Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:56:04.8460919Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:56:04.8496131Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:56:04.8537290Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:56:04.8572495Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:56:04.8607767Z Entering 'third_party/flatbuffers' 2024-06-26T07:56:04.8644962Z Entering 'third_party/fmt' 2024-06-26T07:56:04.8680887Z Entering 'third_party/foxi' 2024-06-26T07:56:04.8716276Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:56:04.8750985Z Entering 'third_party/gloo' 2024-06-26T07:56:04.8784671Z Entering 'third_party/googletest' 2024-06-26T07:56:04.8819379Z Entering 'third_party/ideep' 2024-06-26T07:56:04.8851178Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:56:04.8890365Z Entering 'third_party/ittapi' 2024-06-26T07:56:04.8924546Z Entering 'third_party/kineto' 2024-06-26T07:56:04.8959250Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:56:04.8993893Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:56:04.9026043Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:56:04.9060870Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:56:04.9095260Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:56:04.9129199Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:56:04.9166378Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:56:04.9201326Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:56:04.9235096Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:56:04.9271193Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:56:04.9305588Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:56:04.9338245Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:56:04.9373469Z Entering 'third_party/mimalloc' 2024-06-26T07:56:04.9407212Z Entering 'third_party/nccl/nccl' 2024-06-26T07:56:04.9440334Z Entering 'third_party/nlohmann' 2024-06-26T07:56:04.9475800Z Entering 'third_party/onnx' 2024-06-26T07:56:04.9524145Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:56:04.9559063Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:56:04.9598018Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:56:04.9636839Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:56:04.9671177Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:56:04.9704272Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:56:04.9739682Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:56:04.9772764Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:56:04.9807428Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:56:04.9839380Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:56:04.9872098Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:56:04.9906338Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:56:04.9942887Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:56:04.9992383Z Entering 'third_party/pocketfft' 2024-06-26T07:56:05.0028974Z Entering 'third_party/protobuf' 2024-06-26T07:56:05.0064587Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:56:05.0099601Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:56:05.0137500Z Entering 'third_party/psimd' 2024-06-26T07:56:05.0175409Z Entering 'third_party/pthreadpool' 2024-06-26T07:56:05.0210786Z Entering 'third_party/pybind11' 2024-06-26T07:56:05.0244521Z Entering 'third_party/python-peachpy' 2024-06-26T07:56:05.0278750Z Entering 'third_party/sleef' 2024-06-26T07:56:05.0313409Z Entering 'third_party/tensorpipe' 2024-06-26T07:56:05.0351648Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:56:05.0387740Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:56:05.0423077Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:56:05.0456672Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:56:05.0490827Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:56:05.0540847Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T07:56:05.0563916Z http.https://github.com/.extraheader 2024-06-26T07:56:05.0572719Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-06-26T07:56:05.0596186Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T07:56:05.0807390Z Entering 'android/libs/fbjni' 2024-06-26T07:56:05.0829285Z http.https://github.com/.extraheader 2024-06-26T07:56:05.0853971Z Entering 'third_party/FP16' 2024-06-26T07:56:05.0876271Z http.https://github.com/.extraheader 2024-06-26T07:56:05.0900045Z Entering 'third_party/FXdiv' 2024-06-26T07:56:05.0921171Z http.https://github.com/.extraheader 2024-06-26T07:56:05.0944832Z Entering 'third_party/NNPACK' 2024-06-26T07:56:05.0966933Z http.https://github.com/.extraheader 2024-06-26T07:56:05.0991084Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:56:05.1012152Z http.https://github.com/.extraheader 2024-06-26T07:56:05.1035406Z Entering 'third_party/XNNPACK' 2024-06-26T07:56:05.1055755Z http.https://github.com/.extraheader 2024-06-26T07:56:05.1088254Z Entering 'third_party/benchmark' 2024-06-26T07:56:05.1110731Z http.https://github.com/.extraheader 2024-06-26T07:56:05.1134354Z Entering 'third_party/cpp-httplib' 2024-06-26T07:56:05.1156053Z http.https://github.com/.extraheader 2024-06-26T07:56:05.1180123Z Entering 'third_party/cpuinfo' 2024-06-26T07:56:05.1201103Z http.https://github.com/.extraheader 2024-06-26T07:56:05.1224024Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:56:05.1246357Z http.https://github.com/.extraheader 2024-06-26T07:56:05.1270183Z Entering 'third_party/cutlass' 2024-06-26T07:56:05.1292059Z http.https://github.com/.extraheader 2024-06-26T07:56:05.1319257Z Entering 'third_party/eigen' 2024-06-26T07:56:05.1342324Z http.https://github.com/.extraheader 2024-06-26T07:56:05.1368157Z Entering 'third_party/fbgemm' 2024-06-26T07:56:05.1390971Z http.https://github.com/.extraheader 2024-06-26T07:56:05.1413093Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:56:05.1434155Z http.https://github.com/.extraheader 2024-06-26T07:56:05.1456412Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:56:05.1477886Z http.https://github.com/.extraheader 2024-06-26T07:56:05.1501908Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:56:05.1523561Z http.https://github.com/.extraheader 2024-06-26T07:56:05.1552309Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:56:05.1574466Z http.https://github.com/.extraheader 2024-06-26T07:56:05.1598373Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:56:05.1619706Z http.https://github.com/.extraheader 2024-06-26T07:56:05.1643533Z Entering 'third_party/flatbuffers' 2024-06-26T07:56:05.1666514Z http.https://github.com/.extraheader 2024-06-26T07:56:05.1693361Z Entering 'third_party/fmt' 2024-06-26T07:56:05.1715177Z http.https://github.com/.extraheader 2024-06-26T07:56:05.1737581Z Entering 'third_party/foxi' 2024-06-26T07:56:05.1758940Z http.https://github.com/.extraheader 2024-06-26T07:56:05.1782940Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:56:05.1805494Z http.https://github.com/.extraheader 2024-06-26T07:56:05.1830108Z Entering 'third_party/gloo' 2024-06-26T07:56:05.1852347Z http.https://github.com/.extraheader 2024-06-26T07:56:05.1875733Z Entering 'third_party/googletest' 2024-06-26T07:56:05.1896392Z http.https://github.com/.extraheader 2024-06-26T07:56:05.1917015Z Entering 'third_party/ideep' 2024-06-26T07:56:05.1938198Z http.https://github.com/.extraheader 2024-06-26T07:56:05.1959398Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:56:05.1979395Z http.https://github.com/.extraheader 2024-06-26T07:56:05.2006351Z Entering 'third_party/ittapi' 2024-06-26T07:56:05.2028679Z http.https://github.com/.extraheader 2024-06-26T07:56:05.2050798Z Entering 'third_party/kineto' 2024-06-26T07:56:05.2073534Z http.https://github.com/.extraheader 2024-06-26T07:56:05.2106453Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:56:05.2129767Z http.https://github.com/.extraheader 2024-06-26T07:56:05.2154246Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:56:05.2178040Z http.https://github.com/.extraheader 2024-06-26T07:56:05.2200929Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:56:05.2220936Z http.https://github.com/.extraheader 2024-06-26T07:56:05.2242038Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:56:05.2261729Z http.https://github.com/.extraheader 2024-06-26T07:56:05.2283480Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:56:05.2305530Z http.https://github.com/.extraheader 2024-06-26T07:56:05.2327234Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:56:05.2347848Z http.https://github.com/.extraheader 2024-06-26T07:56:05.2374735Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:56:05.2395923Z http.https://github.com/.extraheader 2024-06-26T07:56:05.2417989Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:56:05.2440794Z http.https://github.com/.extraheader 2024-06-26T07:56:05.2465448Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:56:05.2485784Z http.https://github.com/.extraheader 2024-06-26T07:56:05.2507420Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:56:05.2530319Z http.https://github.com/.extraheader 2024-06-26T07:56:05.2554172Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:56:05.2574236Z http.https://github.com/.extraheader 2024-06-26T07:56:05.2595481Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:56:05.2617348Z http.https://github.com/.extraheader 2024-06-26T07:56:05.2639847Z Entering 'third_party/mimalloc' 2024-06-26T07:56:05.2660209Z http.https://github.com/.extraheader 2024-06-26T07:56:05.2680746Z Entering 'third_party/nccl/nccl' 2024-06-26T07:56:05.2700786Z http.https://github.com/.extraheader 2024-06-26T07:56:05.2721414Z Entering 'third_party/nlohmann' 2024-06-26T07:56:05.2743843Z http.https://github.com/.extraheader 2024-06-26T07:56:05.2766979Z Entering 'third_party/onnx' 2024-06-26T07:56:05.2787236Z http.https://github.com/.extraheader 2024-06-26T07:56:05.2820280Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:56:05.2840264Z http.https://github.com/.extraheader 2024-06-26T07:56:05.2862292Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:56:05.2886486Z http.https://github.com/.extraheader 2024-06-26T07:56:05.2913543Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:56:05.2935776Z http.https://github.com/.extraheader 2024-06-26T07:56:05.2957529Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:56:05.2977776Z http.https://github.com/.extraheader 2024-06-26T07:56:05.2998835Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:56:05.3018388Z http.https://github.com/.extraheader 2024-06-26T07:56:05.3038629Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:56:05.3057899Z http.https://github.com/.extraheader 2024-06-26T07:56:05.3078320Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:56:05.3098162Z http.https://github.com/.extraheader 2024-06-26T07:56:05.3119341Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:56:05.3139465Z http.https://github.com/.extraheader 2024-06-26T07:56:05.3159651Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:56:05.3181159Z http.https://github.com/.extraheader 2024-06-26T07:56:05.3202637Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:56:05.3222543Z http.https://github.com/.extraheader 2024-06-26T07:56:05.3242879Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:56:05.3264040Z http.https://github.com/.extraheader 2024-06-26T07:56:05.3287558Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:56:05.3307052Z http.https://github.com/.extraheader 2024-06-26T07:56:05.3329902Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:56:05.3350851Z http.https://github.com/.extraheader 2024-06-26T07:56:05.3389671Z Entering 'third_party/pocketfft' 2024-06-26T07:56:05.3411080Z http.https://github.com/.extraheader 2024-06-26T07:56:05.3432063Z Entering 'third_party/protobuf' 2024-06-26T07:56:05.3454218Z http.https://github.com/.extraheader 2024-06-26T07:56:05.3477250Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:56:05.3499176Z http.https://github.com/.extraheader 2024-06-26T07:56:05.3520593Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:56:05.3541954Z http.https://github.com/.extraheader 2024-06-26T07:56:05.3567284Z Entering 'third_party/psimd' 2024-06-26T07:56:05.3589942Z http.https://github.com/.extraheader 2024-06-26T07:56:05.3612722Z Entering 'third_party/pthreadpool' 2024-06-26T07:56:05.3636209Z http.https://github.com/.extraheader 2024-06-26T07:56:05.3660077Z Entering 'third_party/pybind11' 2024-06-26T07:56:05.3680840Z http.https://github.com/.extraheader 2024-06-26T07:56:05.3705083Z Entering 'third_party/python-peachpy' 2024-06-26T07:56:05.3727243Z http.https://github.com/.extraheader 2024-06-26T07:56:05.3749715Z Entering 'third_party/sleef' 2024-06-26T07:56:05.3770625Z http.https://github.com/.extraheader 2024-06-26T07:56:05.3794712Z Entering 'third_party/tensorpipe' 2024-06-26T07:56:05.3815727Z http.https://github.com/.extraheader 2024-06-26T07:56:05.3840747Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:56:05.3861536Z http.https://github.com/.extraheader 2024-06-26T07:56:05.3884141Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:56:05.3905673Z http.https://github.com/.extraheader 2024-06-26T07:56:05.3927301Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:56:05.3949497Z http.https://github.com/.extraheader 2024-06-26T07:56:05.3972998Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:56:05.3994626Z http.https://github.com/.extraheader 2024-06-26T07:56:05.4017197Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:56:05.4037747Z http.https://github.com/.extraheader 2024-06-26T07:56:05.4136806Z A job completed hook has been configured by the self-hosted runner administrator 2024-06-26T07:56:05.4154374Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T07:56:05.4160972Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:56:05.4161479Z ##[endgroup] 2024-06-26T07:56:06.1976738Z Cleaning up orphan processes